This test requires restart of infrastructure components (database and network) - these steps must be coordinated with the other testers. Only the steps under "Database crash test" and "Network recovery test" below need to be coordinated.Resubmit jobs after restart, restart of failed jobs, upload of old files at harvester restart, scheduler skips old jobs.
Uses heritrix3 templates default_order_xml
Install and Start System
On email@example.com, run the tests as specified under "Release Test Information" on 5.4 Release Test, and with TESTX set to TEST6
Check that the GUI is available and that the System Status does not show any startup problems.
Start a selective harvest
Start an hourly selective harvest for the 'netarkivet.dk' domain.
Create a new template:
Edit Harvest Templates, download the template "default_orderxml", by choosing "Save to disk" in the pull-down menu, and clicking "Retrieve".
- Edit the template so that max-size-bytes is 5000 in the WARCWriterProcessor. Do this as follows.
<bean id="simpleOverrides"and then under
metadata.description=Default Profile generating small warc-files (5000 bytes)
warcWriter.maxFileSizeBytes = 5000
<bean id="metadata", modify the line to read:
<property name="jobName" value="default_orderxml_smallwarcs" />
- Upload the template with the filename changed to "default_orderxml_smallwarcs.xml"
Modify domain templates
To do the following configures, go to Definitions→Find Domain(s) and search for the domain-URL in question. The click on the given domain, and you'll find the defaultconfig under Configurations, which you can edit.
- Configure the defaultconfig for kum.dk to use template 'default_orderxml_smallwarcs'.
- Configure the defaultconfig for dbc.dk to use template 'default_orderxml', and max-hops=0
- Configure the defaultconfig for bs.dk to use template 'default_orderxml', and max-hops=10
Make a new snapshot harvest definition with a name you can remember
- Create a new snapshot harvest, with 'Max number of bytes per domain’ set to 1,000,000 bytes (1 mbyte). Activate it.
- Go to 'Harvest status'->'All Jobs' in the left menu, and check that there are jobs for the harvest you just created. Check that the status of one of these changes from "New" to "Started", then go to "Systemstate"→"Overview of the system state", and check that no errors or warnings are present in the system overview.
Now immediately proceed to the following test ("Stop the Test Automatically During Upload").
Stop the Test Automatically During Upload
- Using the GUI, go to "Harvest status"→"All Jobs", and by clicking each Job ID for the snapshot harvest in turn, find the job ID for the job in which kum.dk is being harvested.
- Go back to "Harvest status"→"All Jobs", and reload the page until the job you just identified has status "Started"... then immediately go to "Harvest status"→"H3 Remote Access", keep reloading the page until the job ID found above appears, and click the job ID (this may take several tries until it is ready) then immediately pause the job.
- Go to "Harvest status"→"H3 Remote Access" and click the job ID you identified, then click "View/Search in cached Crawllog", then "Update cache". Go to "Harvest status"→"All Running Jobs" and search for "kum.dk" to find the job, then note down the name of the harvest machine (Host) for that job.
- Download the attached script and modify it to point at the correct harvester and job number
- Copy the script to
kb-prod-udv-001.kb.dk:/home/devel/, give it a "
chmod 755"then run it. It monitors the "warcs" directory and as soon as the first warcfile is uploaded it detects that uploading has started and shuts down the test instance.
- Go to "Harvest status"→"H3 Remote Access"→identified Job ID, and unpause the job (no explicit logout is necessary)
- Wait for the job to complete, after which the TEST6 instance is stopped, starting with the apps on machine harvesting kum.dk
Save the Metadata Warcfile
kb-prod-udv-001.kb.dk, log into the harvester where kum.dk was being harvested (with user netarkdv if harvester is in Aarhus, and user
develif harvester is in Kbh).
- Find the crawldir in TEST6/harvester_low
- Find the metadata warcfile in the metadata subdirectory and copy it to TEST6/
Create a Fake Crawl Dir
Wait atleast 3 Hours then Restart the System
Wait atleast 3 Hours then Restart the System (by running
Verify the restarted system. On devel@kb-test-adm-001
Check the log for warnings and errors.
When checking for warnings/errors, be sure to ignore any warnings/error that happened before the above restart. Also, the following kinds of entries are normal/known, and can be ignored:
The following kind of warning can be ignored, unless it appears repeatedly:
The following warning may occur after a while, and can be ignored as well:
Any other warning should be considered a release test failure.
- Go to the system overview page and check that all the expected applications are listening and are up without warnings or errors.
If there is a warning of this kind:
Remote JMX bean generated exception:
then refresh the system state overview page. The warning should disappear.
- Check that the scheduler schedules only one job for the hourly selective harvest.
Check that a job can be resubmitted
- Go to "Harvest status"→"All Jobs", select job status "Failed", and press "Show". Check that you can reject a job for resubmission using the "Reject?" button so that it is no longer visible when you list failed jobs.
- Check that you can see the rejected job when you now list all jobs.
- Click on one or more "Genstart"/"Restart?" buttons to resubmit. Note that you only can resubmit jobs failed due to harvesting errors, not due to upload errors.
- Check that the job-status changes to "resubmitted" and that a new Job is made from the same harvestdefinition with the same configurations.
- Check that resubmitted jobs contain information about which job they were resubmitted (NAS-1466)
Check Report Generation
Use a browser set up as a viewerproxy connection for this test (see https://sbprojects.statsbiblioteket.dk/pages/viewpage.action?pageId=37597440#TheNetarkivetDistributedTest/DevelEnvironment-ViewerProxyUsage ). Select any completed job and click on the "Browse reports for jobs" link.
You should see a list like
Check that a few (like, 3) of the entries are present and browse each in turn. (Note that the HeritrixVersion, harvestIf, and jobId will differ). Some of the entries might be empty.
The following two tests ("Database crash test" and "Network recovery test") must be coordinated with the other testers.
Database crash test
Tests that the system can survive a database crash/stop and resume operation after the database is restarted
Log in as root on kb-test-adm-001
Stop the postgresdb and wait a couple of minutes.
- Verify that the GUI has lost the connection to the database by listing domains or harvest definitions.
Restart the database
- Check that the different GUI pages works as usual.
- Create a new active selecive and verify the a job is created and started.
Network recovery test
Tests that the system can survive a network crash/stop and resume operation after the becomes available.
Disable the network/switch/DC for some minutes and see that all batch processes reconnects and continue after restart
login on to kb-test-adm-001 as root and stop the networkinterface by installing a cron-job that does this for you:
Install script restartNetworkWithWait.sh as root cronjob (Add 0 17 * * * (/root/restartNetworkWithWait.sh) to restart network at 5 PM)
# stopping network
# starting network
- Check that the connection to the GUI is lost.
- After 5 minutes verify the system comes back online
- Verify that the GUI pages are working properly.
- Create a new active selective harvest definition and verify that a new job is created and started.
- Run a batch job or two and verify these work correctly.
Shutdown the system