Test snapshot harvesting in detail and subsequent follow-up harvesting
On email@example.com start a standard test installation with TESTX=TEST6 as described on the Release Test homepage.
Check Domain Statistics
Go to the GUI on http://kb-test-adm-001.kb.dk:8074/HarvestDefinition/ (replace port number as needed) and
- Check that the installation has initially 16 domains loaded
- Check that you can search for domains by name or wildcard
- Check that you can create a new domain (add unknown001.dk: it doesn't exist)
Global Crawler Traps
- Download the file crawlertrapsCollection.txt and upload it as a global-crawler-trap list.
- Download the list again from the GUI and compare it with original list. If necessary use "sort -u" to remove any duplicates and ensure that the two versions are ordered identically. A sample Unix command line:
diff <(sort -u crawlertrapsCollection.txt ) <(sort -u crawlertrap ) (note that any empty lines are also removed during upload)
Update Byte Limits
Update the "Maximum number of bytes" for the defaultconfig for six domains as follows:
Add Alias Domain
- Using the GUI, set netarkivet.dk to be an alias of kb.dk.
- Go to edit page of the domain 'netarkivet.dk' using the HarvestDefinition/Definitions-find-domains.jsp
- In the "Alias of" field, type 'kb.dk'
- Click 'save'
- Confirm that it is listed on the "Alias Summary" page.
- Now try to make dbc.dk an alias of netarkivet.dk. This should fail because chains of aliases are not allowed.
- Go to edit page of the domain 'dbc.dk' using the HarvestDefinition/Definitions-find-domains.jsp
- In the "Alias of" field, type 'netarkivet.dk'
- Click save. An error message should be shown on screen;
"Cannot make domain 'dbc.dk' an alias of 'netarkivet.dk', as that domain is already an alias of 'kb.dk'"
Start a Snapshot Harvest
- Create and activate a snapshot harvest with a "Max number of bytes per domain" of 1000000.
- Wait until HarvestJobManager on the Status page shows that jobs have been created for the harvest.
Check that Alias Domain is not Harvested
For each job generated, check that netarkivet.dk is not included in the domains harvested.
Check that Jobs Complete as Expected
Use the Harvest Status section of the GUI to monitor the jobs. When all jobs have finished, check each job in turn to see that the domains report their "Stopped due to" as follows:
bs.dk Domain Completed
kum.dk Max Bytes limit reached
oernhoej.dk Domain Completed
drive-badmintonklub.dk Max Bytes limit reached
sulnudu.dk Domain-config byte limit reached
statsbiblioteket.dk Domain-config byte limit reached
kb.dk Domain-config byte limit reached (currently blocked so Domain Completed)
raeder.dk Domain Completed
trinekc.dk Max Bytes limit reached
kaarefc.dk Domain Completed
kaareogtrine.dk Max Bytes limit reached
dbc.dk Domain-config byte limit reached
unknown001.dk Domain Completed
slothchristensen.dk Max Bytes limit reached
pligtaflevering.dk Max Bytes limit reached
trineogkaare.dk Max Bytes limit reached
sy-jonna.dk Max Bytes limit reached
If any of these have a different reason, investigate to see if the new stop reason makes sense. Generally speaking, domains with DomainCompleted will have significantly under 1000000 bytes harvested, while the other will be somewhat, but not vastly, over this limit.
Add a New Alias
Set sulnudu.dk to be an alias of kb.dk
Change a Byte Limit
For kb.dk set the Maximum Number of Bytes on defaultconfig to -1 (no limit).
Start a Second Phase Snapshot Harvest
Create a new snapshot harvest with a 'Max number of bytes per domain' of 5MB (i.e, 5000000 bytes) , harvesting domains not completed in the previous harvest.
Test NAS Heritrix Integration
In "Harvest status > All Running Jobs" wait until one of the jobs has nonzero progress. Click on the Job ID to enter H3 Remote Access.
Pause the job.
Check the following functionality:
- Crawllog: check cache update, filtering, paging
- Reports: click on at least two of them. Check that the Processors report shows that deduplication has been disabled.
- Show/delete frontier: delete some items from the frontier
- Add RejectRules: add a new rule
- Modify budget: add an object limit to some domain or subdomain
Now restart the job. Then immediatiely ...
Restart The System
- Stop NAS with the "stop_test.sh" command
- On kb-test-adm-001/TEST2/conf change the settings for HarvestJobManager so deduplicaiton.enabled is "true"
- restart the NAS system with the "start_test.sh" command. After some time, the job you paused and unpaused should appear in the state "Failed".
- Restart the job by clicking the "Restart?" button. A new job should be created and the old one should have the status "Resubmitted (Job X)".
- Wait for the job to finish.
- Meanwhile click on the JobID for the failed job, then click on "Browse reports for jobs".
- You should see a list of available reports including one called "scripting_events.log". This is the log of alterations you made to the frontier via H3 Remote Access. Click on it.
Assuming you have the correct viewerproxy setup (See Setup DK test environment) you should a log line describing your action. Something similar to
2016-02-02T14:05:59.170Z Action from user CSR: Deleted 563 uris matching regex '.*kb.dk.*'
Check the Processors report. You should see a report on the non-zero number of objects which has been deduplicated.
Check that Alias Domains are not Harvested
For each job generated in the last harvest:
- Check that netarkivet.dk and sulnudu.dk are not listed as being harvested.
- Check that neither domain appears in the order templates for any of the jobs with the (possible) exception of the following lines:
This can be done by grepping with a command like
[devel@kb-prod-udv-001 ~]$ ssh firstname.lastname@example.org grep netarkivet.dk /netarkiv/0001/TEST2/filedir/*-metadata-1.warc | grep -v 'metadata:'
or by scp'ing the metadata file to kb-prod-udv-001 and inspecting it with "less". (Or just by displaying the order template in the NAS GUI and searching.)
Check Byte Limits for the Second Harvest
- Confirm that the stop reason "Max Bytes limit reached" or "Domain Completed" is given for all the domains included.
- Confirm that domains for which the domain-config byte limit was reached in the previous harvest are not present in any job in this harvest. (e.g. dbc.dk). The exception to this is the one domain for which you changed the domain limit to unlimited.
Stop the Test and Clean-Up