Where PORT=807? correspond to the test port allocated in the Setup DK test environment page and
MAILRECEIVERSemail@example.com should correspond to your own mail address.
Check Domain Statistics
Go to the GUI on http://kb-test-adm-001.kb.dk:8074/HarvestDefinition/ (replace port number as needed) and
- Check that the installation has initially 17 domains loaded
- Check that you can search for domains by name or wildcard
- Check that you can create a new domain (add unknown001.dk: it doesn't exist)
- Add a new configuration to an existing domain. Confirm that it is listed in the domain definition. (you might need to click on show unused configurations).
Global Crawler Traps
- Download the file crawlertrapsCollection.txt and upload it as a global-crawler-trap list.
- Download the list again from the GUI and compare it with original list. If necessary use "sort -u" to remove any duplicates and ensure that the two versions are ordered identically. A sample Unix command line:
diff <(sort -u crawlertrapsCollection.txt ) <(sort -u crawlertrap )
Update Byte Limits
Update the "Maximum number of bytes" for the defaultconfig for six domains as follows:
Add Alias Domain
- Using the GUI, set netarkivet.dk to be an alias of kb.dk.
- Go to edit page of the domain 'netarkivet.dk' using the HarvestDefinition/Definitions-find-domains.jsp
- In the "Alias of" field, type 'kb.dk'
- Click 'save'
- Confirm that it is listed on the "Alias Summary" page.
- Now try to make dbc.dk an alias of netarkivet.dk. This should fail because chains of aliases are not allowed.
- Go to edit page of the domain 'dbc.dk' using the HarvestDefinition/Definitions-find-domains.jsp
- In the "Alias of" field, type 'netarkivet.dk'
- Click save. An error message should be shown on screen;
"Cannot make domain 'dbc.dk' an alias of 'netarkivet.dk', as that domain is already an alias of 'kb.dk'"
Start a Snapshot Harvest
- Create and activate a snapshot harvest with a "Max number of bytes per domain" of 1000000.
- Wait until HarvestJobManager on the Status page shows that jobs have been created for the harvest.
Check that Alias Domain is not Harvested
For each job generated, check that netarkivet.dk is not included in the domains harvested.
Check that Jobs Complete as Expected
Use the Harvest Status section of the GUI to monitor the jobs. When all jobs have finished, check each job in turn to see that the domains report their "Stopped due to" as follows:
bs.dk Domain Completed kum.dk Max Bytes limit reached oernhoej.dk Domain Completed drive-badmintonklub.dk Max Bytes limit reached sulnudu.dk Domain-config byte limit reached statsbiblioteket.dk Domain-config byte limit reached kb.dk Domain-config byte limit reached raeder.dk Max Bytes limit reached trinekc.dk Max Bytes limit reached kaarefc.dk Domain Completed kaareogtrine.dk Max Bytes limit reached dbc.dk Domain-config byte limit reached unknown001.dk Domain Completed slothchristensen.dk Max Bytes limit reached pligtaflevering.dk Max Bytes limit reached trineogkaare.dk Max Bytes limit reached sy-jonna.dk Max Bytes limit reached
If any of these have a different reason, investigate to see if the new stop reason makes sense. Generally speaking, domains with DomainCompleted will have significantly under 1000000 bytes harvested, while the other will be somewhat, but not vastly, over this limit.
Add a New Alias
Set sulnudu.dk to be an alias of kb.dk
Change a Byte Limit
For kb.dk set the Maximum Number of Bytes on defaultconfig to -1 (no limit).
Start a Second Phase Snapshot Harvest
Create a new snapshot harvest with a 'Max number of bytes per domain' of 5MB (i.e, 5000000 bytes) , harvesting domains not completed in the previous harvest.
Pause a Harvestjob in the Heritrix GUI
In "Harvest status -> Running jobs" the Host field is a link to the relevant Heritrix GUI. Click on this (admin/adminPassword) and pause the job. (If there are firewall problems it might be necessary to login to the harvester machine and start a webbrowser on that machine instead.)
Go to the NAS GUI. In the System Overview for the relevant HarvestController confirm that there is a "Paused" message in the log. (The log message can be up to 15 minutes delayed). Also in the running jobs list, the job should be marked as paused with the red bullet.
Edit the Job and Resume job
- In the Heritrix GUI, choose Jobs -> Edit Configuration and click the Overrides header link to enter a Quick override. Type "netarkivet.dk" in the Domain text box and click "Create/Edit".
Set max-hops to 15
Set delay-factor to 1.5
and click "Done with the override" at the bottom.
- "Override changes saved" confirms the changes are saved.
- Switch back to "Console" and click on "Resume".
- Confirm that the job is running again in the NAS System overview.
Restart The System
- Stop NAS with the "stop_test.sh" command and restart it with the "start_test.sh" command. After some time, a job should appear in the state "Failed".
- Restart the job by clicking the "Restart?" button. A new job should be created and the old one should have the status "Resubmitted (Job X)".
- Wait for the job to finish.
Check the Overrides are Applied
For the failed job, check that the overrides are visible. The easiest way to do this is from test@kb-prod-udv-001: (replacing <jobno> with the number of the original job, here 7)
CLARIFY - WHAT DOES THIS MEAN?: (Note that there should be two setup/order reports. The one containing a timestamp in its name is the original order.xml, the one called simply
metadata://netarkivet.dk/crawl/setup/order.xml is the final modified version.)
Check that Alias Domains are not Harvested
For each job generated in the last harvest:
- Check that netarkivet.dk and sulnudu.dk are not listed as being harvested.
- Check that neither domain appears in the order templates for any of the jobs with the (possible) exception of the following lines:
This can be done by grepping with a command like
or by scp'ing the metadata file to kb-prod-udv-001 and inspecting it with "less".
Check Byte Limits for the Second Harvest
- Confirm that the stop reason "Max Bytes limit reached" or "Domain Completed" is given for all the domains included.
- Confirm that oernehoej.dk and statsbiblioteket.dk are not found in the "Domain" column in any job .
Check that there was no Deduplication
CLARIFY: Using a browser setup for ViewerProxy access, check the processors-report for one of the snapshot-harvest jobs. Confirm that there was no DeDuplicator report with a "Duplicate found" line.
Stop the Test and Clean-Up