- Test functionality of latest software with existing production database
- Test database schema upgrade procedures
- Test consistency of database schemas (production, bundled)
- Validate performance scaling with production database
- Validate bundled heritrix templates
There should be a copy of the current production database available in email@example.com:prod-backup. The directory should contain the subdirectory CS, and the postgres dumps prod_admindb.dump.out.gz and prod_harvestdb.dump.out.gz .
Copy Production Databases to Test System
Copy production databases to the relevant machines. On firstname.lastname@example.org
ssh email@example.com rm -rf /tmp/prod_admindb.dump.tar
ssh firstname.lastname@example.org rm -rf /tmp/prod_harvestdb.dump.tar
ssh email@example.com rm -rf /tmp/CS
scp -r /home/test/prod-backup/pg_prod_harvestdb_*.tar firstname.lastname@example.org:/tmp/prod_harvestdb.dump.tar
scp -r /home/test/prod-backup/pg_prod_admind_*.tar email@example.com:/tmp/prod_admindb.dump.tar
scp -r /home/test/prod-backup/CS firstname.lastname@example.org:/tmp
Install Test and Configure to Use Production Databases
On email@example.com, replace test databases with prod:
pg_restore -U test -d test7_admindb --clean --no-owner /tmp/prod_admindb.dump.tar
pg_restore -U test -d test7_harvestdb --clean --no-owner /tmp/prod_harvestdb.dump.tar
<or possibly better>
psql -U test -c 'drop database if exists test7_harvestdb'
psql -U test -c 'create database test7_harvestdb'
pg_restore -U test -d test7_harvestdb --no-owner --schema public /tmp/prod_harvestdb.dump.out
Ignore errors relating to deleting and restoring the schema "public". There should be three such errors per restore. You can follow the progress of the import as follows
[test@kb-test-adm-001 tmp]$ psql -U test -d test7_harvestdb
test7_harvestdb=> SELECT schemaname,relname,n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;
On firstname.lastname@example.org, replace test CS replica with prod:
rm -rf /home/test/$TESTX/CS
ln -s /tmp/CS /home/test/$TESTX/CS
Upgrade the database
java -Xmx1536m -Ddk.netarkivet.settings.file=./conf/settings_GUIApplication.xml -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djava.util.logging.config.file=./conf/log_GUIApplication.prop -Djava.security.manager -Djava.security.policy=./conf/security.policy dk.netarkivet.harvester.tools.HarvestdatabaseUpdateApplication < /dev/null > start_harvestdatabaseUpdateApplication.log 2>&1
Check the logfile. There may be warnings about new tables but there should be no exceptions thrown.
(NOT CURRENTLY USED) Generate Database Schemas
These steps have become a bit of a mess. We need to define what they're for and what we expect. I think the basic position should be
- There should be only one postgres schema in the release package.
- There should be a separate pgsql script with the basic testdata.
- After running HarvestdatabaseUpdateApplication on the production data, the schema of the prod data should be logically equivalent to that of the bundled schema.
- The same applies to admindb.
To operationalise these into a test we need to find a way of automating logical-comparison of two database schemas.
Here we compare the updated production database schema with the bundled database.
rm -rf /home/test/schemas
ssh kb-test-adm-001 pg_dump -s -U test test7_harvestdb > proddbs_schema.txt
scp kb-test-adm-001:TEST7/scripts/sql/createHarvestDB.pgsql bundleddbs_schema.txt
(NOT CURRENTLY USED) Compare the Database Schemas
Sort the schemas and remove uninteresting lines
grep -v GRANT proddbs_schema.txt |grep -v REVOKE| grep -v INDEX|grep -v ALTER|grep -v ^--|grep -v CONSTRAINT|grep -v SET > proddbs_schema.txt.clean
grep -v GRANT bundleddbs_schema.txt |grep -v REVOKE| grep -v INDEX|grep -v ALTER|grep -v INSERT|grep -v ^--|grep -v CONSTRAINT|grep -v SET|grep -v VALUES > bundleddbs_schema.txt.clean
Check for Difference
diff -b -B bundleddbs_schema.txt.clean proddbs_schema.txt.clean > test-bundled-diff
The only allowed differences are in the order of the table fields.
Start the Installation
and edit startall_K.sh and startall_S.sh to comment out any lines for harvester machine (kb-test-har*, sb-test-har*). Then
Verify that Jobs are no Longer Resubmitted
[test@kb-prod-udv-001 TEST7]$ ssh kb-test-adm-001 grep \--before-context=1 esubmitt /home/test/$TESTX/log/HarvestJobManagerApplication0.log.0
Aug 9, 2013 1:24:21 PM dk.netarkivet.harvester.scheduler.JobSupervisor rescheduleLeftOverJobs
INFO: 0 jobs has been resubmitted.
Go to https://sbforge.org/jenkins/view/NetarchiveSuite/job/Netarchivesuite-db-full-migration-test/ and check that the build is green and from a current release candidate.
Check the GUI
Go to http://kb-test-adm-001.kb.dk:8073/BitPreservation/index.jsp . Check that the GUI is running.
Check Job Status for Failed Jobs
In the GUI:
- View jobs with status "Failed". Choose the most recent 1000 jobs (Descending). Confirm that jobs which failed due to harvester failures or scheduler timeout have "Restart" and "Reject" buttons, while any that failed due to upload errors do not.
Validate GUI Performance
- Check response times for all links in "Definitions" and "Harvest Status" site sections.
- Check that in the Harvest Status section one can choose multiple values of status with both increasing and decreasing order with reasonable response times.
- Note that the "Bitpreservation" section of the GUI responds rather sluggishly. Currently it takes 7 minutes to load (v4.4).
Define a Selective Harvest
Create a selective harvest of netarkivet.dk with frequency once per week and activte it. Check that the time for "next run" is shown as now.
Run Checksum and Filestatus updates
[Note: This step and the next two can be run in parallel. For example the CSN checksum job and the snapshot-job-generation step can be run overnight.]
Go to the Bitpreservation site section in the GUI.
- Click on update for filestatus for replica KB. By reloading the Bitpreservation page you can check the progress of this command - it should eventually show all files as missing. It should take under an hour to complete.
Now run a checksum status for replica CSN ("Update checksum and filestatus for CS"). This takes about 11 hours to complete (ver. 3.18.0). You can follow the progress in the GUIApplication log by looking for line like
Apr 29, 2014 12:13:39 PM dk.netarkivet.archive.arcrepositoryadmin.ReplicaCacheDatabase addChecksumInformation
INFO: Processed checksum list entry number 90000 for replica CHECKSUMReplica (CS) CSN
The total number of entries to be processed can be seen in the Bitpreservation section of the GUI. (Now 21 hours in v4.4 with 4.3 million files.)
Ingest DK Domain List
- Find three domain that don't exist in the NAS installation (by searching for them in the GUI). At least one domain should contain Danish and French characters.
- Check that raeder.dk and statsbiblioteket.dk are already known.
- Fetch the latest domain list from email@example.com:prod-backup
- Add the three domains from step 1 to this file.
- Upload the file via the "Create Domains" page in the definitions section of the GUI. The ingest should take no more than five minutes and the page should show progress for every 10000th domain ingested. Ignore warnings about invalid lines like '-----------------------------------'.
- Check that both the the three new domains from step 1 and the two existing domains from step 2 are now to be found in the system.
- Check that the three new domains have no harvesting history and that the two existing domains do.
Generate Jobs for a Snapshot Harvest
- Create a snapshot harvest with 'Max number of bytes per domain’ set to 100.000. Save and activate it.
- Check that new jobs are generated in the Harvest Status section. The process can take up to 9 hours to complete.
- Check that the Status of the GUI server eventually shows "INFO: Created X jobs for harvest definition <harvest name>".
Job creation progress can be monitored by
- Find the total number of known domains by clicking on "Domain Statistics"
- Find the number of jobs generated so far for your harvest from the "Harvest Status" part of the GUI.
Since there are very nearly 10000 domains per job you can quickly see how close scheduling is to completion.
You can also check progress by looking at the HarvestJobManager application log, for example with
ssh kb-test-adm-001 tail TEST7/log/HarvestJobManagerApplication0.log.0
or by grep'ing on the harvest name.
Validate GUI Performance Again
As above, check that the GUI performance is not noticeably degraded after the snapshot job creation process.
Validate Heritrix TemplatesOn test@kb
Confirm Snapshot Harvest generation
- Go to the Snapshot Harvest definition page
- The most recent Snapshot Harvest should have a random six-character name. Click on its "history" link.
- Its start time should be recent (ie after the start time of the Jenkins job) and should it have generated somewhere over 200 jobs. (243 for the database in current use.)