Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Excerpt

The performance test on PROD data.

...

Uses a deploy configuration that disables deduplication, so we don't ask for a dedup index before jobgeneration commences

This test is now largely automated. The following steps should be taken to confirm that the automatic test has run as expected.

Table of Contents
outlinetrue

Numbered Headings
start-numbering-ath3

Goals

  • Test functionality of latest software with existing production database
  • Test database schema upgrade procedures
  • Test consistency of database schemas (production, bundled)
  • Validate performance scaling with production database
  • Validate bundled heritrix templates

Prerequisites

There should be a copy of the current production database available in test@kb-prod-udv-001.kb.dk:prod-backup. The directory should contain the subdirectory CS, and the postgres dumps prod_admindb.dump.out.gz and prod_harvestdb.dump.out.gz .

Procedure

Prepare Installation

On test@kb-prod-udv-001.kb.dk:

Code Block
export TESTX=TEST7
export PORT=807?
export MAILRECEIVERS=foo@bar.dk
stop_test.sh
cleanup_all_test.sh
prepare_test.sh deploy_config_dedup_disabled.xml

Copy Production Databases to Test System

Copy production databases to the relevant machines. On test@kb-prod-udv-001.kb.dk

Code Block
ssh test@kb-test-adm-001.kb.dk rm -rf /tmp/prod_admindb.dump.tar
ssh test@kb-test-adm-001.kb.dk rm -rf /tmp/prod_harvestdb.dump.tar
ssh test@kb-test-acs-001.kb.dk rm -rf /tmp/CS
scp -r /home/test/prod-backup/pg_prod_harvestdb_*.tar test@kb-test-adm-001.kb.dk:/tmp/prod_harvestdb.dump.tar
scp -r /home/test/prod-backup/pg_prod_admind_*.tar test@kb-test-adm-001.kb.dk:/tmp/prod_admindb.dump.tar
scp -r   /home/test/prod-backup/CS test@kb-test-acs-001.kb.dk:/tmp

Install Test and Configure to Use Production Databases

  1. On test@kb-prod-udv-001.kb.dk: 

    Code Block
    install_test.sh
    
  2. On test@kb-test-adm-001.kb.dk, replace test databases with prod:

    Code Block
    export TESTX=TEST7
    pg_restore -U test -d test7_admindb --clean --no-owner /tmp/prod_admindb.dump.tar
    pg_restore -U test -d test7_harvestdb --clean --no-owner /tmp/prod_harvestdb.dump.tar
     
    <or possibly better>
    psql -U test -c 'drop database if exists test7_harvestdb'
    psql -U test -c 'create database  test7_harvestdb'
    pg_restore -U test -d test7_harvestdb --no-owner --schema public /tmp/prod_harvestdb.dump.out 
     

    Ignore errors relating to deleting and restoring the schema "public". There should be three such errors per restore. You can follow the progress of the import as follows

    Code Block
    [test@kb-test-adm-001 tmp]$ psql -U test -d test7_harvestdb
    test7_harvestdb=> SELECT schemaname,relname,n_live_tup FROM pg_stat_user_tables  ORDER BY n_live_tup DESC;
  3. On test@kb-test-acs-001.kb.dk, replace test CS replica with prod:

    Code Block
    export TESTX=TEST7
    rm -rf /home/test/$TESTX/CS
    ln -s /tmp/CS /home/test/$TESTX/CS
    

Upgrade the database

On test@kb-test-adm-001.kb.dk

Code Block
export TESTX=TEST7
cd /home/test/$TESTX
export CLASSPATH=./lib/dk.netarkivet.harvester.jar:./lib/dk.netarkivet.archive.jar:./lib/dk.netarkivet.monitor.jar:$CLASSPATH;
java -Xmx1536m  -Ddk.netarkivet.settings.file=./conf/settings_GUIApplication.xml -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djava.util.logging.config.file=./conf/log_GUIApplication.prop -Djava.security.manager -Djava.security.policy=./conf/security.policy dk.netarkivet.harvester.tools.HarvestdatabaseUpdateApplication < /dev/null > start_harvestdatabaseUpdateApplication.log 2>&1

Check the logfile. There may be warnings about new tables but there should be no exceptions thrown.

(NOT CURRENTLY USED) Generate Database Schemas

These steps have become a bit of a mess. We need to define what they're for and what we expect. I think the basic position should be

  1. There should be only one postgres schema in the release package.
  2. There should be a separate pgsql script with the basic testdata.
  3. After running HarvestdatabaseUpdateApplication on the production data, the schema of the prod data should be logically equivalent to that of the bundled schema.
  4. The same applies to admindb.

To operationalise these into a test we need to find a way of automating logical-comparison of two database schemas.

 

Here we compare the updated production database schema with the bundled database.

  1. On test@kb-prod-udv-001.kb.dk

    Code Block
    rm -rf /home/test/schemas
    mkdir /home/test/schemas
    cd /home/test/schemas
    ssh kb-test-adm-001 pg_dump -s -U test test7_harvestdb > proddbs_schema.txt
    scp kb-test-adm-001:TEST7/scripts/sql/createHarvestDB.pgsql bundleddbs_schema.txt

(NOT CURRENTLY USED) Compare the Database Schemas

  1. Sort the schemas and remove uninteresting lines

    Code Block
    grep -v GRANT proddbs_schema.txt |grep -v REVOKE| grep -v INDEX|grep -v ALTER|grep -v ^--|grep -v CONSTRAINT|grep -v SET > proddbs_schema.txt.clean
    grep -v GRANT bundleddbs_schema.txt |grep -v REVOKE| grep -v INDEX|grep -v ALTER|grep -v INSERT|grep -v ^--|grep -v CONSTRAINT|grep -v SET|grep -v VALUES > bundleddbs_schema.txt.clean
    
  2. Check for Difference

    Code Block
    diff -b -B bundleddbs_schema.txt.clean proddbs_schema.txt.clean  > test-bundled-diff
    

    The only allowed differences are in the order of the table fields.

Start the Installation

On test@kb-prod-udv-001.kb.dk

Code Block
cd /home/test/release_software_dist/$TESTX/

and edit startall_K.sh and startall_S.sh to comment out any lines for harvester machine (kb-test-har*, sb-test-har*). Then

Code Block
start_test.sh

Verify that Jobs are no Longer Resubmitted

On test@kb-prod-udv-001.kb.dk

Code Block
[test@kb-prod-udv-001 TEST7]$ ssh kb-test-adm-001 grep \--before-context=1 esubmitt /home/test/$TESTX/log/HarvestJobManagerApplication0.log.0
Aug 9, 2013 1:24:21 PM dk.netarkivet.harvester.scheduler.JobSupervisor rescheduleLeftOverJobs
INFO: 0 jobs has been resubmitted.

 

Check Jenkins

Go to https://sbforge.org/jenkins/view/NetarchiveSuite/job/Netarchivesuite-db-full-migration-test/ and check that the build is green and from a current release candidate.

Check the GUI

Go to http://kb-test-adm-001.kb.dk:8073/BitPreservation/index.jsp . Check that the GUI is running.

Check Job Status for Failed Jobs

In the GUI:

  • View jobs with status "Failed". Choose the most recent 1000 jobs (Descending). Confirm that jobs which failed due to harvester failures or scheduler timeout have "Restart" and "Reject" buttons, while any that failed due to upload errors do not.

Validate GUI Performance

  • Check response times for all links in "Definitions" and "Harvest Status" site sections.
  • Check that in the Harvest Status section one can choose multiple values of status with both increasing and decreasing order with reasonable response times.
  • Note that the "Bitpreservation" section of the GUI responds rather sluggishly. Currently it takes 7 minutes to load (v4.4).

Define a Selective Harvest

Create a selective harvest of netarkivet.dk with frequency once per week and activte it. Check that the time for "next run" is shown as now.

Run Checksum and Filestatus updates

[Note: This step and the next two can be run in parallel. For example the CSN checksum job and the snapshot-job-generation step can be run overnight.]

Go to the Bitpreservation site section in the GUI.

  1. Click on update for filestatus for replica KB. By reloading the Bitpreservation page you can check the progress of this command - it should eventually show all files as missing. It should take under an hour to complete.
  2. Now run a checksum status for replica CSN ("Update checksum and filestatus for CS"). This takes about 11 hours to complete (ver. 3.18.0). You can follow the progress in the GUIApplication log by looking for line like

    Code Block
    Apr 29, 2014 12:13:39 PM dk.netarkivet.archive.arcrepositoryadmin.ReplicaCacheDatabase addChecksumInformation
    INFO: Processed checksum list entry number 90000 for replica CHECKSUMReplica (CS) CSN

    The total number of entries to be processed can be seen in the Bitpreservation section of the GUI. (Now 21 hours in v4.4 with 4.3 million files.)

Ingest DK Domain List

  1. Find three domain that don't exist in the NAS installation (by searching for them in the GUI). At least one domain should contain Danish and French characters.
  2. Check that raeder.dk and statsbiblioteket.dk are already known.
  3. Fetch the latest domain list from test@kb-prod-udv-001.kb.dk:prod-backup
  4. Add the three domains from step 1 to this file.
  5. Upload the file via the "Create Domains" page in the definitions section of the GUI. The ingest should take no more than five minutes and the page should show progress for every 10000th domain ingested. Ignore warnings about invalid lines like '-----------------------------------'.
  6. Check that both the the three new domains from step 1 and the two existing domains from step 2 are now to be found in the system.
  7. Check that the three new domains have no harvesting history and that the two existing domains do.

Generate Jobs for a Snapshot Harvest

  1. Create a snapshot harvest with 'Max number of bytes per domain’  set to 100.000. Save and activate it.
  2. Check that new jobs are generated in the Harvest Status section. The process can take up to 9 hours to complete.
  3. Check that the Status of the GUI server eventually shows "INFO: Created X jobs for harvest definition <harvest name>".

Job creation progress can be monitored by

  1. Find the total number of known domains by clicking on "Domain Statistics"
  2. Find the number of jobs generated so far for your harvest from the "Harvest Status" part of the GUI.

Since there are very nearly 10000 domains per job you can quickly see how close scheduling is to completion.

You can also check progress by looking at the HarvestJobManager application log, for example with

Code Block
ssh kb-test-adm-001 tail TEST7/log/HarvestJobManagerApplication0.log.0

or by grep'ing on the harvest name.

Validate GUI Performance Again

As above, check that the GUI performance is not noticeably degraded after the snapshot job creation process.

Validate Heritrix Templates

On test@kb

Confirm Snapshot Harvest generation

  1. Go to the Snapshot Harvest definition page
  2. The most recent Snapshot Harvest should have a random six-character name. Click on its "history" link.
  3. Its start time should be recent (ie after the start time of the Jenkins job) and should it have generated somewhere over 200 jobs. (243 for the database in current use.)

Appendix

Children Display

pg_dump -s -U test test7_admindb

http://kb-test-adm-001.kb.dk

...

  1. Stop the test.

    Code Block
    ${TESTX}/conf/killall.sh
    
  2. Download all templates

    Code Block
    export DATABASE=jdbc:postgresql://localhost:5432/test7_harvestdb
    
    export TEMPLATES=$(java -Xmx1024m -Ddk.netarkivet.settings.file=/home/test/$TESTX/conf/settings_GUIApplication.xml -Dsettings.harvester.datamodel.database.url=$DATABASE -cp /home/test/$TESTX/lib/dk.netarkivet.harvester.jar dk.netarkivet.harvester.tools.HarvestTemplateApplication showall)
    
    for I in $TEMPLATES; do
        java -Xmx1024m -Ddk.netarkivet.settings.file=/home/test/$TESTX/conf/settings_GUIApplication.xml -Dsettings.harvester.datamodel.database.url=$DATABASE -cp /home/test/$TESTX/lib/dk.netarkivet.harvester.jar dk.netarkivet.harvester.tools.HarvestTemplateApplication download $I
    done
    
  3. Now upload all the templates

    Code Block
    export TEMPLATESXML=*.xml
    
    for I in $TEMPLATESXML; do
        export NAME=`basename $I .xml`
        java -Xmx1024m -Ddk.netarkivet.settings.file=/home/test/$TESTX/conf/settings_GUIApplication.xml -Dsettings.harvester.datamodel.database.url=$DATABASE -cp /home/test/$TESTX/lib/dk.netarkivet.harvester.jar dk.netarkivet.harvester.tools.HarvestTemplateApplication update $NAME $I
    done
    

Close Down the Test

On test@kb-prod-udv-001.kb.dk

Code Block
cleanup_all_test.sh

Appendix

Children Display

pg_dump -s -U test test7_admindb

:8073/BitPreservation/Bitpreservation-filestatus.jsp