Page tree
Skip to end of metadata
Go to start of metadata
Test of the Bitpreservation functionality

Recent issues: NAS-2078 - Getting issue details... STATUS

Goals

  • Database based Bitpreservation

Prerequisites

None special.

Procedure

Prepare Installation

See Setup DK test environment.

Upload a Small Bitarchive

ssh devel@kb-prod-udv-001.kb.dk
scp -r ${HOME}/bitarchive_testdata kb-test-adm-001:
ssh kb-test-adm-001 chmod 755  bitarchive_testdata/upload.sh
#ssh kb-test-adm-001 'source .bash_profile;bitarchive_testdata/upload.sh $TESTX arcfiles'
#ssh kb-test-adm-001 'source .bash_profile;bitarchive_testdata/upload.sh $TESTX warcfiles'
ssh devel@kb-test-adm-001
bash bitarchive_testdata/upload.sh $TESTX arcfiles
bitarchive_testdata/upload.sh $TESTX warcfiles

Check Bitpreservation Status

Go to the Bitpreservation section of the GUI and update information for each replica.

Each replica should have the same number of files (25) and there should be no errors.

Delete a File

ssh devel@kb-prod-udv-001.kb.dk
mkdir -p temp_test3a
scp netarkdv@sb-test-bar-001.statsbiblioteket.dk:/netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata* temp_test3a/
ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk rm /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*

Check that File is Missing in GUI

In the GUI, update the file status for SB. There should be one file missing.

Restore File Manually

Copy the file back:

scp temp_test3a/2-metadata* netarkdv@sb-test-bar-001.statsbiblioteket.dk:/netarkiv/0001/netarkdv/${TESTX}/filedir/

Now go to the Bitpreservation GUI and and check the box Get info for the "missing" file. The page should now say "There are no more missing files".

Delete the File Again

Delete the file again as in the above procedure.

Restore the File via the GUI

  • Click on Show missing files.  

  • Click to check the box where it says Get info for <foobar>.arc, and then press execute.  

  • Verify that upload status for the file is UPLOAD_COMPLETED for KBN and CSN, whereas it is UPLOAD_FAILED for SBN.  
  • Verify that the file has the same checksum in KBN and CSN, whereas it should not have any checksum for SBN.  
  • Click to check the box where it says Add to replica SBN, and then press execute.  

  • Verify that you get the message: The file <foobar>.arc has been restored in bitarchive replica SBN.

Corrupt a File in the Bitarchive

ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk ls -l /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*
ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk "echo foobar >> /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*"
ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk ls -l /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*

The file size should have increased by a few bytes.

In the GUI, update the checksums for the replica SBN.

There should now be one file with an error. The file should have status UPLOAD_FAILED for SBN. Remain on this page in the GUI.

The number of files in all the replicas should be equal.

Corrupt the File Further

ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk "echo barfoo >> /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*"

Try to Correct the File Through the GUI

Try to correct the file through the GUI (the credentials should be given as "examplecredentials").

It should fail with the message ...failed due to checksum mismatch... .

Correct the File Through the GUI

  • Now go back to the Bitpreservation.
  • Click on "Show files with errors" and "info".
  • Now try to restore the file but using the wrong credentials. This should fail.
  • Now restore the file using the correct credentials (examplecredentials). This should succeed with a suitable information message like "The file 2-metadata-1.arc has been replaced at replica BITARCHIVEReplica (SB) SBN"

Delete a Checksum Entry

ssh kb-test-acs-001 "sed -i '/2-metadata/d' ${TESTX}/CS/checksum_CS.md5"

Verify the Deleted Entry

  • Click on Update for the filelist for the replica.  

  • Check that it now contains one missing file.  
  • Check that the number of files are 1 lower than the other replicas.  
  • Check that the Last update for the filelist is now (or just a moment ago).

Correct the Deleted Entry

  • Click on Show missing files.  

  • Click to check the box where it says Get info for 2-metadata-1.arc, and then press execute.  

  • Verify that upload status for the file is UPLOAD_COMPLETED for KBN and SBN, whereas it is UPLOAD_FAILED for CSN.  
  • Verify that the file has the same checksum in KBN and SBN, whereas it should not have any checksum for CSN.  
  • Click to check the box where it says Add to replica, and then press execute.  

  • Verify that you get the message: The file 2-metadata-1.arc has been restored in replica CSN.

Corrupt a Checksum Entry

ssh kb-test-acs-001 "sed -i '/2-metadata/s/$/foobar/' ${TESTX}/CS/checksum_CS.md5"
  • Click on Update for the checksum status for the replica.  

  • Verify that it now contains one file with error.  
  • Verify that no files are missing from the replica.  
  • Verify that the number of files are the same as the other replicas.  
  • Verify that the Last update for the checksum status and filelist is now (or just a moment ago).

Correct the Corrupted Entry

  • Click on Show files with error.  

  • Verify that only one file is shown.  
  • Click on the info link for the file 1-metadata-1.arc.  

  • Verify that upload status for the file is UPLOAD_COMPLETED for KBN and SBN, whereas it is UPLOAD_FAILED for CSN.  
  • Verify that the file has the same checksum in KBN and SBN, whereas it should be a different one for CSN.  
  • Write the credentials (default examplecredentials) and press the button: The file 1-metadata-1.arc has been replaced at bitarchive CHECKSUMReplica (CS) CSN.

  • Return to the Filestatus page and check that there are no errors or missing files in any replica.

Stop the Test and Clean-Up

cleanup_all_test.sh
  • No labels

3 Comments

  1. The instruction "ssh kb-test-adm-001 bitarchive_testdata/upload.sh $TESTX arcfiles"
    results in Working on TEST3A on files in /arcfiles
    Exception in thread "main" java.lang.UnsupportedClassVersionError: dk/netarkivet/archive/tools/Upload : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    1. If you do this in two steps:

      ssh kb-test-adm-001

      bash bitarchive_testdata/upload.sh TEST3A arcfiles

      It works like a charm

  2. The remote ssh commands like this "ssh kb-test-acs-001 "sed -i '/2-metadata/s/$/foobar/' ${TESTX}/CS/checksum_CS.md5" don't work in the current DK test-environment unless $TESTX is replaced with its value (TEST3A)