Child pages
  • Appendix A - How To Do Examples

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

Contents

Table of Contents

Install the QuickStart according to the Quick Start Manual, e.g. in /home/test/netarchive.

  • Add some domains to harvest using the the ADMGUI e.g. netarkivet.dk, kb.dk, statsbiblioteket.dk
  • Create and run a snapshot with a byte limit of 100.000
  • Wait until the job is done
  • Setup your browser for browsing and index your harvest job
    Code Block
    cd /home/test/netarchive/scripts/simple_harvest/bitarchive1/filedir
    export CLASSPATH=/home/test/netarchive/lib/dk.netarkivet.common.jar
    ls 
    e.g.

Arc Merge:

Code Block
export FILEONE=1-1-20090519073602-00000-dia-test-int-01.kb.dk.arc
export FILETWO=1-1-20090519073602-00001-dia-test-int-01.kb.dk.arc
java dk.netarkivet.common.tools.ArcMerge $FILEONE $FILETWO  > resulting.arc 

Extract CDX:

Code Block
export FILEONE=1-1-20090519083732-00002-dia-test-int-01.kb.dk.arc
java dk.netarkivet.common.tools.ExtractCDX $FILEONE > output.cdx 

Get Record using Lucene:

Code Block
#e.g. an URI from the harvest found in your "viewerproxy"
export URI=http://netarkivet.dk/index-da.php
cd /home/test/netarchive/scripts/simple_harvest/cache/fullcrawllogindex
cp -r 1-cache 1-cache.unzip
cd 1-cache.unzip/
ls
gunzip *
export SETTINGSFILE=/home/test/netarchive/scripts/simple_harvest/settings.xml
export LUCENE_INDEX=/home/test/netarchive/scripts/simple_harvest/cache/\
  fullcrawllogindex/1-cache.unzip
export OPTS=-Ddk.netarkivet.settings.file=$SETTINGSFILE \
      -Dsettings.common.remoteFile.port=5000 
java $OPTS dk.netarkivet.archive.tools.GetRecord $LUCENE_INDEX $URI

Upload:

Code Block
cd /home/test/netarchive/scripts/simple_harvest/
cp /home/test/netarchive/scripts/simple_harvest/bitarchive1/filedir/resulting.arc new_resulting.arc
export SETTINGSFILE=/home/test/netarchive/scripts/simple_harvest/settings.xml
export OPTS=-Ddk.netarkivet.settings.file=$SETTINGSFILE -Dsettings.common.remoteFile.port=5000
java $OPTS -cp /home/test/netarchive/lib/dk.netarkivet.archive.jar dk.netarkivet.archive.tools.Upload new_resulting.arc
#just press <CTRL-C> to stop the job

Batch e.g. with checksum:

Code Block
cd /home/test/netarchive
mkdir batchprogs

#copy example batchprog ChecksumJob.java to  batchprogs/.
cd batchprogs
javac ChecksumJob.java
export SETTINGSFILE=/home/test/netarchive/scripts/simple_harvest/settings.xml
java -cp lib/dk.netarkivet.archive.jar -Dsettings.common.remoteFile.port=5000 \
-Ddk.netarkivet.settings.file=$SETTINGSFILE dk.netarkivet.archive.tools.RunBatch \
-Cbatchprogs/ChecksumJob.class -Ooutput.checksum

ChecksumJob.java

Section



Column

Column
width100%

Column