dk.netarkivet.harvester.indexserver
Class CrawlLogIndexCache

java.lang.Object
  extended by dk.netarkivet.harvester.indexserver.FileBasedCache<java.util.Set<T>>
      extended by dk.netarkivet.harvester.indexserver.MultiFileBasedCache<T>
          extended by dk.netarkivet.harvester.indexserver.CombiningMultiFileBasedCache<java.lang.Long>
              extended by dk.netarkivet.harvester.indexserver.CrawlLogIndexCache
All Implemented Interfaces:
JobIndexCache
Direct Known Subclasses:
DedupCrawlLogIndexCache, FullCrawlLogIndexCache

public abstract class CrawlLogIndexCache
extends CombiningMultiFileBasedCache<java.lang.Long>
implements JobIndexCache

A cache that serves Lucene indices of crawl logs for given job IDs. Uses the DigestIndexer in the deduplicator software: http://deduplicator.sourceforge.net/apidocs/is/hi/bok/deduplicator/DigestIndexer.html Upon combination of underlying files, each file in the Lucene index is gzipped and the compressed versions are stored in the directory given by getCacheFile(). The subclass has to determine in its constructor call which mime types are included.


Field Summary
 
Fields inherited from class dk.netarkivet.harvester.indexserver.CombiningMultiFileBasedCache
rawcache
 
Fields inherited from class dk.netarkivet.harvester.indexserver.FileBasedCache
cacheDir
 
Constructor Summary
CrawlLogIndexCache(java.lang.String name, boolean blacklist, java.lang.String mimeFilter)
          Constructor for the CrawlLogIndexCache class.
 
Method Summary
protected  void combine(java.util.Map<java.lang.Long,java.io.File> rawfiles)
          Combine a number of crawl.log files into one Lucene index.
protected static is.hi.bok.deduplicator.DigestIndexer createStandardIndexer(java.lang.String indexLocation)
          Create standard deduplication indexer.
protected static java.io.File getSortedCDX(java.io.File cdxFile)
          Get a sorted, temporary CDX file corresponding to the given CDXfile.
protected static java.io.File getSortedCrawlLog(java.io.File file)
          Get a sorted, temporary crawl.log file from an unsorted one.
protected static void indexFile(java.lang.Long id, java.io.File crawllogfile, java.io.File cdxfile, is.hi.bok.deduplicator.DigestIndexer indexer, DigestOptions options)
          Ingest a single crawl.log file using the corresponding CDX file to find offsets.
protected  java.util.Map<java.lang.Long,java.io.File> prepareCombine(java.util.Set<java.lang.Long> ids)
          Prepare data for combining.
 
Methods inherited from class dk.netarkivet.harvester.indexserver.CombiningMultiFileBasedCache
cacheData
 
Methods inherited from class dk.netarkivet.harvester.indexserver.MultiFileBasedCache
getCacheFile
 
Methods inherited from class dk.netarkivet.harvester.indexserver.FileBasedCache
cache, get, getCacheDir, getIndex
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface dk.netarkivet.common.distribute.indexserver.JobIndexCache
getIndex, requestIndex
 

Constructor Detail

CrawlLogIndexCache

public CrawlLogIndexCache(java.lang.String name,
                          boolean blacklist,
                          java.lang.String mimeFilter)
Constructor for the CrawlLogIndexCache class.

Parameters:
name - The name of the CrawlLogIndexCache
blacklist - Shall the mimefilter be considered a blacklist or a whitelist?
mimeFilter - A regular expression for the mimetypes to exclude/include
Method Detail

prepareCombine

protected java.util.Map<java.lang.Long,java.io.File> prepareCombine(java.util.Set<java.lang.Long> ids)
Prepare data for combining. This class overrides prepareCombine to make sure that CDX data is available.

Overrides:
prepareCombine in class CombiningMultiFileBasedCache<java.lang.Long>
Parameters:
ids - Set of IDs that will be combined.
Returns:
Map of ID->File of data to combine for the IDs where we could find data.

combine

protected void combine(java.util.Map<java.lang.Long,java.io.File> rawfiles)
Combine a number of crawl.log files into one Lucene index. This index is placed as gzip files under the directory returned by getCacheFile().

Specified by:
combine in class CombiningMultiFileBasedCache<java.lang.Long>
Parameters:
rawfiles - The map from job ID into crawl.log contents. No null values are allowed in this map.

indexFile

protected static void indexFile(java.lang.Long id,
                                java.io.File crawllogfile,
                                java.io.File cdxfile,
                                is.hi.bok.deduplicator.DigestIndexer indexer,
                                DigestOptions options)
Ingest a single crawl.log file using the corresponding CDX file to find offsets.

Parameters:
id - ID of a job to ingest.
crawllogfile - The file containing the crawl.log data for the job
cdxfile - The file containing the cdx data for the job
options - The digesting options used.
indexer - The indexer to add to.

getSortedCDX

protected static java.io.File getSortedCDX(java.io.File cdxFile)
Get a sorted, temporary CDX file corresponding to the given CDXfile.

Parameters:
cdxFile - A cdxfile
Returns:
A temporary file with CDX info for that just sorted according to the standard CDX sorting rules. This file will be removed at the exit of the JVM, but should be attempted removed when it is no longer used.

getSortedCrawlLog

protected static java.io.File getSortedCrawlLog(java.io.File file)
Get a sorted, temporary crawl.log file from an unsorted one.

Parameters:
file - The file containing an unsorted crawl.log file.
Returns:
A temporary file containing the entries sorted according to URL. The file will be removed upon exit of the JVM, but should be attempted removed when it is no longer used.

createStandardIndexer

protected static is.hi.bok.deduplicator.DigestIndexer createStandardIndexer(java.lang.String indexLocation)
                                                                     throws java.io.IOException
Create standard deduplication indexer.

Parameters:
indexLocation - The full path to the indexing directory
Returns:
the created deduplication indexer.
Throws:
java.io.IOException - If unable to open the index.