dk.netarkivet.common.utils.warc
Class WARCUtils

java.lang.Object
  extended by dk.netarkivet.common.utils.warc.WARCUtils

public class WARCUtils
extends java.lang.Object

Various utilities on WARC-records. We have borrowed code from wayback. See org.archive.wayback.resourcestore.indexer.WARCRecordToSearchResultAdapter


Field Summary
protected static org.apache.commons.logging.Log log
          Logging output place.
 
Constructor Summary
WARCUtils()
           
 
Method Summary
static org.archive.io.warc.WARCWriter createWARCWriter(java.io.File newFile)
          Create new WARCWriter, writing to warcfile newFile.
static java.lang.String getRecordType(org.archive.io.warc.WARCRecord record)
          Find out what type of WARC-record this is.
static void insertWARCFile(java.io.File warcFile, org.archive.io.warc.WARCWriter writer)
          Insert the contents of a WARC file into another WARCFile.
static boolean isWarc(java.lang.String filename)
          Check if the given filename represents a WARC file.
static byte[] readWARCRecord(org.archive.io.warc.WARCRecord record)
          Read the contents (payload) of an WARC record into a byte array.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

log

protected static final org.apache.commons.logging.Log log
Logging output place.

Constructor Detail

WARCUtils

public WARCUtils()
Method Detail

createWARCWriter

public static org.archive.io.warc.WARCWriter createWARCWriter(java.io.File newFile)
Create new WARCWriter, writing to warcfile newFile.

Parameters:
newFile - the WARCfile, that the WARCWriter writes to.
Returns:
new WARCWriter, writing to warcfile newFile.

insertWARCFile

public static void insertWARCFile(java.io.File warcFile,
                                  org.archive.io.warc.WARCWriter writer)
Insert the contents of a WARC file into another WARCFile.

Parameters:
warcFile - An WARC file to read.
writer - A place to write the arc records
Throws:
IOFailure - if there are problems reading the file.

readWARCRecord

public static byte[] readWARCRecord(org.archive.io.warc.WARCRecord record)
                             throws IOFailure
Read the contents (payload) of an WARC record into a byte array.

Parameters:
record - An WARC record to read from. After reading, the WARC Record will no longer have its own data available for reading.
Returns:
A byte array containing the payload of the WARC record. Note that the size of the payload is calculated by subtracting the contentBegin value from the length of the record (both values included in the record header).
Throws:
IOFailure - If there is an error reading the data, or if the record is longer than Integer.MAX_VALUE (since we can't make bigger arrays).

getRecordType

public static java.lang.String getRecordType(org.archive.io.warc.WARCRecord record)
Find out what type of WARC-record this is.

Parameters:
record - a given WARCRecord
Returns:
the type of WARCRecord as a String.

isWarc

public static boolean isWarc(java.lang.String filename)
Check if the given filename represents a WARC file.

Parameters:
filename - A given filename
Returns:
true, if the filename ends with .warc or .warc.gz