Package dk.netarkivet.common.utils.warc

Class WARCUtils

  • public class WARCUtils
extends java.lang.Object
    Various utilities on WARC-records. We have borrowed code from wayback. See org.archive.wayback.resourcestore.indexer.WARCRecordToSearchResultAdapter

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected static org.slf4j.Logger log
      Logging output place.

    • Constructor Summary

      Constructors 
      Constructor Description
      WARCUtils()  

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static org.archive.io.warc.WARCWriter createWARCWriter​(java.io.File newFile)
      Create new WARCWriter, writing to warcfile newFile.
      static java.lang.String getRecordType​(org.archive.io.warc.WARCRecord record)
      Find out what type of WARC-record this is.
      static void insertWARCFile​(java.io.File warcFile, org.archive.io.warc.WARCWriter writer)
      Insert the contents of a WARC file into another WARCFile.
      static boolean isWarc​(java.lang.String filename)
      Check if the given filename represents a WARC file.
      static byte[] readWARCRecord​(org.archive.io.warc.WARCRecord record)
      Read the contents (payload) of an WARC record into a byte array.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • log

        protected static final org.slf4j.Logger log
        Logging output place.

    • Method Detail

      • createWARCWriter

        public static org.archive.io.warc.WARCWriter createWARCWriter​(java.io.File newFile)
        Create new WARCWriter, writing to warcfile newFile.
        Parameters:
        newFile - the WARCfile, that the WARCWriter writes to.
        Returns:
        new WARCWriter, writing to warcfile newFile.

      • insertWARCFile

        public static void insertWARCFile​(java.io.File warcFile,
                                  org.archive.io.warc.WARCWriter writer)
        Insert the contents of a WARC file into another WARCFile.
        Parameters:
        warcFile - An WARC file to read.
        writer - A place to write the arc records
        Throws:
        IOFailure - if there are problems reading the file.

      • readWARCRecord

        public static byte[] readWARCRecord​(org.archive.io.warc.WARCRecord record)
                             throws IOFailure
        Read the contents (payload) of an WARC record into a byte array.
        Parameters:
        record - An WARC record to read from. After reading, the WARC Record will no longer have its own data available for reading.
        Returns:
        A byte array containing the payload of the WARC record. Note that the size of the payload is calculated by subtracting the contentBegin value from the length of the record (both values included in the record header).
        Throws:
        IOFailure - If there is an error reading the data, or if the record is longer than Integer.MAX_VALUE (since we can't make bigger arrays).

      • getRecordType

        public static java.lang.String getRecordType​(org.archive.io.warc.WARCRecord record)
        Find out what type of WARC-record this is.
        Parameters:
        record - a given WARCRecord
        Returns:
        the type of WARCRecord as a String.

      • isWarc

        public static boolean isWarc​(java.lang.String filename)
        Check if the given filename represents a WARC file.
        Parameters:
        filename - A given filename
        Returns:
        true, if the filename ends with .warc or .warc.gz