Uploaded image for project: 'NetarchiveSuite'
  1. NetarchiveSuite
  2. NAS-1356

dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.cacheData assumes wrongly that all indices are compressed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Minor
    • None
    • 3.8
    • Archive
    • None

    Description

      When requesting for uncompressed indices using the new CreateIndex tool
      I get the error seen below. It seems that it assumes that all indices are compressed during transmission.
      The createIndex currently only works for indices of type dedup:
      java -Dsettings.common.cacheDir=/tmp/cache -Dsettings.common.environmentName=QUICKSTART -Dsettings.common.remoteFile.port=5000 dk.netarkivet.archive.tools.CreateIndex -t cdx -l 1
      May 19 2009 5:54:56 PM dk.netarkivet.archive.indexserver.FileBasedCache <init>
      INFO: Metadata cache for 'CDX' uses directory '/tmp/cache/CDX'
      Creating CDX index for ids: [1]
      May 19 2009 5:54:56 PM dk.netarkivet.archive.indexserver.distribute.IndexRequestClient cacheData
      INFO: Requesting an index of type 'CDX' for the jobs [1]
      May 19 2009 5:54:56 PM dk.netarkivet.common.utils.Settings getAll
      WARNING: The list of loaded data settings is empty.Is this OK?
      May 19 2009 5:54:56 PM dk.netarkivet.common.distribute.JMSConnectionSunMQ <init>
      INFO: Creating instance of dk.netarkivet.common.distribute.JMSConnectionSunMQ
      May 19 2009 5:54:56 PM dk.netarkivet.common.distribute.JMSConnection initConnection
      INFO: Initializing a JMS connection of type 'class dk.netarkivet.common.distribute.JMSConnectionSunMQ' to Broker at localhost:7676.
      May 19 2009 5:54:57 PM org.slf4j.impl.JCLLoggerAdapter info
      INFO: Logging to org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
      May 19 2009 5:54:57 PM org.slf4j.impl.JCLLoggerAdapter info
      INFO: jetty-6.1.6
      May 19 2009 5:54:57 PM org.slf4j.impl.JCLLoggerAdapter info
      INFO: Started SocketConnector@0.0.0.0:5000
      May 19 2009 5:54:57 PM dk.netarkivet.archive.indexserver.distribute.IndexRequestClient cacheData
      WARNING: IOFailure during unzipping of index
      dk.netarkivet.common.exceptions.IOFailure: Error ungzipping './tests/commontempdir/remotefile-unzip16976.gz'
      at dk.netarkivet.common.utils.ZipUtils.gunzipFile(ZipUtils.java:300)
      at dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.unzipAndDeleteRemoteFile(IndexRequestClient.java:258)
      at dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.cacheData(IndexRequestClient.java:194)
      at dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.cacheData(IndexRequestClient.java:59)
      at dk.netarkivet.archive.indexserver.FileBasedCache.cache(FileBasedCache.java:164)
      at dk.netarkivet.archive.indexserver.FileBasedCache.getIndex(FileBasedCache.java:229)
      at dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.getIndex(IndexRequestClient.java:59)
      at dk.netarkivet.archive.tools.CreateIndex.main(CreateIndex.java:114)
      Caused by: java.io.IOException: Not in GZIP format
      at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:137)
      at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
      at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:68)
      at dk.netarkivet.common.utils.LargeFileGZIPInputStream.<init>(LargeFileGZIPInputStream.java:56)
      at dk.netarkivet.common.utils.ZipUtils.gunzipFile(ZipUtils.java:296)
      ... 7 more
      May 19 2009 5:54:57 PM dk.netarkivet.archive.indexserver.FileBasedCache getIndex
      INFO: Requested index of type 'CDX' data '[1]' not available. Retrying with available subset '[]'
      May 19 2009 5:54:57 PM dk.netarkivet.archive.indexserver.distribute.IndexRequestClient cacheData
      INFO: Requesting an index of type 'CDX' for the jobs []
      May 19 2009 5:54:57 PM dk.netarkivet.archive.indexserver.distribute.IndexRequestClient cacheData
      WARNING: IOFailure during unzipping of index
      dk.netarkivet.common.exceptions.IOFailure: Error ungzipping './tests/commontempdir/remotefile-unzip16977.gz'
      at dk.netarkivet.common.utils.ZipUtils.gunzipFile(ZipUtils.java:300)
      at dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.unzipAndDeleteRemoteFile(IndexRequestClient.java:258)
      at dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.cacheData(IndexRequestClient.java:194)
      at dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.cacheData(IndexRequestClient.java:59)
      at dk.netarkivet.archive.indexserver.FileBasedCache.cache(FileBasedCache.java:164)
      at dk.netarkivet.archive.indexserver.FileBasedCache.getIndex(FileBasedCache.java:229)
      at dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.getIndex(IndexRequestClient.java:59)
      at dk.netarkivet.archive.tools.CreateIndex.main(CreateIndex.java:114)
      Caused by: java.io.EOFException
      at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:207)
      at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:197)
      at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:136)
      at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
      at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:68)
      at dk.netarkivet.common.utils.LargeFileGZIPInputStream.<init>(LargeFileGZIPInputStream.java:56)
      at dk.netarkivet.common.utils.ZipUtils.gunzipFile(ZipUtils.java:296)
      ... 7 more

      Attachments

        Activity

          People

            Unassigned Unassigned
            svc Søren Vejrup Carlsen (Inactive)
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated: