Uploaded image for project: 'NetarchiveSuite'
  1. NetarchiveSuite
  2. NAS-2394

Replace CDX index rollover naming strategy

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 4.4.1, 5.0
    • None
    • Wayback
    • None
    • Hide

      The renaming condition is only checked after each aggregation operation, so to test it one must ensure that aggregation is done twice.

      1. Start a TEST12 instance as normal.
      2. Killall jobs on kb-test-way-001.
      3. Set the two maximum file sizes (maxIntermediateIndexFileSize and maxMainIndexFileSize) in the aggregator settings to "1". (ie 1Mb)
      4. Restart the applications
      5. Go to the step "Upload a Small Bitarchive" in TEST 12 but only upload arcfiles.
      6. Wait for the indexer and aggregator to run and create an index in TEST12/indexDir
      7. Now upload the warcfiles
      8. Wait for the indexer and aggregator to run again.

      There should now be a renamed index file with a timestamp in its name.

      Show
      The renaming condition is only checked after each aggregation operation, so to test it one must ensure that aggregation is done twice. Start a TEST12 instance as normal. Killall jobs on kb-test-way-001. Set the two maximum file sizes (maxIntermediateIndexFileSize and maxMainIndexFileSize) in the aggregator settings to "1". (ie 1Mb) Restart the applications Go to the step "Upload a Small Bitarchive" in TEST 12 but only upload arcfiles. Wait for the indexer and aggregator to run and create an index in TEST12/indexDir Now upload the warcfiles Wait for the indexer and aggregator to run again. There should now be a renamed index file with a timestamp in its name.

    Description

      The rollover naming strategy for cdx files makes backup and sync very difficult, so instead just give new index files a new unique name.

      Attachments

        Issue Links

          Activity

            People

              csr Colin Rosenthal
              csr Colin Rosenthal
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: