Uploaded image for project: 'NetarchiveSuite'
  1. NetarchiveSuite
  2. NAS-2290

Generate revisit records

    XMLWordPrintable

Details

    • New Feature
    • Resolution: Fixed
    • Major
    • 5.2
    • 5.0-Milestone1
    • H3-extensions, WARC
    • None
    • BNF
    • NAS 5.2
    • Hide

      Verify by making standard netarkivet.dk selective harvest using the hourly schedule. Deduplication must be enabled both in the template, and in the NAS instance.
      Activate and make it run twice. Verify, that revisit-records are present in the second harvest

      Show
      Verify by making standard netarkivet.dk selective harvest using the hourly schedule. Deduplication must be enabled both in the template, and in the NAS instance. Activate and make it run twice. Verify, that revisit-records are present in the second harvest

    Description

      Currently, neither one of the two WARC Writers (the default WARCArchiver from Heritrix org.archive.crawler.writer.WARCWriterProcessor or the one from NAS
      dk.netarkivet.harvester.harvesting.WARCWriterProcessor)
      is producing WARC revisit records for duplicates although they are identified in the crawl.log

      According to Søren it is not the WarcWriter which needs to be changed to
      enable generation of revisit records, but the deduplication module. The
      estimated effort to do this is 1-2 weeks.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. NAS-2521 Support WARC Revisit records

          • Major - Major loss of function.
          • Closed
          spawned

          Improvement - An improvement or enhancement to an existing feature or task. NAS-2547 Create WARC 1.1 records instead of WARC 1.0 records

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              svc Søren Vejrup Carlsen (Inactive)
              sara Sara Aubry
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 55m
                  55m