Package dk.netarkivet.harvester.harvesting

This module handles defining, scheduling, and execution of harvests.

See:
Description

Class Summary

ArcFilesReportGenerator	This class generate a report that lists ARC files along with the opening date, closing date (if file was properly closed), and size in bytes.
ArcFilesReportGenerator.ArcFileStatus	Stores the opening date, closing date and size of an ARC file.
ContentSizeAnnotationPostProcessor	A post processor that adds an annotation content-size: for each succesfully harvested URI.
DomainnameQueueAssignmentPolicy	Using the domain as the queue-name.
HarvestController	This class handles all the things in a single harvest that are not related directly related either to launching Heritrix or to handling JMS messages.
HarvestControllerApplication	This application controls the Heritrix harvester which does the actual harvesting, and is also responsible for uploading the harvested data to the ArcRepository.
HarvestDocumentation	This class contains code for documenting a harvest.
HeritrixFiles	This class encapsulates all the files that Heritrix gets from our system, and all files we read from Heritrix.
HeritrixLauncher	A HeritrixLauncher object wraps around an instance of the web crawler Heritrix.
HeritrixLauncherFactory	Factory class for instanciating a specific implementation of HeritrixLauncher.
IngestableFiles	Encapsulation of files to be ingested into the archive.
MetadataFile	Wraps information for an Heritrix file that should be stored in the metadata ARC.
OnNSDomainsDecideRule	Class that re-creates the SurtPrefixSet to include only domain names according to the domain definition of NetarchiveSuite.

Package dk.netarkivet.harvester.harvesting Description

This module handles defining, scheduling, and execution of harvests.

• Harvesting uses the Heritrix crawler developed by Internet Archive. The harvesting module allows for flexible automated definitions of harvests. The system gives access to the full power of the Heritrix crawler, given adequate knowledge of the Heritrix crawler. NetarchiveSuite wraps the crawler in an easy-to-use interface that handles scheduling and configuring of the crawls, and distributes it to several crawling servers.
• The harvester module allows for de-duplication, using an index of URLs already crawled and stored in the archive to avoid storing duplicates more than once. This function uses the de-duplicator module from Kristinn Sigurdsson.
• The harvester module supports packaging metadata about the harvest together with the harvested data.