Package dk.netarkivet.wayback.aggregator

The Aggregator takes care of sorting the raw index files generated by the indexer and merge the files into larger index files usable by Wayback.

See:
Description

Class Summary

AggregationWorker	The AggregationWorker singleton contains the schedule and file bookkeeping functionality needed in the aggregation of indexes.
AggregatorApplication	This wrapper class is used to start the AggregationWorker inside Jetty.
IndexAggregator	Encapsulates the functionality for sorting and merging index files.

Package dk.netarkivet.wayback.aggregator Description

The Aggregator takes care of sorting the raw index files generated by the indexer and merge the files into larger index files usable by Wayback. Once every WaybackSettings.WAYBACK_AGGREGATOR_AGGREGATION_INTERVAL an index aggregation is run. An Aggregation process consists of:

• All new index files found in the WaybackSettings.WAYBACK_AGGREGATOR_INPUT_DIR are sorted and merged into a temp intermediate index file.
• The temp intermediate index file is merged into the working intermediate index
• If the Intermediate Index file size exceeds WaybackSettings.WAYBACK_AGGREGATOR_MAX_INTERMEDIATE_INDEX_FILE_SIZE the following sequence occures
  • The currently active wayback index file is checked to see if it can contain indexes from the new intermediate file, without reaching the WaybackSettings.WAYBACK_AGGREGATOR_MAX_MAIN_INDEX_FILE_SIZE limit
  • If the limit isn't reach the intermediate index file is merged into the active index file ('wayback.index')
  • Else a new wayback main index file is created, this is set to be the active index file, and the intermediate indexes are merged to this file. The older main index files are name wayback.index.1, wayback.index.2 ..., where the number indicates sequential older index files.
• The original unsorted index files are deleted