dk.netarkivet.harvester
Class HarvesterSettings

java.lang.Object
  extended by dk.netarkivet.harvester.HarvesterSettings

public class HarvesterSettings
extends java.lang.Object

Settings specific to the harvester module of NetarchiveSuite.


Field Summary
static java.lang.String CRAWLER_TIMEOUT_NON_RESPONDING
          settings.harvester.harvesting.heritrix.noresponseTimeout:
The timeout value (in seconds) used in HeritrixLauncher for aborting crawl when no bytes are being received from web servers.
static java.lang.String DEFAULT_SEEDLIST
          settings.harvester.datamodel.domain.defaultSeedlist:
Default name of the seedlist to use when new domains are created.
static java.lang.String DOMAIN_CONFIG_MAXBYTES
          settings.harvester.datamodel.domain.defaultMaxbytes:
Default byte limit for domain configuration.
static java.lang.String DOMAIN_CONFIG_MAXRATE
          settings.harvester.datamodel.domain.defaultMaxrate:
Default download rate for domain configuration.
static java.lang.String DOMAIN_DEFAULT_CONFIG
          settings.harvester.datamodel.domain.defaultConfig:
The name of a configuration that is created by default and which is initially used for snapshot harvests.
static java.lang.String DOMAIN_DEFAULT_ORDERXML
          settings.harvester.datamodel.domain.defaultOrderxml:
Name of order xml template used for domains if nothing else is specified.
static java.lang.String ERRORFACTOR_PERMITTED_BESTGUESS
          settings.harvester.scheduler.errorFactorBestGuess:
Used when calculating expected size of a harvest of some configuration during job-creation process.
static java.lang.String ERRORFACTOR_PERMITTED_PREVRESULT
          settings.harvester.scheduler.errorFactorPrevResult:
Used when calculating expected size of a harvest of some configuration during job-creation process.
static java.lang.String EXPECTED_AVERAGE_BYTES_PER_OBJECT
          settings.harvester.scheduler.expectedAverageBytesPerObject:
How many bytes the average object is expected to be on domains where we don't know any better.
static java.lang.String HARVEST_CONTROLLER_OLDJOBSDIR
          settings.harvester.harvesting.oldjobsDir:
The directory in which data from old jobs is kept after uploading.
static java.lang.String HARVEST_CONTROLLER_PRIORITY
          settings.harvester.harvesting.queuePriority:
Pool to take jobs from.
static java.lang.String HARVEST_CONTROLLER_SERVERDIR
          settings.harvester.harvesting.serverDir:
Each job gets a subdir of this dir.
static java.lang.String HARVEST_SERVERDIR_MINSPACE
          settings.harvester.harvesting.minSpaceLeft:
The minimum amount of free bytes in the serverDir required before accepting any harvest-jobs.
static java.lang.String HERITRIX_ADMIN_NAME
          settings.harvester.harvesting.heritrix.adminName:
The name used to access the Heritrix GUI.
static java.lang.String HERITRIX_ADMIN_PASSWORD
          settings.harvester.harvesting.heritrix.adminPassword:
The password used to access the Heritrix GUI.
static java.lang.String HERITRIX_CONTROLLER_CLASS
          settings.harvester.harvesting.heritrixControllerClass:
The implementation of the HeritrixController interface to be used.
static java.lang.String HERITRIX_GUI_PORT
          settings.harvester.harvesting.heritrix.guiPort:
Port used to access the Heritrix web user interface.
static java.lang.String HERITRIX_HEAP_SIZE
          settings.harvester.harvesting.heritrix.heapSize:
The heap size to use for the Heritrix sub-process.
static java.lang.String HERITRIX_JMX_PASSWORD
          settings.harvester.harvesting.heritrix.jmxPassword:
The password used to connect to Heritrix JMX interface The password must correspond to the value stored in the jmxremote.password file (name defined in setting settings.common.jmx.passwordFile).
static java.lang.String HERITRIX_JMX_PORT
          settings.harvester.harvesting.heritrix.jmxPort:
The port that Heritrix uses to expose its JMX interface.
static java.lang.String HERITRIX_JMX_USERNAME
          settings.harvester.harvesting.heritrix.jmxUsername:
The username used to connect to Heritrix JMX interface The username must correspond to the value stored in the jmxremote.password file (name defined in setting settings.common.jmx.passwordFile).
static java.lang.String HERITRIX_JVM_OPTS
          settings.harvester.harvesting.heritrix.javaOpts:
Additional JVM options for the Heritrix sub-process.
static java.lang.String INACTIVITY_TIMEOUT_IN_SECS
          settings.harvester.harvesting.heritrix.inactivityTimeout:
The timeout setting for aborting a crawl based on crawler-inactivity.
static java.lang.String JOB_TIMEOUT_TIME
          settings.harvester.scheduler.jobtimeouttime:
Time before a STARTED job times out and change status to FAILED.
static java.lang.String JOBS_MAX_RELATIVE_SIZE_DIFFERENCE
          settings.harvester.scheduler.jobs.maxRelativeSizeDifference:
The maximum allowed relative difference in expected number of objects retrieved in a single job definition.
static java.lang.String JOBS_MAX_TOTAL_JOBSIZE
          settings.harvester.scheduler.jobs.maxTotalSize:
When this limit is exceeded no more configurations may be added to a job.
static java.lang.String JOBS_MIN_ABSOLUTE_SIZE_DIFFERENCE
          settings.harvester.scheduler.jobs.minAbsoluteSizeDifference:
Size differences for jobs below this threshold are ignored, regardless of the limits for the relative size difference.
static java.lang.String MAX_CONFIGS_PER_JOB_CREATION
          settings.harvester.scheduler.configChunkSize:
How many domain configurations we will process in one go before making jobs out of them.
static java.lang.String MAX_DOMAIN_SIZE
          settings.harvester.scheduler.maxDomainSize:
The initial guess of the domain size (number of objects) of an unknown domain.
 
Constructor Summary
HarvesterSettings()
           
 
Method Summary
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

DEFAULT_SEEDLIST

public static java.lang.String DEFAULT_SEEDLIST
settings.harvester.datamodel.domain.defaultSeedlist:
Default name of the seedlist to use when new domains are created.


DOMAIN_DEFAULT_CONFIG

public static java.lang.String DOMAIN_DEFAULT_CONFIG
settings.harvester.datamodel.domain.defaultConfig:
The name of a configuration that is created by default and which is initially used for snapshot harvests.


DOMAIN_DEFAULT_ORDERXML

public static java.lang.String DOMAIN_DEFAULT_ORDERXML
settings.harvester.datamodel.domain.defaultOrderxml:
Name of order xml template used for domains if nothing else is specified. The newly created configurations use this. This template must exist before harvesting can commence


DOMAIN_CONFIG_MAXRATE

public static java.lang.String DOMAIN_CONFIG_MAXRATE
settings.harvester.datamodel.domain.defaultMaxrate:
Default download rate for domain configuration. Not currently enforced.


DOMAIN_CONFIG_MAXBYTES

public static java.lang.String DOMAIN_CONFIG_MAXBYTES
settings.harvester.datamodel.domain.defaultMaxbytes:
Default byte limit for domain configuration.


ERRORFACTOR_PERMITTED_PREVRESULT

public static java.lang.String ERRORFACTOR_PERMITTED_PREVRESULT
settings.harvester.scheduler.errorFactorPrevResult:
Used when calculating expected size of a harvest of some configuration during job-creation process. This defines how great a possible factor we will permit a harvest to be larger then the expectation, when basing the expectation on a previous completed job.


ERRORFACTOR_PERMITTED_BESTGUESS

public static java.lang.String ERRORFACTOR_PERMITTED_BESTGUESS
settings.harvester.scheduler.errorFactorBestGuess:
Used when calculating expected size of a harvest of some configuration during job-creation process. This defines how great a possible factor we will permit a harvest to be larger then the expectation, when basing the expectation on previous uncompleted harvests or no harvest data at all.


EXPECTED_AVERAGE_BYTES_PER_OBJECT

public static java.lang.String EXPECTED_AVERAGE_BYTES_PER_OBJECT
settings.harvester.scheduler.expectedAverageBytesPerObject:
How many bytes the average object is expected to be on domains where we don't know any better. This number should grow over time, as of end of 2005 empirical data shows 38000.


MAX_DOMAIN_SIZE

public static java.lang.String MAX_DOMAIN_SIZE
settings.harvester.scheduler.maxDomainSize:
The initial guess of the domain size (number of objects) of an unknown domain.


JOBS_MAX_RELATIVE_SIZE_DIFFERENCE

public static java.lang.String JOBS_MAX_RELATIVE_SIZE_DIFFERENCE
settings.harvester.scheduler.jobs.maxRelativeSizeDifference:
The maximum allowed relative difference in expected number of objects retrieved in a single job definition. To avoid job splitting, set the value as Long.MAX_VALUE.


JOBS_MIN_ABSOLUTE_SIZE_DIFFERENCE

public static java.lang.String JOBS_MIN_ABSOLUTE_SIZE_DIFFERENCE
settings.harvester.scheduler.jobs.minAbsoluteSizeDifference:
Size differences for jobs below this threshold are ignored, regardless of the limits for the relative size difference. To avoid job splitting, set the value as Long.MAX_VALUE.


JOBS_MAX_TOTAL_JOBSIZE

public static java.lang.String JOBS_MAX_TOTAL_JOBSIZE
settings.harvester.scheduler.jobs.maxTotalSize:
When this limit is exceeded no more configurations may be added to a job. To avoid job splitting, set the value as Long.MAX_VALUE.


MAX_CONFIGS_PER_JOB_CREATION

public static java.lang.String MAX_CONFIGS_PER_JOB_CREATION
settings.harvester.scheduler.configChunkSize:
How many domain configurations we will process in one go before making jobs out of them. This amount of domains will be stored in memory at the same time. To avoid job splitting, set this value as Long.MAX_VALUE.


JOB_TIMEOUT_TIME

public static java.lang.String JOB_TIMEOUT_TIME
settings.harvester.scheduler.jobtimeouttime:
Time before a STARTED job times out and change status to FAILED. In seconds.


HARVEST_CONTROLLER_SERVERDIR

public static java.lang.String HARVEST_CONTROLLER_SERVERDIR
settings.harvester.harvesting.serverDir:
Each job gets a subdir of this dir. Job data is written and Heritrix writes to that subdir.


HARVEST_SERVERDIR_MINSPACE

public static java.lang.String HARVEST_SERVERDIR_MINSPACE
settings.harvester.harvesting.minSpaceLeft:
The minimum amount of free bytes in the serverDir required before accepting any harvest-jobs.


HARVEST_CONTROLLER_OLDJOBSDIR

public static java.lang.String HARVEST_CONTROLLER_OLDJOBSDIR
settings.harvester.harvesting.oldjobsDir:
The directory in which data from old jobs is kept after uploading. Each directory from serverDir will be moved to here if any data remains, either due to failed uploads or because it wasn't attempted uploaded.


HARVEST_CONTROLLER_PRIORITY

public static java.lang.String HARVEST_CONTROLLER_PRIORITY
settings.harvester.harvesting.queuePriority:
Pool to take jobs from. There are two pools to choose from, labelled "HIGHPRIORITY" (pool for selective harvest jobs), and "LOWPRIORITY" (pool for snapshot harvest jobs) respectively. NOTE: this one is also used in SingleMBeanObject parsing information to System state


INACTIVITY_TIMEOUT_IN_SECS

public static java.lang.String INACTIVITY_TIMEOUT_IN_SECS
settings.harvester.harvesting.heritrix.inactivityTimeout:
The timeout setting for aborting a crawl based on crawler-inactivity. If the crawler is inactive for this amount of seconds the crawl will be aborted. The inactivity is measured on the crawlController.activeToeCount().


CRAWLER_TIMEOUT_NON_RESPONDING

public static java.lang.String CRAWLER_TIMEOUT_NON_RESPONDING
settings.harvester.harvesting.heritrix.noresponseTimeout:
The timeout value (in seconds) used in HeritrixLauncher for aborting crawl when no bytes are being received from web servers.


HERITRIX_ADMIN_NAME

public static java.lang.String HERITRIX_ADMIN_NAME
settings.harvester.harvesting.heritrix.adminName:
The name used to access the Heritrix GUI.


HERITRIX_ADMIN_PASSWORD

public static java.lang.String HERITRIX_ADMIN_PASSWORD
settings.harvester.harvesting.heritrix.adminPassword:
The password used to access the Heritrix GUI.


HERITRIX_GUI_PORT

public static java.lang.String HERITRIX_GUI_PORT
settings.harvester.harvesting.heritrix.guiPort:
Port used to access the Heritrix web user interface. This port must not be used by anything else on the machine. Note that apart from pausing a job, modifications done directly on Heritrix may cause unexpected breakage.


HERITRIX_JMX_PORT

public static java.lang.String HERITRIX_JMX_PORT
settings.harvester.harvesting.heritrix.jmxPort:
The port that Heritrix uses to expose its JMX interface. This port must not be used by anything else on the machine, but does not need to be accessible from other machines unless you want to be able to use jconsole to access Heritrix directly. Note that apart from pausing a job, modifications done directly on Heritrix may cause unexpected breakage.


HERITRIX_JMX_USERNAME

public static java.lang.String HERITRIX_JMX_USERNAME
settings.harvester.harvesting.heritrix.jmxUsername:
The username used to connect to Heritrix JMX interface The username must correspond to the value stored in the jmxremote.password file (name defined in setting settings.common.jmx.passwordFile).


HERITRIX_JMX_PASSWORD

public static java.lang.String HERITRIX_JMX_PASSWORD
settings.harvester.harvesting.heritrix.jmxPassword:
The password used to connect to Heritrix JMX interface The password must correspond to the value stored in the jmxremote.password file (name defined in setting settings.common.jmx.passwordFile).


HERITRIX_HEAP_SIZE

public static java.lang.String HERITRIX_HEAP_SIZE
settings.harvester.harvesting.heritrix.heapSize:
The heap size to use for the Heritrix sub-process. This should probably be fairly large. It can be specified in the same way as for the -Xmx argument to Java, e.g. 512M, 2G etc.


HERITRIX_JVM_OPTS

public static java.lang.String HERITRIX_JVM_OPTS
settings.harvester.harvesting.heritrix.javaOpts:
Additional JVM options for the Heritrix sub-process. By default there is no additional JVM option.


HERITRIX_CONTROLLER_CLASS

public static java.lang.String HERITRIX_CONTROLLER_CLASS
settings.harvester.harvesting.heritrixControllerClass:
The implementation of the HeritrixController interface to be used.

Constructor Detail

HarvesterSettings

public HarvesterSettings()