Package dk.netarkivet.harvester.scheduler.jobgen

Class FixedDomainConfigurationCountJobGenerator

java.lang.Object

dk.netarkivet.harvester.scheduler.jobgen.FixedDomainConfigurationCountJobGenerator

All Implemented Interfaces:

JobGenerator

public class FixedDomainConfigurationCountJobGenerator
extends java.lang.Object

Job generator implementation. Generates jobs with a fixed number of domain configurations. Configuration allows to choose a different count for partial and full harvests. The last job generated may have less configurations in it, as job generation happens on a per-harvest basis.

See Also:

HarvesterSettings.JOBGEN_FIXED_CONFIG_COUNT_SNAPSHOT, HarvesterSettings.JOBGEN_FIXED_CONFIG_COUNT_FOCUSED

Field Summary

Fields

Modifier and Type	Field	Description
static FixedDomainConfigurationCountJobGenerator	instance	The singleton instance.

Method Summary

Modifier and Type	Method	Description
boolean	canAccept(Job job, DomainConfiguration cfg, DomainConfiguration previousCfg)	Tests if a configuration fits into this Job.
protected boolean	checkSpecificAcceptConditions(Job job, DomainConfiguration cfg)	Called by JobGenerator.canAccept(Job, DomainConfiguration, DomainConfiguration).
protected void	editJobOrderXml(Job job)	Once the job has been filled with DomainConfigurations, performs the following operations: Edit the harvest template to add/remove deduplicator configuration.
int	generateJobs(HarvestDefinition harvest)	Generates a series of jobs for the given harvest definition.
protected java.util.Comparator<DomainConfiguration>	getDomainConfigurationSubsetComparator(HarvestDefinition harvest)	Returns a comparator used to sort the subset of DOMAIN_CONFIG_SUBSET_SIZE configurations that are scanned at each iteration.
static FixedDomainConfigurationCountJobGenerator	getInstance()
Job	getNewJob(HarvestDefinition harvest, DomainConfiguration cfg)	Instantiates a new job.
boolean	ignoreConfiguration(DomainConfiguration cfg)	Test if this configuration should be ignored
protected int	processDomainConfigurationSubset(HarvestDefinition harvest, java.util.Iterator<DomainConfiguration> domainConfSubset)	Create new jobs from a collection of configurations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

instance

public static FixedDomainConfigurationCountJobGenerator instance

The singleton instance.

Method Detail

getInstance

public static FixedDomainConfigurationCountJobGenerator getInstance()

Returns:

the singleton instance, builds it if necessary.

getDomainConfigurationSubsetComparator

protected java.util.Comparator<DomainConfiguration> getDomainConfigurationSubsetComparator(HarvestDefinition harvest)

Returns a comparator used to sort the subset of DOMAIN_CONFIG_SUBSET_SIZE configurations that are scanned at each iteration.

Parameters:

harvest - the HarvestDefinition being processed.

Returns:

a comparator

checkSpecificAcceptConditions

protected boolean checkSpecificAcceptConditions(Job job,
                                                DomainConfiguration cfg)

Called by JobGenerator.canAccept(Job, DomainConfiguration, DomainConfiguration). Tests the implementation-specific conditions to accept the given DomainConfiguration in the given Job. It is assumed that checkAddDomainConfInvariant(Job, DomainConfiguration, DomainConfiguration) has already passed.

Parameters:

job - the Job n=being built

cfg - the DomainConfiguration to test

Returns:

true if the configuration passes the conditions.

generateJobs

public int generateJobs(HarvestDefinition harvest)

Description copied from interface: JobGenerator

Generates a series of jobs for the given harvest definition. Note that a job generator is expected to follow the singleton pattern, so implementations of this method should be thread-safe.

Specified by:

generateJobs in interface JobGenerator

Parameters:

harvest - the harvest definition to process.

Returns:

the number of jobs that were generated.

processDomainConfigurationSubset

protected int processDomainConfigurationSubset(HarvestDefinition harvest,
                                               java.util.Iterator<DomainConfiguration> domainConfSubset)

Create new jobs from a collection of configurations. All configurations must use the same order.xml file.Jobs

Parameters:

harvest - the HarvestDefinition being processed.

domainConfSubset - the configurations to use to create the jobs

Returns:

The number of jobs created

getNewJob

public Job getNewJob(HarvestDefinition harvest,
                     DomainConfiguration cfg)

Instantiates a new job.

Parameters:

cfg - the DomainConfiguration being processed

harvest - the HarvestDefinition being processed

Returns:

an instance of Job

canAccept

public boolean canAccept(Job job,
                         DomainConfiguration cfg,
                         DomainConfiguration previousCfg)

Description copied from interface: JobGenerator

Tests if a configuration fits into this Job. First tests if it's the right type of order-template and bytelimit, and whether the bytelimit is right for the job. The Job limits are compared against the configuration estimates and if no limits are exceeded true is returned otherwise false is returned.

Specified by:

canAccept in interface JobGenerator

Parameters:

job - the job being built.

cfg - the configuration to check

previousCfg - if not null, the configuration added to this job immediately prior

Returns:

true if adding the configuration to this Job does not exceed any of the Job limits.

editJobOrderXml

protected void editJobOrderXml(Job job)

Once the job has been filled with DomainConfigurations, performs the following operations:

1. Edit the harvest template to add/remove deduplicator configuration.

Parameters:

job - the job

ignoreConfiguration

public boolean ignoreConfiguration(DomainConfiguration cfg)

Description copied from interface: JobGenerator

Test if this configuration should be ignored

Specified by:

ignoreConfiguration in interface JobGenerator

Parameters:

cfg - a domain configuration

Returns:

true if we should ignore this configuration (It could be that it is disabled in some way, or all seeds are prefixed with a '#' and so there are no active seeds