dk.netarkivet.harvester.scheduler
Class HarvestDispatcher

java.lang.Object
  dk.netarkivet.harvester.distribute.HarvesterMessageHandler
      dk.netarkivet.harvester.scheduler.HarvestDispatcher

All Implemented Interfaces:

ComponentLifeCycle, HarvesterMessageVisitor, javax.jms.MessageListener

public class HarvestDispatcher
extends HarvesterMessageHandler
implements javax.jms.MessageListener, ComponentLifeCycle

This class handles dispatching of scheduled Harvest jobs to the Harvest servers

The scheduler loads all active harvest definitions on a regular basis and extracts the scheduling information for each definition. When a harvest definition is scheduled to start the scheduler creates the corresponding harvest jobs and submits these to the active HarvestServers.

The dispatching of jobs occurs when the internal state (built by processing HarvesterStatusMessages) tracksq at least one available harvester for a given priority. A job of the same priority is then issued for each available harvester.

Note: Only one HarvestDispatcher should be running at a time.

Field Summary

protected static org.apache.commons.logging.Log

log The logger to use.

Constructor Summary

HarvestDispatcher()
Create new instance of the HarvestDispatcher.

Method Summary

(package private) void	dispatchJobs() Dispatches new jobs.
void	doOneCrawl(Job job, java.lang.String origHarvestName, java.lang.String origHarvestDesc, java.lang.String origHarvestSchedule, java.util.List<MetadataEntry> metadata) Submit an doOneCrawl request to a HarvestControllerServer with correct priority.
void	shutdown() Release allocated resources (JMS connections) and stops dispatching harvest jobs, all without logging.
void	start() Start the thread responsible for reading Harvest definitions from the database, and dispatching the harvest job to the servers.
(package private) void	submitNewJobs() For each JobPriority, submits as many jobs as the internal state records available harvesters.
void	visit(HarvesterStatusMessage msg) This method should be overridden and implemented by a sub class if message handling is wanted.

Methods inherited from class dk.netarkivet.harvester.distribute.HarvesterMessageHandler

onMessage, visit, visit, visit, visit, visit, visit

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface javax.jms.MessageListener

onMessage

Field Detail

log

protected static final org.apache.commons.logging.Log log

The logger to use.

Constructor Detail

HarvestDispatcher

public HarvestDispatcher()

Create new instance of the HarvestDispatcher.

Method Detail

start

public void start()

Start the thread responsible for reading Harvest definitions from the database, and dispatching the harvest job to the servers.

Specified by:

start in interface ComponentLifeCycle

dispatchJobs

void dispatchJobs()

Dispatches new jobs. First stops jobs with status STARTED, which have been on running for more than settings.harvester.scheduler.jobtimeouttime time. Then submits new jobs if crawlers are free.

submitNewJobs

void submitNewJobs()

For each JobPriority, submits as many jobs as the internal state records available harvesters.

doOneCrawl

public void doOneCrawl(Job job,
                       java.lang.String origHarvestName,
                       java.lang.String origHarvestDesc,
                       java.lang.String origHarvestSchedule,
                       java.util.List<MetadataEntry> metadata)
                throws ArgumentNotValid,
                       IOFailure

Submit an doOneCrawl request to a HarvestControllerServer with correct priority.

Parameters:

job - the specific job to send

origHarvestName - the harvest definition's name

origHarvestDesc - the harvest definition's description

origHarvestSchedule - the harvest definition schedule name

metadata - pre-harvest metadata to store in arcfile.

Throws:

ArgumentNotValid - one of the parameters are null

IOFailure - if unable to send the doOneCrawl request to a harvestControllerServer

shutdown

public void shutdown()

Release allocated resources (JMS connections) and stops dispatching harvest jobs, all without logging.

Specified by:

shutdown in interface ComponentLifeCycle

visit

public void visit(HarvesterStatusMessage msg)

Description copied from class: HarvesterMessageHandler

This method should be overridden and implemented by a sub class if message handling is wanted.

Specified by:

visit in interface HarvesterMessageVisitor

Overrides:

visit in class HarvesterMessageHandler

Parameters:

msg - a HarvesterStatusMessage