dk.netarkivet.harvester.harvesting
Class JMXHeritrixController

java.lang.Object
  extended by dk.netarkivet.harvester.harvesting.JMXHeritrixController
All Implemented Interfaces:
HeritrixController

public class JMXHeritrixController
extends java.lang.Object
implements HeritrixController

This implementation of the HeritrixController interface starts Heritrix as a separate process and uses JMX to communicate with it. Each instance executes exactly one process that runs exactly one crawl job.


Constructor Summary
JMXHeritrixController(HeritrixFiles files)
          Create a JMXHeritrixController object.
 
Method Summary
 boolean atFinish()
          Query whether Heritrix is in a state where it can finish crawling.
 void beginCrawlStop()
          Tell Heritrix to stop crawling.
 void cleanup()
          Cleanup after an Heritrix process.
 boolean crawlIsEnded()
          Check if the crawl has ended, either because Heritrix finished of its own, or because we terminated it.
 int getActiveToeCount()
          Get the number of currently active ToeThreads (crawler threads).
 int getCurrentProcessedKBPerSec()
          Get an estimate of the rate, in kb, at which documents are currently being processed by the crawler.
 java.lang.String getHarvestInformation()
          Return the URL for monitoring this instance.
 java.lang.String getProgressStats()
          Get a human-readable set of statistics on the progress of the crawl.
 long getQueuedUriCount()
          Get the number of URIs currently on the queue to be processed.
 void initialize()
          Initialize a new CrawlController for executing a Heritrix crawl.
 boolean isPaused()
          Returns true if the crawler has been paused, and thus not supposed to fetch anything.
 void requestCrawlStart()
          Request that Heritrix start crawling.
 void requestCrawlStop(java.lang.String reason)
          Request that crawling stops.
 java.lang.String toString()
          Get a string that describes the current controller in terms of job ID, harvest ID, and crawldir.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

JMXHeritrixController

public JMXHeritrixController(HeritrixFiles files)
Create a JMXHeritrixController object.

Parameters:
files - Files that are used to set up Heritrix.
Method Detail

initialize

public void initialize()
Description copied from interface: HeritrixController
Initialize a new CrawlController for executing a Heritrix crawl. This does not start the crawl.

Specified by:
initialize in interface HeritrixController
Throws:
IOFailure - If Heritrix dies before initialization, or we encounter any problems during the initialization.
See Also:
HeritrixController.initialize()

requestCrawlStart

public void requestCrawlStart()
Description copied from interface: HeritrixController
Request that Heritrix start crawling. When this method returns, either Heritrix has failed in the early stages, or the crawljob has been successfully created. Actual crawling will commence at some point hereafter.

Specified by:
requestCrawlStart in interface HeritrixController
Throws:
IOFailure - if unable to communicate with Heritrix
See Also:
HeritrixController.requestCrawlStart()

atFinish

public boolean atFinish()
Description copied from interface: HeritrixController
Query whether Heritrix is in a state where it can finish crawling. Returns true if no uris remain to be harvested, or it has met either the maxbytes limit, the document limit, or the time-limit for the current harvest.

Specified by:
atFinish in interface HeritrixController
Returns:
True if Heritrix thinks it is time to stop crawling.
See Also:
HeritrixController.atFinish()

beginCrawlStop

public void beginCrawlStop()
Description copied from interface: HeritrixController
Tell Heritrix to stop crawling. Heritrix may take a while to actually stop, so you cannot assume that crawling is stopped when this method returns.

Specified by:
beginCrawlStop in interface HeritrixController
Throws:
IOFailure - if unable to communicate with Heritrix
See Also:
HeritrixController.beginCrawlStop()

getActiveToeCount

public int getActiveToeCount()
Description copied from interface: HeritrixController
Get the number of currently active ToeThreads (crawler threads).

Specified by:
getActiveToeCount in interface HeritrixController
Returns:
Number of ToeThreads currently active within Heritrix.
See Also:
HeritrixController.getActiveToeCount()

requestCrawlStop

public void requestCrawlStop(java.lang.String reason)
Description copied from interface: HeritrixController
Request that crawling stops. This makes a call to beginCrawlStop(), unless the crawler is already shutting down. In that case it does nothing.

Specified by:
requestCrawlStop in interface HeritrixController
Parameters:
reason - A human-readable reason the crawl is being stopped.
See Also:
HeritrixController.requestCrawlStop(String)

getQueuedUriCount

public long getQueuedUriCount()
Description copied from interface: HeritrixController
Get the number of URIs currently on the queue to be processed. This number may not be exact and should only be used in informal texts.

Specified by:
getQueuedUriCount in interface HeritrixController
Returns:
How many URIs Heritrix have lined up for processing.
See Also:
HeritrixController.getQueuedUriCount()

getCurrentProcessedKBPerSec

public int getCurrentProcessedKBPerSec()
Description copied from interface: HeritrixController
Get an estimate of the rate, in kb, at which documents are currently being processed by the crawler.

Specified by:
getCurrentProcessedKBPerSec in interface HeritrixController
Returns:
Number of KB data downloaded by Heritrix over an undefined interval up to now.
See Also:
HeritrixController.getCurrentProcessedKBPerSec()

getProgressStats

public java.lang.String getProgressStats()
Description copied from interface: HeritrixController
Get a human-readable set of statistics on the progress of the crawl. The statistics is discovered uris, queued uris, downloaded uris, doc/s(avg), KB/s(avg), dl-failures, busy-thread, mem-use-KB, heap-size-KB, congestion, max-depth, avg-depth. If no statistics are available, the string "No statistics available" is returned. Note: this method may disappear in the future.

Specified by:
getProgressStats in interface HeritrixController
Returns:
Some ascii-formatted statistics on the progress of the crawl.
See Also:
HeritrixController.getProgressStats()

isPaused

public boolean isPaused()
Description copied from interface: HeritrixController
Returns true if the crawler has been paused, and thus not supposed to fetch anything. Heritrix may still be fetching stuff, as it takes some time for it to go into full pause mode. This method can be used as an indicator that we should not be worried if Heritrix appears to be idle.

Specified by:
isPaused in interface HeritrixController
Returns:
True if the crawler has been paused, e.g. by using the Heritrix GUI.
See Also:
HeritrixController.isPaused()

crawlIsEnded

public boolean crawlIsEnded()
Check if the crawl has ended, either because Heritrix finished of its own, or because we terminated it.

Specified by:
crawlIsEnded in interface HeritrixController
Returns:
True if the crawl has ended, either because Heritrix finished or because we terminated it. Otherwise we return false.
See Also:
HeritrixController.crawlIsEnded()

cleanup

public void cleanup()
Cleanup after an Heritrix process. This entails sending the shutdown command to the Heritrix process, and killing it forcefully, if it is still alive after waiting the period of time specified by the CommonSettings.PROCESS_TIMEOUT setting.

Specified by:
cleanup in interface HeritrixController
See Also:
HeritrixController.cleanup()

getHarvestInformation

public java.lang.String getHarvestInformation()
Return the URL for monitoring this instance.

Specified by:
getHarvestInformation in interface HeritrixController
Returns:
the URL for monitoring this instance.

toString

public java.lang.String toString()
Get a string that describes the current controller in terms of job ID, harvest ID, and crawldir.

Overrides:
toString in class java.lang.Object
Returns:
A human-readable string describing this controller.