Package dk.netarkivet.harvester.datamodel

Class H1HeritrixTemplate

java.lang.Object

dk.netarkivet.harvester.datamodel.HeritrixTemplate

dk.netarkivet.harvester.datamodel.H1HeritrixTemplate

All Implemented Interfaces:

java.io.Serializable

public class H1HeritrixTemplate
extends HeritrixTemplate
implements java.io.Serializable

Class encapsulating the Heritrix order.xml. Enables verification that dom4j Document obey the constraints required by our software, specifically the Job class.

The class assumes the type of order.xml used in configuring Heritrix version 1.10+. Information about the Heritrix crawler, and its processes and modules can be found in the Heritrix developer and user manuals found on http://crawler.archive.org

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	ARC_ARCHIVER_PATH_XPATH	Xpath to check, that all templates use the same ARC archiver path, Constants.ARCDIRECTORY_NAME.
static java.lang.String	ARCHIVEFILE_PREFIX_XPATH	Xpath for the arcfile 'prefix' in the order.xml .
static java.lang.String	ARCS_ENABLED_XPATH
static java.lang.String	ARCSDIR_XPATH	Xpath for the ARCs dir in the order.xml.
static java.lang.String	ARCWRITERPROCESSOR_XPATH
static java.lang.String	DECIDERULES_ACCEPT_IF_PREREQUISITE_XPATH	Xpath needed by Job.editOrderXML_crawlerTraps().
static java.lang.String	DECIDERULES_MAP_XPATH	Xpath needed by Job.editOrderXML_crawlerTraps().
static java.lang.String	DECIDINGSCOPE_XPATH	Xpath to check, that all templates use the DecidingScope.
static java.lang.String	DEDUPLICATOR_ENABLED	Xpath for the boolean telling if the deduplicator is enabled in order.xml documents.
static java.lang.String	DEDUPLICATOR_INDEX_LOCATION_XPATH	Xpath for the deduplicator index directory node in order.xml documents.
static java.lang.String	DEDUPLICATOR_XPATH	Xpath for the deduplicator node in order.xml documents.
static java.lang.String	DISK_PATH_XPATH	Xpath for the 'disk-path' in the order.xml .
static java.lang.String	GROUP_MAX_ALL_KB_XPATH	Xpath needed by Job.editOrderXML_maxBytesPerDomain().
static java.lang.String	GROUP_MAX_FETCH_SUCCESS_XPATH	Xpath needed by Job.editOrderXML_maxObjectsPerDomain().
static java.lang.String	HERITRIX_FROM_XPATH	Xpath checked by Heritrix for correct mail address.
static java.lang.String	HERITRIX_USER_AGENT_XPATH	Xpath checked by Heritrix for correct user-agent field in requests.
static java.lang.String	MAXTIMESEC_PATH_XPATH	Xpath to check, that all templates have the max-time-sec attribute.
static java.lang.String	METADATA_ITEMS_XPATH	Xpath for the WARC metadata in the order.xml.
static java.lang.String	QUEUE_TOTAL_BUDGET_XPATH	Xpath needed by Job.editOrderXML_maxObjectsPerDomain().
static java.lang.String	QUOTA_ENFORCER_ENABLED_XPATH	Xpath needed by Job.editOrderXML_maxBytesPerDomain().
static java.lang.String	SEEDS_FILE_XPATH	Xpath for the 'seedsfile' in the order.xml.
static java.lang.String	WARC_ARCHIVER_PATH_XPATH	Xpath to check, that all templates use the same WARC archiver path, Constants.WARCDIRECTORY_NAME.
static java.lang.String	WARCS_ENABLED_XPATH	Xpath for the WARCs dir in the order.xml.
static java.lang.String	WARCS_SKIP_IDENTICAL_DIGESTS_XPATH
static java.lang.String	WARCS_WRITE_METADATA_OUTLINKS_XPATH
static java.lang.String	WARCS_WRITE_METADATA_XPATH
static java.lang.String	WARCS_WRITE_REQUESTS_XPATH
static java.lang.String	WARCS_WRITE_REVISIT_FOR_IDENTICAL_DIGESTS_XPATH
static java.lang.String	WARCS_WRITE_REVISIT_FOR_NOT_MODIFIED_XPATH
static java.lang.String	WARCSDIR_XPATH	Xpath for the WARCs dir in the order.xml.
static java.lang.String	WARCWRITERPROCESSOR_XPATH

Fields inherited from class dk.netarkivet.harvester.datamodel.HeritrixTemplate

HARVESTINFO_AUDIENCE, HARVESTINFO_CHANNEL, HARVESTINFO_HARVESTFILENAMEPREFIX, HARVESTINFO_HARVESTNUM, HARVESTINFO_JOBID, HARVESTINFO_JOBSUBMITDATE, HARVESTINFO_MAXBYTESPERDOMAIN, HARVESTINFO_MAXOBJECTSPERDOMAIN, HARVESTINFO_OPERATOR, HARVESTINFO_ORDERXMLDESCRIPTION, HARVESTINFO_ORDERXMLNAME, HARVESTINFO_ORDERXMLUPDATEDATE, HARVESTINFO_ORIGHARVESTDEFINITIONCOMMENTS, HARVESTINFO_ORIGHARVESTDEFINITIONID, HARVESTINFO_ORIGHARVESTDEFINITIONNAME, HARVESTINFO_PERFORMER, HARVESTINFO_SCHEDULENAME, HARVESTINFO_VERSION, HARVESTINFO_VERSION_NUMBER, template_id

Constructor Summary

Constructors

Constructor	Description
H1HeritrixTemplate(long template_id, java.lang.String templateAsString)
H1HeritrixTemplate(org.dom4j.Document doc)	Alternate constructor, which always verifies the given document.
H1HeritrixTemplate(org.dom4j.Document doc, boolean verify)	Constructor for HeritrixTemplate class.

Method Summary

Modifier and Type	Method	Description
void	configureQuotaEnforcer(boolean maxObjectsIsSetByQuotaEnforcer, long forceMaxBytesPerDomain, long forceMaxObjectsPerDomain)	Activates or deactivate the quota-enforcer, depending on budget definition.
static void	editOrderXML_configureQuotaEnforcer(org.dom4j.Document orderXMLdoc, boolean maxObjectsIsSetByQuotaEnforcer, long forceMaxBytesPerDomain, long forceMaxObjectsPerDomain)	Activates or deactivate the quota-enforcer, depending on budget definition.
static void	editOrderXML_maxObjectsPerDomain(org.dom4j.Document orderXMLdoc, long forceMaxObjectsPerDomain, boolean maxObjectsIsSetByQuotaEnforcer)	Auxiliary method to modify the orderXMLdoc Document with respect to setting the maximum number of objects to be retrieved per domain.
static void	editOrderXMLAddCrawlerTraps(org.dom4j.Document orderXMLdoc, java.lang.String elementName, java.util.List<java.lang.String> crawlerTraps)	Method to add a list of crawler traps with a given element name.
void	enableOrDisableDeduplication(boolean enabled)
java.lang.Long	getMaxBytesPerDomain()
java.lang.Long	getMaxObjectsPerDomain()
org.dom4j.Document	getTemplate()	return the template.
java.lang.String	getText()	Only available for H1 templates.
java.lang.String	getXML()	Return HeritrixTemplate as XML.
boolean	hasContent()
void	insertAttributes(java.util.List<EAV.AttributeAndType> attributesAndTypes)	Try to insert the given list of attributes into the template.
void	insertCrawlerTraps(java.lang.String elementName, java.util.List<java.lang.String> crawlerTraps)	Method to add a list of crawler traps with a given element name.
void	insertUmbrabean(java.lang.String jobName, java.lang.String rabbitMQUrl, java.lang.String limitSearchRegEx)	Inserts all nevessary umbra-related beans in this template.
void	insertWarcInfoMetadata(Job ajob, java.lang.String origHarvestdefinitionName, java.lang.String origHarvestdefinitionComments, java.lang.String scheduleName, java.lang.String performer)	Method to add settings to the WARCWriterProcesser, so that it can generate a proper WARCINFO record.
boolean	IsDeduplicationEnabled()	Return true if the templatefile has deduplication enabled.
boolean	isValid()
boolean	isVerified()	Has Template been verified?
void	removeDeduplicatorIfPresent()	Try to remove the deduplicator, if present in the template.
void	setArchiveFilePrefix(java.lang.String archiveFilePrefix)
void	setArchiveFormat(java.lang.String archiveFormat)	Make sure that Heritrix will archive its data in the chosen archiveFormat.
void	setDeduplicationIndexLocation(java.lang.String absolutePath)
void	setDiskPath(java.lang.String absolutePath)
void	setMaxBytesPerDomain(java.lang.Long forceMaxBytesPerDomain)	Auxiliary method to modify the orderXMLdoc Document with respect to setting the maximum number of bytes to retrieve per domain.
void	setMaxJobRunningTime(java.lang.Long maxJobRunningTimeSecondsL)	Set the maxRunning time for the harvest
void	setMaxObjectsPerDomain(java.lang.Long maxobjectsL)
void	setRecoverlogNode(java.io.File recoverlogGzFile)
void	setSeedsFilePath(java.lang.String absolutePath)
void	writeTemplate(java.io.OutputStream os)
void	writeTemplate(javax.servlet.jsp.JspWriter out)
void	writeToFile(java.io.File orderXmlFile)

Methods inherited from class dk.netarkivet.harvester.datamodel.HeritrixTemplate

editOrderXMLAddPerDomainCrawlerTraps, getTemplateFromString, isActive, read, read, setIsActive

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

QUOTA_ENFORCER_ENABLED_XPATH

public static final java.lang.String QUOTA_ENFORCER_ENABLED_XPATH

Xpath needed by Job.editOrderXML_maxBytesPerDomain().

See Also:

Constant Field Values

GROUP_MAX_ALL_KB_XPATH

public static final java.lang.String GROUP_MAX_ALL_KB_XPATH

Xpath needed by Job.editOrderXML_maxBytesPerDomain().

See Also:

Constant Field Values

GROUP_MAX_FETCH_SUCCESS_XPATH

public static final java.lang.String GROUP_MAX_FETCH_SUCCESS_XPATH

Xpath needed by Job.editOrderXML_maxObjectsPerDomain().

See Also:

Constant Field Values

QUEUE_TOTAL_BUDGET_XPATH

public static final java.lang.String QUEUE_TOTAL_BUDGET_XPATH

Xpath needed by Job.editOrderXML_maxObjectsPerDomain().

See Also:

Constant Field Values

DECIDERULES_MAP_XPATH

public static final java.lang.String DECIDERULES_MAP_XPATH

Xpath needed by Job.editOrderXML_crawlerTraps().

See Also:

Constant Field Values

DECIDERULES_ACCEPT_IF_PREREQUISITE_XPATH

public static final java.lang.String DECIDERULES_ACCEPT_IF_PREREQUISITE_XPATH

Xpath needed by Job.editOrderXML_crawlerTraps().

See Also:

Constant Field Values

HERITRIX_USER_AGENT_XPATH

public static final java.lang.String HERITRIX_USER_AGENT_XPATH

Xpath checked by Heritrix for correct user-agent field in requests.

See Also:

Constant Field Values

HERITRIX_FROM_XPATH

public static final java.lang.String HERITRIX_FROM_XPATH

Xpath checked by Heritrix for correct mail address.

See Also:

Constant Field Values

DECIDINGSCOPE_XPATH

public static final java.lang.String DECIDINGSCOPE_XPATH

Xpath to check, that all templates use the DecidingScope.

See Also:

Constant Field Values

DEDUPLICATOR_XPATH

public static final java.lang.String DEDUPLICATOR_XPATH

Xpath for the deduplicator node in order.xml documents.

See Also:

Constant Field Values

ARC_ARCHIVER_PATH_XPATH

public static final java.lang.String ARC_ARCHIVER_PATH_XPATH

Xpath to check, that all templates use the same ARC archiver path, Constants.ARCDIRECTORY_NAME. The archive path tells Heritrix to which directory it shall write its arc files.

See Also:

Constant Field Values

WARC_ARCHIVER_PATH_XPATH

public static final java.lang.String WARC_ARCHIVER_PATH_XPATH

Xpath to check, that all templates use the same WARC archiver path, Constants.WARCDIRECTORY_NAME. The archive path tells Heritrix to which directory it shall write its arc files.

See Also:

Constant Field Values

DEDUPLICATOR_INDEX_LOCATION_XPATH

public static final java.lang.String DEDUPLICATOR_INDEX_LOCATION_XPATH

Xpath for the deduplicator index directory node in order.xml documents.

See Also:

Constant Field Values

DEDUPLICATOR_ENABLED

public static final java.lang.String DEDUPLICATOR_ENABLED

Xpath for the boolean telling if the deduplicator is enabled in order.xml documents.

See Also:

Constant Field Values

DISK_PATH_XPATH

public static final java.lang.String DISK_PATH_XPATH

Xpath for the 'disk-path' in the order.xml .

See Also:

Constant Field Values

ARCHIVEFILE_PREFIX_XPATH

public static final java.lang.String ARCHIVEFILE_PREFIX_XPATH

Xpath for the arcfile 'prefix' in the order.xml .

See Also:

Constant Field Values

ARCSDIR_XPATH

public static final java.lang.String ARCSDIR_XPATH

Xpath for the ARCs dir in the order.xml.

See Also:

Constant Field Values

WARCWRITERPROCESSOR_XPATH

public static final java.lang.String WARCWRITERPROCESSOR_XPATH

See Also:

Constant Field Values

ARCWRITERPROCESSOR_XPATH

public static final java.lang.String ARCWRITERPROCESSOR_XPATH

See Also:

Constant Field Values

WARCSDIR_XPATH

public static final java.lang.String WARCSDIR_XPATH

Xpath for the WARCs dir in the order.xml.

See Also:

Constant Field Values

SEEDS_FILE_XPATH

public static final java.lang.String SEEDS_FILE_XPATH

Xpath for the 'seedsfile' in the order.xml.

See Also:

Constant Field Values

ARCS_ENABLED_XPATH

public static final java.lang.String ARCS_ENABLED_XPATH

See Also:

Constant Field Values

WARCS_ENABLED_XPATH

public static final java.lang.String WARCS_ENABLED_XPATH

Xpath for the WARCs dir in the order.xml.

See Also:

Constant Field Values

WARCS_WRITE_REQUESTS_XPATH

public static final java.lang.String WARCS_WRITE_REQUESTS_XPATH

See Also:

Constant Field Values

WARCS_WRITE_METADATA_XPATH

public static final java.lang.String WARCS_WRITE_METADATA_XPATH

See Also:

Constant Field Values

WARCS_WRITE_METADATA_OUTLINKS_XPATH

public static final java.lang.String WARCS_WRITE_METADATA_OUTLINKS_XPATH

See Also:

Constant Field Values

WARCS_SKIP_IDENTICAL_DIGESTS_XPATH

public static final java.lang.String WARCS_SKIP_IDENTICAL_DIGESTS_XPATH

See Also:

Constant Field Values

WARCS_WRITE_REVISIT_FOR_IDENTICAL_DIGESTS_XPATH

public static final java.lang.String WARCS_WRITE_REVISIT_FOR_IDENTICAL_DIGESTS_XPATH

See Also:

Constant Field Values

WARCS_WRITE_REVISIT_FOR_NOT_MODIFIED_XPATH

public static final java.lang.String WARCS_WRITE_REVISIT_FOR_NOT_MODIFIED_XPATH

See Also:

Constant Field Values

METADATA_ITEMS_XPATH

public static final java.lang.String METADATA_ITEMS_XPATH

Xpath for the WARC metadata in the order.xml.

See Also:

Constant Field Values

MAXTIMESEC_PATH_XPATH

public static final java.lang.String MAXTIMESEC_PATH_XPATH

Xpath to check, that all templates have the max-time-sec attribute.

See Also:

Constant Field Values

Constructor Detail

H1HeritrixTemplate

public H1HeritrixTemplate(org.dom4j.Document doc,
                          boolean verify)

Constructor for HeritrixTemplate class.

Parameters:

doc - the order.xml

verify - If true, verifies if the given dom4j Document contains the elements required by our software.

Throws:

ArgumentNotValid - if doc is null, or verify is true and doc does not obey the constraints required by our software.

H1HeritrixTemplate

public H1HeritrixTemplate(org.dom4j.Document doc)

Alternate constructor, which always verifies the given document.

Parameters:

doc -

H1HeritrixTemplate

public H1HeritrixTemplate(long template_id,
                          java.lang.String templateAsString)
                   throws org.dom4j.DocumentException

Throws:

org.dom4j.DocumentException

Method Detail

getTemplate

public org.dom4j.Document getTemplate()

return the template.

Returns:

the template

isVerified

public boolean isVerified()

Has Template been verified?

Returns:

true, if verified on construction, otherwise false

getXML

public java.lang.String getXML()

Return HeritrixTemplate as XML.

Specified by:

getXML in class HeritrixTemplate

Returns:

HeritrixTemplate as XML

editOrderXMLAddCrawlerTraps

public static void editOrderXMLAddCrawlerTraps(org.dom4j.Document orderXMLdoc,
                                               java.lang.String elementName,
                                               java.util.List<java.lang.String> crawlerTraps)

Method to add a list of crawler traps with a given element name. It is used both to add per-domain traps and global traps.

Parameters:

elementName - The name of the added element.

crawlerTraps - A list of crawler trap regular expressions to add to this job.

editOrderXML_maxObjectsPerDomain

public static void editOrderXML_maxObjectsPerDomain(org.dom4j.Document orderXMLdoc,
                                                    long forceMaxObjectsPerDomain,
                                                    boolean maxObjectsIsSetByQuotaEnforcer)

Auxiliary method to modify the orderXMLdoc Document with respect to setting the maximum number of objects to be retrieved per domain. This method updates 'group-max-fetch-success' element of the QuotaEnforcer pre-fetch processor node (org.archive.crawler.frontier.BdbFrontier) with the value of the argument forceMaxObjectsPerDomain

Parameters:

orderXMLdoc -

forceMaxObjectsPerDomain - The maximum number of objects to retrieve per domain, or 0 for no limit.

Throws:

PermissionDenied - If unable to replace the frontier node of the orderXMLdoc Document

IOFailure - If the group-max-fetch-success element is not found in the orderXml. TODO The group-max-fetch-success check should also be performed in TemplateDAO.create, TemplateDAO.update

editOrderXML_configureQuotaEnforcer

public static void editOrderXML_configureQuotaEnforcer(org.dom4j.Document orderXMLdoc,
                                                       boolean maxObjectsIsSetByQuotaEnforcer,
                                                       long forceMaxBytesPerDomain,
                                                       long forceMaxObjectsPerDomain)

Activates or deactivate the quota-enforcer, depending on budget definition. Object limit can be defined either by using the queue-total-budget property or the quota enforcer. Which is chosen is set by the argument maxObjectsIsSetByQuotaEnforcer}'s value. So quota enforcer is set as follows:

• Object limit is not set by quota enforcer, disabled only if there is no byte limit.
• Object limit is set by quota enforcer, so it should be enabled whether a byte or object limit is set.

Parameters:

orderXMLdoc - the template to modify

maxObjectsIsSetByQuotaEnforcer - Decides whether the maxObjectsIsSetByQuotaEnforcer or not.

forceMaxBytesPerDomain - The number of max bytes per domain enforced (can be no limit)

forceMaxObjectsPerDomain - The number of max objects per domain enforced (can be no limit)

isValid

public boolean isValid()

Specified by:

isValid in class HeritrixTemplate

Returns:

true, if the template is valid, otherwise false

configureQuotaEnforcer

public void configureQuotaEnforcer(boolean maxObjectsIsSetByQuotaEnforcer,
                                   long forceMaxBytesPerDomain,
                                   long forceMaxObjectsPerDomain)

Description copied from class: HeritrixTemplate

Activates or deactivate the quota-enforcer, depending on budget definition. Object limit can be defined either by using the queue-total-budget property or the quota enforcer. Which is chosen is set by the argument maxObjectsIsSetByQuotaEnforcer}'s value. So quota enforcer is set as follows:

• Object limit is not set by quota enforcer, disabled only if there is no byte limit.
• Object limit is set by quota enforcer, so it should be enabled whether a byte or object limit is set.

Specified by:

configureQuotaEnforcer in class HeritrixTemplate

Parameters:

maxObjectsIsSetByQuotaEnforcer - Decides whether the maxObjectsIsSetByQuotaEnforcer or not.

forceMaxBytesPerDomain - The number of max bytes per domain enforced (can be no limit)

forceMaxObjectsPerDomain - The number of max objects per domain enforced (can be no limit)

setMaxBytesPerDomain

public void setMaxBytesPerDomain(java.lang.Long forceMaxBytesPerDomain)

Auxiliary method to modify the orderXMLdoc Document with respect to setting the maximum number of bytes to retrieve per domain. This method updates 'group-max-all-kb' element of the 'QuotaEnforcer' node, which again is a subelement of 'pre-fetch-processors' node. with the value of the argument forceMaxBytesPerDomain

Specified by:

setMaxBytesPerDomain in class HeritrixTemplate

Parameters:

forceMaxBytesPerDomain - The maximum number of byte to retrieve per domain, or -1 for no limit. Note that the number is divided by 1024 before being inserted into the orderXml, as Heritrix expects KB.

Throws:

PermissionDenied - If unable to replace the QuotaEnforcer node of the orderXMLdoc Document

IOFailure - If the group-max-all-kb element cannot be found. TODO This group-max-all-kb check also be performed in TemplateDAO.create, TemplateDAO.update

getMaxBytesPerDomain

public java.lang.Long getMaxBytesPerDomain()

Specified by:

getMaxBytesPerDomain in class HeritrixTemplate

setMaxObjectsPerDomain

public void setMaxObjectsPerDomain(java.lang.Long maxobjectsL)

Specified by:

setMaxObjectsPerDomain in class HeritrixTemplate

getMaxObjectsPerDomain

public java.lang.Long getMaxObjectsPerDomain()

Specified by:

getMaxObjectsPerDomain in class HeritrixTemplate

IsDeduplicationEnabled

public boolean IsDeduplicationEnabled()

Return true if the templatefile has deduplication enabled.

Specified by:

IsDeduplicationEnabled in class HeritrixTemplate

Returns:

True if Deduplicator is enabled.

setArchiveFormat

public void setArchiveFormat(java.lang.String archiveFormat)

Description copied from class: HeritrixTemplate

Make sure that Heritrix will archive its data in the chosen archiveFormat.

Specified by:

setArchiveFormat in class HeritrixTemplate

Parameters:

archiveFormat - the chosen archiveformat ('arc' or 'warc' supported) Throws ArgumentNotValid If the chosen archiveFormat is not supported.

setMaxJobRunningTime

public void setMaxJobRunningTime(java.lang.Long maxJobRunningTimeSecondsL)

Description copied from class: HeritrixTemplate

Set the maxRunning time for the harvest

Specified by:

setMaxJobRunningTime in class HeritrixTemplate

Parameters:

maxJobRunningTimeSecondsL - Limit the harvest to this number of seconds

writeTemplate

public void writeTemplate(java.io.OutputStream os)
                   throws java.io.IOException,
                          ArgumentNotValid

Specified by:

writeTemplate in class HeritrixTemplate

Throws:

java.io.IOException

ArgumentNotValid

getText

public java.lang.String getText()

Only available for H1 templates.

Returns:

the template as a String.

insertCrawlerTraps

public void insertCrawlerTraps(java.lang.String elementName,
                               java.util.List<java.lang.String> crawlerTraps)

Description copied from class: HeritrixTemplate

Method to add a list of crawler traps with a given element name. It is used both to add per-domain traps and global traps.

Specified by:

insertCrawlerTraps in class HeritrixTemplate

Parameters:

elementName - The name of the added element.

crawlerTraps - A list of crawler trap regular expressions to add to this job.

hasContent

public boolean hasContent()

Specified by:

hasContent in class HeritrixTemplate

writeToFile

public void writeToFile(java.io.File orderXmlFile)

Specified by:

writeToFile in class HeritrixTemplate

setRecoverlogNode

public void setRecoverlogNode(java.io.File recoverlogGzFile)

Specified by:

setRecoverlogNode in class HeritrixTemplate

setDeduplicationIndexLocation

public void setDeduplicationIndexLocation(java.lang.String absolutePath)

Specified by:

setDeduplicationIndexLocation in class HeritrixTemplate

setSeedsFilePath

public void setSeedsFilePath(java.lang.String absolutePath)

Specified by:

setSeedsFilePath in class HeritrixTemplate

setArchiveFilePrefix

public void setArchiveFilePrefix(java.lang.String archiveFilePrefix)

Specified by:

setArchiveFilePrefix in class HeritrixTemplate

setDiskPath

public void setDiskPath(java.lang.String absolutePath)

Specified by:

setDiskPath in class HeritrixTemplate

removeDeduplicatorIfPresent

public void removeDeduplicatorIfPresent()

Description copied from class: HeritrixTemplate

Try to remove the deduplicator, if present in the template.

Specified by:

removeDeduplicatorIfPresent in class HeritrixTemplate

enableOrDisableDeduplication

public void enableOrDisableDeduplication(boolean enabled)

Specified by:

enableOrDisableDeduplication in class HeritrixTemplate

insertWarcInfoMetadata

public void insertWarcInfoMetadata(Job ajob,
                                   java.lang.String origHarvestdefinitionName,
                                   java.lang.String origHarvestdefinitionComments,
                                   java.lang.String scheduleName,
                                   java.lang.String performer)

Description copied from class: HeritrixTemplate

Method to add settings to the WARCWriterProcesser, so that it can generate a proper WARCINFO record.

Specified by:

insertWarcInfoMetadata in class HeritrixTemplate

Parameters:

ajob - a HarvestJob

origHarvestdefinitionName - The name of the harvestdefinition behind this job

scheduleName - The name of the schedule used. (Will be null, if the job is not a selectiveHarvest).

performer - The name of organisation/person doing this harvest

insertAttributes

public void insertAttributes(java.util.List<EAV.AttributeAndType> attributesAndTypes)

Description copied from class: HeritrixTemplate

Try to insert the given list of attributes into the template.

Specified by:

insertAttributes in class HeritrixTemplate

writeTemplate

public void writeTemplate(javax.servlet.jsp.JspWriter out)
                   throws IOFailure

Specified by:

writeTemplate in class HeritrixTemplate

Throws:

IOFailure

insertUmbrabean

public void insertUmbrabean(java.lang.String jobName,
                            java.lang.String rabbitMQUrl,
                            java.lang.String limitSearchRegEx)

Description copied from class: HeritrixTemplate

Inserts all nevessary umbra-related beans in this template.

Specified by:

insertUmbrabean in class HeritrixTemplate

Parameters:

jobName - a String representing the job - must be unique for the this NAS environment for all time

rabbitMQUrl - the URL of the rabbitMQ socket connection (amqp://) to which umbra requests are to be sent

limitSearchRegEx - the regular expression used to limit the heritrix search-path of urls to be sent to Umbra.