Package is.hi.bok.deduplicator

Class DeDuplicator

java.lang.Object

org.archive.modules.Processor

is.hi.bok.deduplicator.DeDuplicator

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.beans.factory.InitializingBean, org.springframework.context.Lifecycle

public class DeDuplicator
extends org.archive.modules.Processor
implements org.springframework.beans.factory.InitializingBean

Heritrix compatible processor.

Will determine if CrawlURIs are duplicates.

Duplicate detection can only be performed after the fetch processors have run. Modified by SVC to use Lucene 4.X

Author:

Kristinn Sigurðsson, Søren Vejrup Carlsen other option: DIGEST Other option: WHITELIST Other options: NONE, TIMESTAMP_AND_ETAG Other options: NONE,PROCESSOR // /** // (FROM deduplicator-commons/src/main/java/is/landsbokasafn/deduplicator/IndexFields.java) // * These enums correspond to the names of fields in the Lucene index //

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DeDuplicator.AnalysisMode
static class	DeDuplicator.FilterMode
static class	DeDuplicator.MatchingMethod
static class	DeDuplicator.OriginHandling

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	ATTR_ANALYZE_MODE
static java.lang.String	ATTR_CHANGE_CONTENT_SIZE
static java.lang.String	ATTR_EQUIVALENT
static java.lang.String	ATTR_FILTER_MODE
static java.lang.String	ATTR_JUMP_TO
static java.lang.String	ATTR_MIME_FILTER
static java.lang.String	ATTR_ORIGIN
static java.lang.String	ATTR_ORIGIN_HANDLING
static java.lang.String	ATTR_REVISIT_IN_WARCS
static java.lang.String	ATTR_STATS_PER_HOST
static java.lang.String	DEFAULT_MIME_FILTER
static DeDuplicator.OriginHandling	DEFAULT_ORIGIN_HANDLING
protected org.apache.lucene.index.IndexReader	indexReader
protected org.apache.lucene.search.IndexSearcher	indexSearcher
protected boolean	lookupByURL
protected java.util.HashMap<java.lang.String,is.hi.bok.deduplicator.Statistics>	perHostStats
protected org.archive.modules.net.ServerCache	serverCache
protected is.hi.bok.deduplicator.Statistics	stats
protected boolean	statsPerHost
protected boolean	useOrigin
protected boolean	useOriginFromIndex

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor	Description
DeDuplicator()

Method Summary

Modifier and Type	Method	Description
void	afterPropertiesSet()
protected void	doAnalysis(org.archive.modules.CrawlURI curi, is.hi.bok.deduplicator.Statistics currHostStats, boolean isDuplicate)
protected void	doTimestampAnalysis(org.archive.modules.CrawlURI curi, org.apache.lucene.document.Document urlHit, is.hi.bok.deduplicator.Statistics currHostStats, boolean isDuplicate)
DeDuplicator.AnalysisMode	getAnalysisMode()
boolean	getAnalyzeTimestamp()
java.lang.Boolean	getBlacklist()
java.lang.Boolean	getChangeContentSize()
boolean	getEnabled()
DeDuplicator.FilterMode	getFilterMode()
java.lang.String	getIndexLocation()
java.lang.String	getJumpTo()
DeDuplicator.MatchingMethod	getMatchingMethod()
java.lang.String	getMimeFilter()
java.lang.String	getOrigin()
DeDuplicator.OriginHandling	getOriginHandling()
protected static java.lang.String	getPercentage(double portion, double total)
java.lang.Boolean	getRevisitInWarcs()
org.archive.modules.net.ServerCache	getServerCache()
java.lang.Boolean	getStatsPerHost()
java.lang.Boolean	getTryEquivalent()
protected void	innerProcess(org.archive.modules.CrawlURI puri)
protected org.archive.modules.ProcessResult	innerProcessResult(org.archive.modules.CrawlURI curi)
protected org.apache.lucene.document.Document	lookupByDigest(org.archive.modules.CrawlURI curi, is.hi.bok.deduplicator.Statistics currHostStats)	Process a CrawlURI looking up in the index by content digest
protected org.apache.lucene.document.Document	lookupByURL(org.archive.modules.CrawlURI curi, is.hi.bok.deduplicator.Statistics currHostStats)	Process a CrawlURI looking up in the index by URL
protected org.apache.lucene.search.Query	queryField(java.lang.String fieldName, java.lang.String value)	Run a simple Lucene query for a single term in a single field.
java.lang.String	report()
void	setAnalysisMode(DeDuplicator.AnalysisMode analyzeMode)
void	setChangeContentSize(java.lang.Boolean changeContentSize)	SPRING SETTER
void	setEnabled(boolean enabled)
void	setfilterMode(DeDuplicator.FilterMode filterMode)	SPRING SETTER method
void	setIndexLocation(java.lang.String indexLocation)	SETTER used by Spring
void	setJumpTo(java.lang.String jumpTo)	SPRING SETTER.
void	setMatchingMethod(DeDuplicator.MatchingMethod method)	SETTER used by Spring
void	setMimeFilter(java.lang.String mimeFilter)
void	setOrigin(java.lang.String origin)	SPRING SETTER
void	setOriginHandling(DeDuplicator.OriginHandling originHandling)
void	setRevisitInWarcs(java.lang.Boolean revisitOn)
void	setServerCache(org.archive.modules.net.ServerCache serverCache)
void	setStatsPerHost(java.lang.Boolean statsPerHost)
void	setTryEquivalent(java.lang.Boolean tryEquivalent)	SPRING SETTER
protected boolean	shouldProcess(org.archive.modules.CrawlURI curi)

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerRejectProcess, isRunning, isSuccess, process, setBeanName, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ATTR_JUMP_TO

public static final java.lang.String ATTR_JUMP_TO

See Also:

Constant Field Values

ATTR_ORIGIN

public static final java.lang.String ATTR_ORIGIN

See Also:

Constant Field Values

ATTR_EQUIVALENT

public static final java.lang.String ATTR_EQUIVALENT

See Also:

Constant Field Values

ATTR_MIME_FILTER

public static final java.lang.String ATTR_MIME_FILTER

See Also:

Constant Field Values

DEFAULT_MIME_FILTER

public static final java.lang.String DEFAULT_MIME_FILTER

See Also:

Constant Field Values

ATTR_FILTER_MODE

public static final java.lang.String ATTR_FILTER_MODE

See Also:

Constant Field Values

ATTR_ANALYZE_MODE

public static final java.lang.String ATTR_ANALYZE_MODE

See Also:

Constant Field Values

ATTR_CHANGE_CONTENT_SIZE

public static final java.lang.String ATTR_CHANGE_CONTENT_SIZE

See Also:

Constant Field Values

ATTR_STATS_PER_HOST

public static final java.lang.String ATTR_STATS_PER_HOST

See Also:

Constant Field Values

ATTR_ORIGIN_HANDLING

public static final java.lang.String ATTR_ORIGIN_HANDLING

See Also:

Constant Field Values

DEFAULT_ORIGIN_HANDLING

public static final DeDuplicator.OriginHandling DEFAULT_ORIGIN_HANDLING

ATTR_REVISIT_IN_WARCS

public static final java.lang.String ATTR_REVISIT_IN_WARCS

See Also:

Constant Field Values

serverCache

protected org.archive.modules.net.ServerCache serverCache

indexSearcher

protected org.apache.lucene.search.IndexSearcher indexSearcher

indexReader

protected org.apache.lucene.index.IndexReader indexReader

lookupByURL

protected boolean lookupByURL

statsPerHost

protected boolean statsPerHost

useOrigin

protected boolean useOrigin

useOriginFromIndex

protected boolean useOriginFromIndex

stats

protected is.hi.bok.deduplicator.Statistics stats

perHostStats

protected java.util.HashMap<java.lang.String,is.hi.bok.deduplicator.Statistics> perHostStats

Constructor Detail

DeDuplicator

public DeDuplicator()

Method Detail

getEnabled

public boolean getEnabled()

Overrides:

getEnabled in class org.archive.modules.Processor

setEnabled

public void setEnabled(boolean enabled)

Overrides:

setEnabled in class org.archive.modules.Processor

getIndexLocation

public java.lang.String getIndexLocation()

setIndexLocation

public void setIndexLocation(java.lang.String indexLocation)

SETTER used by Spring

getMatchingMethod

public DeDuplicator.MatchingMethod getMatchingMethod()

setMatchingMethod

public void setMatchingMethod(DeDuplicator.MatchingMethod method)

SETTER used by Spring

getJumpTo

public java.lang.String getJumpTo()

setJumpTo

public void setJumpTo(java.lang.String jumpTo)

SPRING SETTER. TODO Are we using this property?? The netarkivet are not

getOrigin

public java.lang.String getOrigin()

setOrigin

public void setOrigin(java.lang.String origin)

SPRING SETTER

getTryEquivalent

public java.lang.Boolean getTryEquivalent()

setTryEquivalent

public void setTryEquivalent(java.lang.Boolean tryEquivalent)

SPRING SETTER

getMimeFilter

public java.lang.String getMimeFilter()

setMimeFilter

public void setMimeFilter(java.lang.String mimeFilter)

getFilterMode

public DeDuplicator.FilterMode getFilterMode()

getBlacklist

public java.lang.Boolean getBlacklist()

setfilterMode

public void setfilterMode(DeDuplicator.FilterMode filterMode)

SPRING SETTER method

getAnalyzeTimestamp

public boolean getAnalyzeTimestamp()

setAnalysisMode

public void setAnalysisMode(DeDuplicator.AnalysisMode analyzeMode)

getAnalysisMode

public DeDuplicator.AnalysisMode getAnalysisMode()

getChangeContentSize

public java.lang.Boolean getChangeContentSize()

setChangeContentSize

public void setChangeContentSize(java.lang.Boolean changeContentSize)

SPRING SETTER

getStatsPerHost

public java.lang.Boolean getStatsPerHost()

setStatsPerHost

public void setStatsPerHost(java.lang.Boolean statsPerHost)

getOriginHandling

public DeDuplicator.OriginHandling getOriginHandling()

setOriginHandling

public void setOriginHandling(DeDuplicator.OriginHandling originHandling)

setRevisitInWarcs

public void setRevisitInWarcs(java.lang.Boolean revisitOn)

getRevisitInWarcs

public java.lang.Boolean getRevisitInWarcs()

getServerCache

public org.archive.modules.net.ServerCache getServerCache()

setServerCache

@Autowired
public void setServerCache(org.archive.modules.net.ServerCache serverCache)

afterPropertiesSet

public void afterPropertiesSet()
                        throws java.lang.Exception

Specified by:

afterPropertiesSet in interface org.springframework.beans.factory.InitializingBean

Throws:

java.lang.Exception

shouldProcess

protected boolean shouldProcess(org.archive.modules.CrawlURI curi)

Specified by:

shouldProcess in class org.archive.modules.Processor

innerProcess

protected void innerProcess(org.archive.modules.CrawlURI puri)

Specified by:

innerProcess in class org.archive.modules.Processor

innerProcessResult

protected org.archive.modules.ProcessResult innerProcessResult(org.archive.modules.CrawlURI curi)
                                                        throws java.lang.InterruptedException

Overrides:

innerProcessResult in class org.archive.modules.Processor

Throws:

java.lang.InterruptedException

lookupByURL

protected org.apache.lucene.document.Document lookupByURL(org.archive.modules.CrawlURI curi,
                                                          is.hi.bok.deduplicator.Statistics currHostStats)

Process a CrawlURI looking up in the index by URL

Parameters:

curi - The CrawlURI to process

currHostStats - A statistics object for the current host. If per host statistics tracking is enabled this must be non null and the method will increment appropriate counters on it.

Returns:

The result of the lookup (a Lucene document). If a duplicate is not found null is returned.

lookupByDigest

protected org.apache.lucene.document.Document lookupByDigest(org.archive.modules.CrawlURI curi,
                                                             is.hi.bok.deduplicator.Statistics currHostStats)

Process a CrawlURI looking up in the index by content digest

Parameters:

curi - The CrawlURI to process

currHostStats - A statistics object for the current host. If per host statistics tracking is enabled this must be non null and the method will increment appropriate counters on it.

Returns:

The result of the lookup (a Lucene document). If a duplicate is not found null is returned.

report

public java.lang.String report()

Overrides:

report in class org.archive.modules.Processor

getPercentage

protected static java.lang.String getPercentage(double portion,
                                                double total)

doAnalysis

protected void doAnalysis(org.archive.modules.CrawlURI curi,
                          is.hi.bok.deduplicator.Statistics currHostStats,
                          boolean isDuplicate)

doTimestampAnalysis

protected void doTimestampAnalysis(org.archive.modules.CrawlURI curi,
                                   org.apache.lucene.document.Document urlHit,
                                   is.hi.bok.deduplicator.Statistics currHostStats,
                                   boolean isDuplicate)

queryField

protected org.apache.lucene.search.Query queryField(java.lang.String fieldName,
                                                    java.lang.String value)

Run a simple Lucene query for a single term in a single field.

Parameters:

fieldName - name of the field to look in.

value - The value to query for

Returns:

A Query for the given value in the given field.