Modifier and Type | Method and Description |
---|---|
static ArcRepository |
ArcRepository.getInstance()
Returns the unique ArcRepository instance.
|
void |
ArcRepository.store(RemoteFile rf,
StoreMessage replyInfo)
Stores a file in all known replicas.
|
Constructor and Description |
---|
ArcRepository()
Constructor for the ArcRepository.
|
Modifier and Type | Method and Description |
---|---|
void |
FileBasedActiveBitPreservation.findChangedFiles(Replica replica)
Deprecated.
This method finds out which files in a given bitarchive are misrepresented in the admin data: Either having the
wrong checksum or not being marked as uploaded when it actually is.
|
protected static File |
WorkFiles.getSortedFile(File unsortedFile)
Get a sorted file from an unsorted one, updating if necessary.
|
void |
FileBasedActiveBitPreservation.replaceChangedFile(Replica replica,
String filename,
String credentials,
String checksum)
Deprecated.
Check that file checksum is indeed different to admin data and reference replica.
|
void |
FileBasedActiveBitPreservation.uploadMissingFiles(Replica replica,
String... filenames)
Deprecated.
Check that the files we want to restore are indeed missing on the replica, and present in admin data and the
reference bitarchive.
|
void |
DatabaseBasedActiveBitPreservation.uploadMissingFiles(Replica replica,
String... filenames)
This method is used to upload missing files to a replica.
|
Modifier and Type | Method and Description |
---|---|
BatchStatus |
JMSArcRepositoryClient.batch(FileBatchJob job,
String replicaId,
String batchId,
String... args)
Runs a batch job on each file in the ArcRepository.
|
File |
JMSArcRepositoryClient.correct(String replicaId,
String checksum,
File file,
String credentials)
Method for correcting an entry in a replica.
|
BitarchiveRecord |
JMSArcRepositoryClient.get(String arcfile,
long index)
Sends a GetMessage on the "TheArcrepos" queue and waits for a reply.
|
File |
JMSArcRepositoryClient.getAllChecksums(String replicaId)
Retrieves all the checksum from the replica through a GetAllChecksumMessage.
|
File |
JMSArcRepositoryClient.getAllFilenames(String replicaId)
Retrieves the names of all the files in the replica through a GetAllFilenamesMessage.
|
String |
JMSArcRepositoryClient.getChecksum(String replicaId,
String filename)
Retrieves the checksum of a specific file.
|
void |
JMSArcRepositoryClient.getFile(String arcfilename,
Replica replica,
File toFile)
Synchronously retrieves a file from a bitarchive and places it in a local file.
|
File |
JMSArcRepositoryClient.removeAndGetFile(String fileName,
String bitarchiveId,
String checksum,
String credentials)
Removes a file from the bitarchives, if given credentials and checksum are correct.
|
void |
JMSArcRepositoryClient.store(File file)
Sends a StoreMessage via the synchronized JMS connection method sendAndWaitForOneReply().
|
void |
JMSArcRepositoryClient.updateAdminData(String fileName,
String replicaId,
ReplicaStoreState newval)
Request update of admin data to specific state.
|
Modifier and Type | Method and Description |
---|---|
void |
DerbyEmbeddedSpecifics.backupDatabase(Connection c,
File backupDir)
Backup the database.
|
protected static void |
ReplicaCacheHelpers.insertReplicaIntoDB(Replica rep,
Connection con)
Method for inserting a Replica into the replica table.
|
protected void |
AdminData.read()
Deprecated.
Reads the admin data from a file.
|
protected static void |
ReplicaCacheHelpers.updateReplicaFileInfo(long replicafileinfoGuid,
String checksum,
Date date,
ReplicaStoreState state,
Connection con)
Method for updating an entry in the replicafileinfo table.
|
protected static void |
ReplicaCacheHelpers.updateReplicaFileInfo(long replicafileinfoGuid,
String checksum,
ReplicaStoreState state,
Connection con)
Method for updating an entry in the replicafileinfo table.
|
Modifier and Type | Method and Description |
---|---|
BatchStatus |
Bitarchive.batch(String bitarchiveAppId,
FileBatchJob job)
Run a batch job on all ARC entries in the archive.
|
BitarchiveRecord |
Bitarchive.get(String arcfile,
long index)
Get an ARC or WARC record out of the archive.
|
File |
BitarchiveAdmin.getTemporaryPath(String arcFileName,
long requestedSize)
Returns a temporary place for the the file to be stored.
|
protected boolean |
BitarchiveAdmin.isBitarchiveDirectory(File theDir)
Checks whether a directory is one of the known bitarchive directories.
|
File |
BitarchiveAdmin.moveToStorage(File tempLocation)
Moves a file from temporary storage to file storage.
|
void |
BitarchiveAdmin.updateFileList(File basedir)
Method for updating the filelist for a given basedir.
|
void |
Bitarchive.upload(RemoteFile arcfile,
String fileName)
Upload an ARC file to this archive.
|
Modifier and Type | Method and Description |
---|---|
File |
RemoveAndGetFileMessage.getData()
Writes the the content of the retrieved file into a local file.
|
void |
GetFileMessage.getData(File toFile)
Writes the the content of the retrieved file into a local file.
|
static BitarchiveMonitorServer |
BitarchiveMonitorServer.getInstance()
Returns the unique instance of a BitarchiveMonitorServer.
|
static BitarchiveClient |
BitarchiveClient.getInstance(ChannelID allBaIn,
ChannelID anyBaIn,
ChannelID theBamonIn)
Factory that establish the connection to the server.
|
BatchMessage |
BitarchiveClient.sendBatchJob(ChannelID replyChannel,
FileBatchJob job)
Submit a batch job to the archive.
|
void |
BitarchiveClient.sendUploadMessage(RemoteFile rf,
String precomputedChecksum)
Submit an upload request to the bitarchive.
|
Constructor and Description |
---|
BitarchiveMonitorServer()
Creates an instance of a BitarchiveMonitorServer.
|
HeartBeatSender(ChannelID inReceiver,
BitarchiveServer inBaServer)
Constructs a HearBeatSender that sends heartbeats.
|
Modifier and Type | Method and Description |
---|---|
String |
FileChecksumArchive.calculateChecksum(File f)
Method for calculating the checksum of a file.
|
String |
FileChecksumArchive.calculateChecksum(InputStream is)
Method for calculating the checksum of a inputstream.
|
File |
DatabaseChecksumArchive.correct(String filename,
File correctFile) |
File |
ChecksumArchive.correct(String filename,
File correctFile)
Method for removing a bad entry from the archive.
|
File |
FileChecksumArchive.correct(String filename,
File correctFile)
Method for correcting a bad entry from the archive.
|
File |
FileChecksumArchive.getAllFilenames()
Method for retrieving the names of all the files within the archive as a temporary file.
|
File |
FileChecksumArchive.getArchiveAsFile()
Method for retrieving the archive as a temporary file containing the checksum entries.
|
Modifier and Type | Method and Description |
---|---|
void |
CorrectMessage.getData(File toFile)
Retrieves the content of the remoteFile and writes it into the local file.
|
void |
GetAllChecksumsMessage.getData(File toFile)
Method for retrieving the resulting file.
|
static ChecksumClient |
ChecksumClient.getInstance(ChannelID theCRin)
The method for invoking an instance of this class.
|
RemoteFile |
CorrectMessage.getRemovedFile()
Returns the removed file.
|
Modifier and Type | Method and Description |
---|---|
static void |
LoadDatabaseChecksumArchive.main(String[] args)
Main program for the LoadDatabaseChecksumArchive class
|
Modifier and Type | Method and Description |
---|---|
static File |
BatchGUI.getJarFile(String classPath)
Method for retrieving and validating the arc-file for a given DOOM!
|
static void |
BatchGUI.getPageForClass(javax.servlet.jsp.PageContext context)
Method for creating the page for a batchjob.
|
void |
BatchExecuter.run()
Execution of the batchjob in its own thread (use start() instead).
|
Modifier and Type | Method and Description |
---|---|
void |
TrivialArcRepositoryClient.store(File file)
Store the given file in the ArcRepository.
|
Modifier and Type | Method and Description |
---|---|
static JMSConnection |
JMSConnectionSunMQ.getInstance()
Intialises an Open Message Queue JMS connection.
|
static RemoteFile |
TestRemoteFile.getInstance(File remoteFile,
Boolean useChecksums,
Boolean fileDeletable,
Boolean multipleDownloads) |
static RemoteFile |
FTPRemoteFile.getInstance(File localFile,
Boolean useChecksums,
Boolean fileDeletable,
Boolean multipleDownloads)
Create a remote file that handles the transport of the remote file data.
|
static RemoteFile |
ExtendedFTPRemoteFile.getInstance(File localFile,
Boolean useChecksums,
Boolean fileDeletable,
Boolean multipleDownloads)
This method returns an instance of FTPRemoteFile using the factory method with the same signature in that class.
|
static RemoteFile |
FTPRemoteFile.getInstance(File localFile,
Boolean useChecksums,
Boolean fileDeletable,
Boolean multipleDownloads,
RemoteFileSettings connectionParams) |
static RemoteFile |
ExtendedFTPRemoteFile.getInstance(File localFile,
Boolean useChecksums,
Boolean fileDeletable,
Boolean multipleDownloads,
RemoteFileSettings connectionParams)
This method returns an instance of FTPRemoteFile using the factory method with the same signature in that class.
|
protected void |
JMSConnection.initConnection()
Initializes the JMS connection.
|
void |
JMSConnection.removeListener(ChannelID mq,
javax.jms.MessageListener ml)
Removes the specified MessageListener from the given queue or topic.
|
protected void |
JMSConnection.sendMessage(NetarkivetMessage nMsg,
ChannelID to)
Submit an ObjectMessage to the destination channel.
|
void |
JMSConnection.setListener(ChannelID mq,
javax.jms.MessageListener ml)
Method adds a listener to the given queue or topic.
|
Constructor and Description |
---|
TestRemoteFile(File localFile,
boolean useChecksum,
boolean fileDeletable,
boolean multipleDownloads) |
Modifier and Type | Method and Description |
---|---|
BatchStatus |
LocalArcRepositoryClient.batch(FileBatchJob job,
String replicaId,
String... args)
Runs a batch job on each file in the ArcRepository.
|
File |
LocalArcRepositoryClient.getAllChecksums(String replicaId)
Method for retrieving the checksums of all the files of the replica.
|
File |
LocalArcRepositoryClient.getAllFilenames(String replicaId)
Method for retrieving all the filenames of the replica.
|
void |
LocalArcRepositoryClient.store(File file)
Store the given file in the ArcRepository.
|
void |
PreservationArcRepositoryClient.store(File file)
Store the given file in the ArcRepository.
|
void |
ArcRepositoryClient.store(File file)
Store the given file in the ArcRepository.
|
void |
HarvesterArcRepositoryClient.store(File file)
Store the given file in the ArcRepository.
|
Modifier and Type | Method and Description |
---|---|
void |
SingleMBeanObject.unregister()
Unregister the object from the MBeanServer.
|
Modifier and Type | Method and Description |
---|---|
static void |
FileUtils.copyDirectory(File from,
File to)
Copy an entire directory from one location to another.
|
static String |
Settings.get(String key)
Gets a setting.
|
static String |
StreamUtils.getInputStreamAsString(InputStream in)
Reads an input stream and returns it as a string.
|
static File |
FileUtils.getResourceFileFromClassPath(String filePath)
Loads an file from the class path (for retrieving a file from '.jar').
|
static int |
DBUtils.getTableVersion(Connection connection,
String tablename)
Returns the version of a table according to schemaversions, or 0 for the initial, unnumbered version.
|
static org.dom4j.Document |
XmlUtils.getXmlDoc(File f)
Read and parse an XML-file, and return a Document object representing this object.
|
static File |
FileUtils.makeValidFileFromExisting(String filename)
Makes a valid file from filename passed in String.
|
static byte[] |
FileUtils.readBinaryFile(File file)
Read an entire file, byte by byte, into a byte array, ignoring any locale issues.
|
static void |
XmlUtils.writeXmlToFile(org.dom4j.Document doc,
File f)
Write document tree to file.
|
Modifier and Type | Method and Description |
---|---|
void |
GetMetadataArchiveBatchJob.processRecord(ArchiveRecordBase record,
OutputStream os)
The method for processing the arc-records.
|
Modifier and Type | Method and Description |
---|---|
protected void |
LoadableFileBatchJob.loadBatchJob()
Method for initializing the loaded batchjob.
|
Modifier and Type | Method and Description |
---|---|
static byte[] |
WARCUtils.readWARCRecord(org.archive.io.warc.WARCRecord record)
Read the contents (payload) of an WARC record into a byte array.
|
Modifier and Type | Method and Description |
---|---|
protected void |
LinuxMachine.createApplicationKillScripts(File directory)
Creates the kill scripts for all the applications.
|
protected void |
WindowsMachine.createApplicationKillScripts(File directory)
Creates the kill scripts for all the applications.
|
protected void |
LinuxMachine.createApplicationStartScripts(File directory)
Creates the start scripts for all the applications.
|
protected void |
LinuxMachine.createArchiveDatabaseKillScript(File dir)
Creates a script for killing the archive database on a given machine.
|
protected void |
LinuxMachine.createArchiveDatabaseStartScript(File dir)
Creates a script for starting the archive database on a given machine.
|
protected void |
LinuxMachine.createHarvestDatabaseKillScript(File dir)
Creates a script for killing the harvest database on a given machine.
|
protected void |
LinuxMachine.createHarvestDatabaseStartScript(File dir)
Creates a script for starting the harvest database on a given machine.
|
protected void |
WindowsMachine.createInstallDirScript(File directory)
Function to create the script which installs the new directories.
|
protected void |
Machine.createJmxRemoteAccessFile(File directory)
Creates the jmxremote.password file, based on the settings.
|
protected void |
Machine.createJmxRemotePasswordFile(File directory)
Creates the jmxremote.password file, based on the settings.
|
protected void |
LinuxMachine.createOSLocalKillAllScript(File directory)
This function creates the script to kill all applications on this machine.
|
protected void |
WindowsMachine.createOSLocalKillAllScript(File directory)
This function creates the script to kill all applications on this machine.
|
protected void |
LinuxMachine.createOSLocalStartAllScript(File directory)
This function creates the script to start all applications on this machine.
|
protected void |
WindowsMachine.createOSLocalStartAllScript(File directory)
This function creates the script to start all applications on this machine.
|
protected void |
LinuxMachine.createRestartScript(File dir)
Creates script for restarting all the applications on a machine.
|
protected void |
WindowsMachine.createRestartScript(File dir)
Creates scripts for restarting all the applications on a machine.
|
protected void |
Machine.createSecurityPolicyFile(File directory)
Copy inherited securityPolicyFile to local directory.
|
protected void |
Machine.createSlf4jConfigFiles(File directory)
Creates a the SLF4J config file for every application.
|
protected void |
WindowsMachine.createWaitScript(File dir)
Creates the script for waiting during restart.
|
protected void |
WindowsMachine.windowsStartBatScript(Application app,
File directory)
Creates the batch script for starting the application.
|
protected void |
WindowsMachine.windowsStartVbsScript(Application app,
File directory)
This function creates the VBscript to start the application.
|
Modifier and Type | Method and Description |
---|---|
abstract void |
HarvestChannelDAO.create(HarvestChannel harvestChannel)
Creates a
HarvestChannel object in the storage backend. |
abstract Job |
JobDAO.read(long jobID)
Reads a job from persistent storage.
|
HarvestDefinition |
HarvestDefinitionDBDAO.read(Long harvestDefinitionID)
Read the stored harvest definition for the given ID.
|
abstract HarvestDefinition |
HarvestDefinitionDAO.read(Long harvestDefinitionID)
Read the stored harvest definition for the given ID.
|
abstract void |
HarvestChannelDAO.update(HarvestChannel harvestChannel)
Updates a
HarvestChannel object in the storage backend. |
abstract void |
JobDAO.update(Job job)
Update a Job in persistent storage.
|
void |
H1HeritrixTemplate.writeTemplate(javax.servlet.jsp.JspWriter out) |
void |
H3HeritrixTemplate.writeTemplate(javax.servlet.jsp.JspWriter out) |
void |
H3HeritrixTemplate.writeTemplate(OutputStream os) |
Constructor and Description |
---|
GlobalCrawlerTrapList(InputStream is,
String name,
String description,
boolean isActive)
Construct a new GlobalCrawlerTrapList from an input stream consisting of newline-separated regular expressions.
|
Modifier and Type | Method and Description |
---|---|
void |
ExtendedFieldValueDBDAO.delete(long aExtendedfieldValueID) |
abstract void |
ExtendedFieldDAO.delete(long aExtendedFieldID)
deletes an ExtendedField from persistent storage.
|
void |
ExtendedFieldDBDAO.delete(long aExtendedfieldId) |
abstract ExtendedField |
ExtendedFieldDAO.read(Long aExtendedFieldID)
Reads an ExtendedField from persistent storage.
|
abstract void |
ExtendedFieldDAO.update(ExtendedField aExtendedField)
Update a ExtendedField in persistent storage.
|
void |
ExtendedFieldValueDBDAO.update(ExtendedFieldValue aExtendedFieldValue) |
Modifier and Type | Method and Description |
---|---|
abstract void |
HeritrixLauncher.doCrawl()
Launches the crawl and monitors its progress.
|
static void |
HarvestDocumentation.documentHarvest(IngestableFiles ingestables)
Documents the harvest under the given dir in a packaged metadata arc file in a directory 'metadata' under the
current dir.
|
static void |
HeritrixLauncher.makeTemplateReadyForHeritrix1(HeritrixFiles files)
Updates the diskpath value, archivefile_prefix, seedsfile, and deduplication -information.
|
Modifier and Type | Method and Description |
---|---|
void |
DefaultHeritrixLauncher.doCrawl()
This method launches heritrix in the following way: 1.
|
void |
BnfHeritrixLauncher.doCrawl()
Initializes an Heritrix controller, then launches the Heritrix instance.
|
void |
HeritrixController.requestCrawlStart()
Request that Heritrix start crawling.
|
Modifier and Type | Method and Description |
---|---|
static HarvestControllerServer |
HarvestControllerServer.getInstance()
Returns or creates the unique instance of this singleton The server creates an instance of the HarvestController,
uploads arc-files from unfinished harvests, and starts to listen to JMS messages on the incoming jms queues.
|
void |
HarvestControllerServer.visit(DoOneCrawlMessage msg)
Receives a DoOneCrawlMessage and call onDoOneCrawl.
|
Modifier and Type | Method and Description |
---|---|
static HarvestReport |
HarvestReportFactory.generateHarvestReport(Object... args)
Returns an instance of the default
HarvestReport implementation defined by the setting
HarvesterSettings.HARVEST_REPORT_CLASS . |
Constructor and Description |
---|
BnfHarvestReport(DomainStatsReport dsr)
Constructor for this class.
|
Modifier and Type | Method and Description |
---|---|
abstract void |
HeritrixLauncherAbstract.doCrawl()
Launches the crawl and monitors its progress.
|
static void |
HarvestDocumentation.documentHarvest(IngestableFiles ingestables)
Documents the harvest under the given dir in a packaged metadata arc file in a directory 'metadata' under the
current dir.
|
void |
PostProcessing.doPostProcessing(File crawlDir,
Throwable crawlException)
Do postprocessing of data in a crawldir.
1.
|
static HarvestControllerServer |
HarvestControllerServer.getInstance()
Returns or creates the unique instance of this singleton The server creates an instance of the HarvestController,
uploads arc-files from unfinished harvests, and starts to listen to JMS messages on the incoming jms queues.
|
static void |
HeritrixLauncherAbstract.makeTemplateReadyForHeritrix3(Heritrix3Files files)
This method prepares the crawler-beans.cxml file used by the Heritrix3 crawler.
|
void |
HarvestControllerServer.visit(DoOneCrawlMessage msg)
Checks that we're available to do a crawl, and if so, marks us as unavailable, checks that the job message is
well-formed, and starts the thread that the crawl happens in.
|
Modifier and Type | Method and Description |
---|---|
void |
HeritrixLauncher.doCrawl()
Initializes an Heritrix3controller, then launches the Heritrix3 instance.
|
void |
IHeritrixController.requestCrawlStart()
Request that Heritrix start crawling.
|
Modifier and Type | Method and Description |
---|---|
static HarvestReport |
HarvestReportFactory.generateHarvestReport(Object... args)
Returns an instance of the default
HarvestReport implementation defined by the setting
HarvesterSettings.HARVEST_REPORT_CLASS . |
Modifier and Type | Method and Description |
---|---|
static void |
LuceneUtils.makeDummyIndex(File indexLocation)
Create Dummy Lucene index.
|
protected CrawlDataItem |
CDXOriginCrawlLogIterator.parseLine(String line)
Parse a crawl.log line into a valid CrawlDataItem.
|
Modifier and Type | Method and Description |
---|---|
protected Set<Long> |
IndexRequestClient.cacheData(Set<Long> jobSet)
This method makes sure the actual caching of underlying data is done using the index server.
|
void |
IndexRequestClient.requestIndex(Set<Long> jobSet,
Long harvestId)
Method to request an Index without having the result sent right away.
|
Modifier and Type | Method and Description |
---|---|
void |
JobDispatcher.doOneCrawl(Job job,
String origHarvestName,
String origHarvestDesc,
String origHarvestSchedule,
HarvestChannel channel,
String origHarvestAudience,
List<MetadataEntry> metadata)
Submit an doOneCrawl request to a HarvestControllerServer.
|
Modifier and Type | Method and Description |
---|---|
static String |
TestFileUtils.compareDirsText(File fstDir,
File sndDir)
Compares the content of two directories and report all differences in the returned text string.
|
static void |
TestFileUtils.copyDirectoryNonCVS(File from,
File to)
Copy an entire directory from one location to another, skipping CVS directories.
|
Copyright © 2005–2016 The Royal Danish Library, the Danish State and University Library, the National Library of France and the Austrian National Library.. All rights reserved.