Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 16 Next »

(IN PROGRESS)

At the KB-Denmark Netarkiv we are working on some quite radical changes to our backend architecture - replacing our ArcRepository storage with bitrepository.org software, and implementing a new mass-processing architecture probably based on hadoop. As part of this process we would like to know what parts of NAS are actually in use at our partner institutions so we can develop a strategy for future support. 

NAS Applications

Which of the following NAS applications (services are in use in your production environment?

ApplicationDenmarkFranceAustriaSpainSwedenComments
HarvestControllerServeryy



GUIWebServeryy



HarvestJobManageryy



ChecksumFileServeryn



ViewerProxyyy (but only to access data and metadata files)



WaybackIndexeryn



AggregationWorkeryn



IndexServeryy



ArcRepositoryyy



BitarchiveServeryn



BitarchiveMonitorServeryn



AccessBitarchiveServery/nn


This is a special read-only server which is used in a specific data-extraction system in DK, outside the main Netarkivet installation.

Plugins

Which of the following plugins are used in your production setup? Those marked with a (star) are default values set in the packaged settings file.

InterfaceImplementationDenmarkFranceAustriaSpainSweden
AbstractRemoteFileHTTPRemoteFile





HTTPSRemoteFile





FTPRemoteFile (star)y




ActiveBitPreservationDatabaseBasedActiveBitPreservation





FileBasedActiveBitPreservation (star)y




Admin

UpdateableAdminData







DatabaseAdmin (star)

y




arcrepositoryadmin.DBSpecifics

DerbyServerSpecifics (star)







DerbyEmbeddedSpecifics







MySQLSpecifics





PostgreSQLSpecifics

y




ChecksumArchive

FileChecksumArchive 

y




DatabaseChecksumArchive







JMSConnectionJMSConnectionSunMQ (star)y




ArcRepositoryClientJMSArcRepositoryClienty




LocalArcRepositoryClient





MonitorRegistryClient

PrintMonitorRegistryClient







JMSMonitorRegistryClient (star)y




JobIndexCacheIndexRequestClient  (star)y




Notifications

EMailNotifications (star)

y




PrintNotifications







FreeSpaceProvider

DefaultFreeSpaceProvider (star)

y




FreeSpaceProvider







OnbFreeSpaceProvider







datamodel.DBSpecifics



DerbyServerSpecifics (star)







DerbyEmbeddedSpecifics







MySQLSpecifics







PostgreSQLSpecifics

y




JobGenerator

DefaultJobGenerator (star)

y




FixedDomainConfigurationCountJobGenerator







ArchiveFileNaming

LegacyNamingConvention (star)

y




CollectionPrefixNamingConvention







FrontierReportFilter

TopTotalEnqueuesFilter (star)

y




ExhaustedQueuesFilter







MaxSizeFrontierReportExtract







RetiredQueuesFilter







HeritrixLauncherAbstractHeritrixLauncher (star)y




IHeritrixControllerHeritrixController (star)y




HarvestReport

LegacyHarvestReport (star)

y




BnFHarvestReport







IndexRequestServerInterfaceIndexRequestServer (star)y





Command Line Tools

Over the years, the NetarchiveSuite codebase has accumulated a lot of command line utilities. Some of these were probably developed for a single specialised use-case or for test purposes, but others may have become part of the normal workflow at the various repositories. Here is a partial list of those that look most likely to be of general interest. Please mark any of those you know of that are used as part of your workflows.

ToolPurposeDenmarkFranceAustriaSpainSweden
DeployApplicationCreates deploy scripts from a deploy-configy




HarvestdatabaseUpdateApplicationUpdates HarvestDB schemay




BuildCompleteSettingsMerges module settings files in NAS to one large global default settings file. Run as part of release process.y




GetFileRetrieves a file via the ArcRepository interface





GetRecordRetrieves a (w)arc-record via the ArcRepository interface





LoadDatabaseChecksumArchiveMigration tool from file-based checksums to database-based checksums





ReestablishAdminDatabaseFor reestablishing the admin database from a 'admin.data' file





RunBatchRuns a batch job from the command line





UploadUploads a file to the ArcRepository from the command line. (Handy for testdata.)y




ReestablishAdminDatabase

Should be deprecated (question) Reads old admin.data file.





ClassDependenciesNon NAS Utility (license is not ours)





CreateIndexCLI to talk to IndexServer via IndexClient





RunChecksumCLI to get all checksums from a Bitarchive (deprecated)





SendDedupIndexRequestToIndexserver

Asynchronously starts a dedup indexing on an IndexServer and then exits. Tue Hejlskov Larsen is this what you use to generate deduplication indexes?







MakeIndexRuns a CDX extraction on a single file in a remote ArcRepository





FindRelevantCrawllogLinesFinds crawl-log lines matching a given domain name in a local metadata file





JMXProxy"This tool will simply reregister all MBeans that matches the given query from the JMX hosts read in settings, using* its own platformmbeanserver. It will then wait forever."





DeduplicateToCDXApplicationExtracts CDX records for deduplicate annotations from a local crawl log file





ResetFailedFilesUtility for WaybackIndexer to reset files that have failed more than 3 times so they can be retried





ARCReaderUtilsSplits an arcfile (not warc) and dumps results to a directory





ArcWrapCreates an arcfile by wrapping a file





ExtractCDXExtracts CDX records, unsorted, from a list of local input arcfiles (not warcs)





JMSBrokerChecks that a JMS broker (as specified in NAS settings) is up and running.





WriteBytesToFileJust creates large files full of null bytes





FTPValidator

Tests if an ftp server configuration in a NAS settings file points to a NAS-compliand ftp server.





ArcMergeMerges several arcfiles into one arcfile





ArchiveExtractCDX

Extracts CDX records, unsorted, from a list of local input (w)arcfiles





WARCExtractCDXExtracts CDX records, unsorted, from a list of local input warcfiles





ReformatTranslationFilei) reorders a translation file so keys are in the same order as a reference file, and ii) allows the encoding of the output file to be changed





MailValidatorChecks the validity of a mail-server configured in NAS settings by sending a test-mail





MakeNewMetadataFileCreates a metadata file. For use when postprocessing fails. Is this used?





FindDomainsForCrawllogExtraction?





CheckDuplicateReductionValidates deduplication by comparing a crawl log with a collection of arcfiles. (not warc)





StandaloneApplicationReducedCreates a standalone NetarchiveSuite in a single JVM





MigrateDefaultHarvestDatabaseThis just initialises a SiteSection object which is supposed to upgrade the harvest database as a side-effect





CreateCDXMetadataFileComplex tool that takes a set of filenames and runs a batch job to extracts the cdx'es from each files and pack them in a metadata arc or warc file, one record per input file





HarvesterQueueControlTool to count the number of messages in a given JMS queue





HarvestDatabaseValidatorValidates whether you can connect to the harvest database with the settings in a given settings file





HarvestTemplateApplicationUtility for uploading and updating heritrix templatesy (in test)




CheckDomainCrawltrapsRuns through all domains in the harvest database and checks whether each crawlertrap regexp can validly be included as text-content in an xml documenty




CheckTrapsInFile

Runs through a list of crawler-trap regexes in a fileand checks whether each crawlertrap regex can validly be included as text-content in an xml documen

y(?)








  • No labels