NetarchiveSuite-Webdanica

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fix for WEBDAN-270 - changed the use of a ignoredProtocols list to a acceptedProtocols list

  1. … 8 more files in changeset.
Follow-up after initial accept-test. loadDomains.sh now uses --accepted argument like loadSeeds . PHOENIX_CLIENT_JAR now added to setenv.sh and used by ingestTool.sh. Error in loadDomains.sh fixed. Updates to installation manual after being used to install webdanica in PROD

  1. … 6 more files in changeset.
Forgot to add the improved version of RunNetarchiveSuite.sh

Fixed WEBDAN-266 - now checks the number of NetarchiveSuite apps alive as well

Bumped branch 2.1 to version 2.1-SNAPSHOT

WEBDAN-262 - removed the logging of the ignored cdx-lines. It gives too much noise

Bumped version to 2.0-RC4

Final fix for WEBDAN-262 - the logic in the SingleSeedHarvest.isRecordForJob() method was faulty. An unittest has been added for that method as well

    • -0
    • +288
    /webdanica-core/src/test/resources/batch_result_for_jobid_15.txt
Describing remedy for WEBDAN-263 in workflow install-guide, and remove output if no harvestlogs found

    • -1
    • +5
    /workflow-template/webdanica-analysis-cron.sh
Fix for WEBDAN-262 - changed the way we split up the record key to find the 'jobid' value

Fix for WEBDAN-262 - getReports return too much data

Work on WEBDAN-261 - made a LocalArcRepositoryClientReadableOnly class that shouldn't write privileges

Fixed critical bug WEBDAN-252 - now we check, that settings.common.tempDir exists and is writable by tomcat

Added fixed WEBDAN-257 and updated the bundled crontabs in scripts/cronjobs

    • -2
    • +2
    /workflow-template/verify_pig_bootup.sh
More corrections to the installations manual. moved the sample crontabs to the cronjobs folder

    • -0
    • +12
    /scripts/cronjobs/crontab.test
    • -0
    • +11
    /scripts/cronjobs/crontab.webdanica
Simplified zipball layout. Moved 'scripts' folder including cronjobs to root-folder 'scripts'

    • -0
    • +11
    /scripts/cassandra/create_blacklists.txt
    • -0
    • +68
    /scripts/cassandra/create_criteriaresults.txt
    • -0
    • +19
    /scripts/cassandra/create_harvests.txt
    • -0
    • +25
    /scripts/cassandra/create_ingestlog.txt
    • -0
    • +2
    /scripts/cassandra/create_keyspace.txt
    • -0
    • +28
    /scripts/cassandra/create_seeds.txt
    • -0
    • +23
    /scripts/cronjobs/check_apps_alive.sh
    • -0
    • +14
    /scripts/cronjobs/cleanup_oldjobs.sh
    • -0
    • +2
    /scripts/cronjobs/rebootTomcat
  1. … 68 more files in changeset.
upgraded maven-war-plugin and maven-assembly-plugin to latest version 3.1.0

Removed obsolete templates folder

    • -20
    • +0
    /templates/criteriaRun-combinedCombo-seq.pig
    • -32
    • +0
    /templates/log4j_hadoop-pig.properties
    • -13
    • +0
    /templates/parse-text-extraction.README
    • -13
    • +0
    /templates/parse-text-extraction.sh
    • -20
    • +0
    /templates/scripts/criteriaRun-combo-v1-seq.pig
    • -32
    • +0
    /templates/scripts/criteriaRun-combo.pig
    • -20
    • +0
    /templates/scripts/criteriaRun-comboNov-v1-seq.pig
  1. … 23 more files in changeset.
Minor updates to the automatic workflow. Copied some scripts from workflow folder to tools folder and previously named automatic-workflow folder, which has now been renamed as workflow-template, as it can be used for both a manual and automatic workflow

    • -34
    • +0
    /automatic-workflow/automatic.README
    • -118
    • +0
    /automatic-workflow/automatic.sh
    • -34
    • +0
    /automatic-workflow/conf/.pigbootup
    • -25
    • +0
    /automatic-workflow/conf/log4j.properties.pig
    • -32
    • +0
    /automatic-workflow/conf/log4j_hadoop-pig.properties
    • -5
    • +0
    /automatic-workflow/conf/silent_logback.xml
    • -202
    • +0
    /automatic-workflow/conf/webdanica_settings.xml
    • -52
    • +0
    /automatic-workflow/criteria-workflow-alt.sh
    • -52
    • +0
    /automatic-workflow/criteria-workflow.sh
    • -37
    • +0
    /automatic-workflow/findharvestlogs.sh
    • -80
    • +0
    /automatic-workflow/harvestlog-1470223608175.txt
  1. … 130 more files in changeset.
Fixed WEBDAN-256 - replaced use of HarvestStatusQuery with my own SQL query - plus additional logentries

  1. … 3 more files in changeset.
Improved logging for harvesting from tomcat

Improved logging in HarvestWorkThread and deleted some deprecated classes

Added update to WEDBAN-214 - check if counts > 0 before writing to stdout

Fixed primarily various inconsistencies in the tools folder

WEBDAN-245 corrected the mention of Java 7 as build/rum requirement to a Java 8 build/run requirement

Added a fix for WEBDAN-245 - now checks if java version is at least 8. Otherwise the webapp stops the deployment

Release of 2.0-RC2

Yet more changes to documentation