NetarchiveSuite-Github

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Most of the followup nr 2 to https://sbforge.org/fisheye/cru/CR-NAS-372

Shut up in tests

HadoopProcessorRepo Work In Progress

    • -0
    • +119
    /archive/archive-core/src/test/resources/core-site.xml
    • -0
    • +333
    /archive/archive-core/src/test/resources/hdfs-site.xml
    • -0
    • +663
    /archive/archive-core/src/test/resources/yarn-site.xml
hadoop test job

    • -0
    • +60
    /archive/test-hadoop-job/pom.xml
HadoopBatchJob work in progress

HadoopProcessorRepository almost done and everything compiles. Added the exists() method on RemoteFile. Split BatchStatus into ClassicalBatchStatus and HadoopBatchStatus

  1. … 11 more files in changeset.
Better closing of the new Clients, I think

Small changes to CSRs work with generic jobs and other generics

Small corrections.

Reimplemented the mixed client using the split interfaces. Ready for testing I think.

Refactored to separate out different sub-interfaces from ArcRepository

  1. … 14 more files in changeset.
It builds. FailedFiles are URI. Generic ArcRepositoryClient

  1. … 35 more files in changeset.
Hadoop step 1.1

    • -19
    • +19
    /wayback/wayback-indexer/pom.xml
Merge branch 'hadoop' into local_wayback_indexer

# Conflicts:

# pom.xml

Hadoop step 1

Extracted Interface BatchJob from FileBatchJob

  1. … 8 more files in changeset.
Fixed a lot of little things in local-wayback-indexer

    • -0
    • +20
    /local-wayback-indexer/README.md
    • -0
    • +45
    /local-wayback-indexer/pom.xml
    • -0
    • +42
    /local-wayback-indexer/src/main/java/LocalWaybackIndexerRunner.java
    • -0
    • +192
    /local-wayback-indexer/src/main/resources-filtered/settings_WaybackIndexerApplication.xml
    • -0
    • +22
    /local-wayback-indexer/src/main/resources/logback.xml
    • -18
    • +0
    /local_wayback_indexer/assembly.xml
    • -4
    • +0
    /local_wayback_indexer/port_forward.sh
    • -21
    • +0
    /local_wayback_indexer/src/main/java/LocalWaybackIndexerRunner.java
    • -189
    • +0
    /local_wayback_indexer/src/main/resources/settings_WaybackIndexerApplication.xml
Module to run wayback-indexer locally, but connected to kb-test-adm-001 TEST1 setup

    • -0
    • +22
    /local_wayback_indexer/README.md
    • -0
    • +18
    /local_wayback_indexer/assembly.xml
    • -0
    • +61
    /local_wayback_indexer/pom.xml
    • -0
    • +4
    /local_wayback_indexer/port_forward.sh
    • -0
    • +21
    /local_wayback_indexer/src/main/java/LocalWaybackIndexerRunner.java
    • -0
    • +27
    /local_wayback_indexer/src/main/resources/logback.xml
    • -0
    • +189
    /local_wayback_indexer/src/main/resources/settings_WaybackIndexerApplication.xml
Documented how to use narcana-suite01

Document about simple test that narcana works

Follow-ups following code review: https://sbforge.org/fisheye/cru/CR-NAS-371

Modified to handle case where bitarchive is a symbolic link.

Document about hdfs3 file security

    • -0
    • +19
    /mass-processing/HDFS File Security.md
Added a check that the CollectionID is known to the bitrepository.

Set CollectionID for upload to default to the NAS environment name.

Moved initialisation of bitrepository connection to first upload, so viewer instances don't need a valid bitmag config.

Fixed merge mess-up in pomfile.