Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Added callback support to read and modify the ARC/WARC headers before processing the payload.
    • -0
    • +7
    ./java/org/jwat/archive/ArchiveParser.java
    • -0
    • +39
    ./java/org/jwat/archive/ArchiveRecordParserCallback.java
  1. … 6 more files in changeset.
Improve support for empty files and errors at the end of (W)ARC files in the ArchiveParser/ArchiveParserCallback.
    • -2
    • +2
    ./java/org/jwat/archive/ArchiveParser.java
    • -5
    • +5
    ./java/org/jwat/archive/FileIdent.java
  1. … 1 more file in changeset.
CR-JWAS-33: Follow-up on review.
    • -2
    • +8
    ./java/org/jwat/archive/ArchiveParser.java
    • -3
    • +12
    ./java/org/jwat/archive/Cloner.java
    • -2
    • +8
    ./java/org/jwat/archive/FileIdent.java
    • -4
    • +18
    ./java/org/jwat/archive/ManagedPayload.java
  1. … 70 more files in changeset.
JWAT-77: Unit tests and bug fixes for newly implemented ArcFileWriter/WarcFileWriter and related classes.

JWAT-76: Fix for archiveLengthStr/contentLengthStr set and archiveLength/contentLength null when using payload length validation.

Removed alot of tags and replaced with spaces. (Company policy)

Minor code cleanup.

    • -180
    • +180
    ./java/org/jwat/archive/ArchiveParser.java
    • -87
    • +87
    ./java/org/jwat/archive/Cloner.java
    • -91
    • +91
    ./java/org/jwat/archive/FileIdent.java
    • -260
    • +260
    ./java/org/jwat/archive/ManagedPayload.java
  1. … 91 more files in changeset.
Merge in files from jwat-tools with history.
    • -9
    • +47
    ./java/org/jwat/archive/ArchiveParser.java
    • -1
    • +18
    ./java/org/jwat/archive/Cloner.java
    • -1
    • +19
    ./java/org/jwat/archive/FileIdent.java
    • -4
    • +22
    ./java/org/jwat/archive/ManagedPayload.java
  1. … 9 more files in changeset.
JWAT-78: PayloadManager in JWAT-Tools seems to have a bug related to the closing of the RandomAccessFile and a non null tmpfile object.

Added some unit tests of most common classes.

Tweaked some constant definitions.

    • -26
    • +41
    ./java/org/jwat/archive/ManagedPayload.java
  1. … 5 more files in changeset.
Make buffer sizes configurable in PayloadManager and ByteArrayIOStream.

Added new ManagedPayloadManager to support this.

    • -17
    • +20
    ./java/org/jwat/archive/ManagedPayload.java
    • -0
    • +52
    ./java/org/jwat/archive/ManagedPayloadManager.java
  1. … 1 more file in changeset.
Minor tweaks. Changed version to 0.6.0-SNAPSHOT. Deployed to maven central from now on.
    • -1
    • +0
    ./java/org/jwat/archive/ArchiveParser.java
    • -2
    • +2
    ./java/org/jwat/archive/FileIdent.java
Changed manageRecord back from private to public since it was used after all.
    • -1
    • +1
    ./java/org/jwat/archive/ManagedPayload.java
Adding a containermd task to create the containerMD represnetation of an arc or a warc file.

Use the 1.0.2 version of the jwat core libraries.

Correct the usage of identified payload by closing the handle in the case where a temporary file has to be created (large files) and delete them at the end.

    • -5
    • +16
    ./java/org/jwat/archive/ManagedPayload.java
Changed JWAT dependencies from 1.0.1-SNAPSHOT to 1.0.1.
    • -1
    • +1
    ./java/org/jwat/archive/ManagedPayload.java
Work in progress:

Git style help command.

Improved multithreading for some tasks.

Support for linux file identification.

Rewrote arc2warc to support multiple filedesc records and payload repair.

ManagedPayload added for reloading of payload in different validators.

Fully implemented 2 step XML validation.

Improved FileIdent based on file name and stream peeking.

    • -2
    • +2
    ./java/org/jwat/archive/ArchiveParser.java
    • -0
    • +112
    ./java/org/jwat/archive/Cloner.java
    • -13
    • +36
    ./java/org/jwat/archive/FileIdent.java
    • -0
    • +344
    ./java/org/jwat/archive/ManagedPayload.java
  1. … 3 more files in changeset.
Corrected a spelling mistake in JWAT API.
    • -2
    • +2
    ./java/org/jwat/archive/ArchiveParser.java
Improved various tasks.
    • -3
    • +6
    ./java/org/jwat/archive/FileIdent.java
Changed to 0.5.5-SNAPSHOT. Changed the Command Line Interface.
    • -0
    • +2
    ./java/org/jwat/archive/FileIdent.java
Cleanup and refactoring.
    • -6
    • +7
    ./java/org/jwat/archive/ArchiveParser.java
changed some classes to allow for different recordheader/payloadheader maximum sizes.

Temporarily set to 1024k.

    • -1
    • +14
    ./java/org/jwat/archive/ArchiveParser.java
Added reusable ArchiveParser, changed some stuff to work with GUI. Almost complete parallelized CDX'er.
    • -0
    • +187
    ./java/org/jwat/archive/ArchiveParser.java
    • -0
    • +23
    ./java/org/jwat/archive/ArchiveParserCallback.java
    • -0
    • +2
    ./java/org/jwat/archive/FileIdent.java
Refactoring for use with JWAT-Tools GUI.

Forgot to add env.cmd and some other files.

    • -0
    • +84
    ./java/org/jwat/archive/FileIdent.java