Dashboard

added FAILED check to JMSBitmagArcRepositoryClient.java

Fixed conflicts with fileresolver

    • -39
    • +13
    /wayback/wayback-indexer/pom.xml
  1. … 3 more files in changeset.
Review changes https://sbforge.org/fisheye/cru/CR-NAS-393, changes to uber-jar set up and small pom fixes

    • -0
    • +105
    /hadoop-uber-jar/pom.xml
  1. … 4 more files in changeset.
Just save it for further improvement

Merge remote-tracking branch 'origin/bitmag' into bitmag

# Conflicts:

# common/common-core/src/main/java/dk/netarkivet/common/utils/warc/WarcRecordClient.java

Merge branch 'NARK-1987-httpclientpooling' into NARK-1962-FileResolverClient

# Conflicts:

# common/common-core/src/main/java/dk/netarkivet/common/utils/warc/WarcRecordClient.java

# common/common-core/src/test/java/dk/netarkivet/common/utils/warc/WarcRecordClientTester.java

Merge branch 'bitmag' into NARK-1962-FileResolverClient

Ah the problem was that I for some reason had two different 'testdata' directories in netarchivesuite-docker-compose and only one of them was mounted in the container - my test files were in the wr...

Ah the problem was that I for some reason had two different 'testdata' directories in netarchivesuite-docker-compose and only one of them was mounted in the container - my test files were in the wrong one.

But maybe the test files should be added to git? One of the files is already added, so I'm not sure why the others aren't.

This last test fails for me. A NullPointer is thrown at line 253 because getBitarchiveRecord(..) at line 251 returns null.

This last test fails for me. A NullPointer is thrown at line 253 because getBitarchiveRecord(..) at line 251 returns null.

Note: this is now the correct way to initialise a FileResolver since we now have two different implementations.

Note: this is now the correct way to initialise a FileResolver since we now have two different implementations.

Note: this is going to cause some reworking in Rasmus Bohl Kristensen's issue NARK-1973

Note: this is going to cause some reworking in Rasmus Bohl Kristensen's issue NARK-1973

This is now fixed, but in the NARK-1987-httpclientpooling branch. General Comment: we have too many active issues/branches !!! https://sbforge.org/fisheye/static/pmwukm/2static/images/wiki/icons/e...

This is now fixed, but in the NARK-1987-httpclientpooling branch.

General Comment: we have too many active issues/branches !!!

Yes, do that. Making harvester-core depend on wayback classes looks wrong.

Yes, do that. Making harvester-core depend on wayback classes looks wrong.

This is going to have to be merged up with the new way of getting a FileResolver from the NARK-1962-FileResolverClient branch. This should be an extra JIRA task.

This is going to have to be merged up with the new way of getting a FileResolver from the NARK-1962-FileResolverClient branch. This should be an extra JIRA task.

Do you have a test with a metadata file and METADATA_DO_DEDUP=true ?

Do you have a test with a metadata file and METADATA_DO_DEDUP=true ?

Tidied up logic in client and tests

well ... the problem is that the 3rd API needs a signal that the incoming stream is either a warc-record or an arc-record and this can be achieved by just giving it a filename with the appropriate ...

well ... the problem is that the 3rd API needs a signal that the incoming stream is either a warc-record or an arc-record and this can be achieved by just giving it a filename with the appropriate suffix in the constructor. But there needs to be a comment in the code to this effect [kfc]

NARK-1973 - Fetching of Metadata/CDX
NARK-1973 - Fetching of Metadata/CDX
Issue here: https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1973

Issue here: https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1973

I was a bit unsure if it was okay to link the two modules this way. I figure an alternative could be to move the CDXMapper/-Indexer classes to the common-core module?

I was a bit unsure if it was okay to link the two modules this way. I figure an alternative could be to move the CDXMapper/-Indexer classes to the common-core module?

Merge branch 'NARK-1954-filenameharvester-reimplementation' into bitmag