Dashboard

Welcome Message

Welcome to Atlassian Crucible with Fisheye .

To report a bug or suggest a feature please raise an issue in our public JIRA.

Customize this welcome note.

Support

You are running Crucible 4.7.0 with Fisheye 4.7.0 (20190214100246 2019-02-14)

If you are experiencing any issues, please contact your administrator. You may also want to browse the documentation or search the support forums. Still stuck? You can start a support ticket at support.atlassian.com or raise a bug or feature request at jira.atlassian.com.

Customize the support message.

Friday 20 Nov
3:51 pm

Rasmus Bohl Kristensen

Fixed conflicts with fileresolver

Options
    • -5
    • +17
    /common/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
    • -32
    • +0
    /common/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopFileUtils.java
    • -2
    • +19
    /common/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java
    • -1
    • +1
    /common/common-core/src/test/java/dk/netarkivet/common/utils/warc/WarcRecordClientTest.java
    • -5
    • +0
    /harvester/harvest-scheduler/pom.xml
    • -27
    • +28
    /harvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java
    • -3
    • +101
    /harvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java
    • -16
    • +16
    /harvester/heritrix3/heritrix3-bundler/pom.xml
    • -1
    • +1
    /harvester/heritrix3/heritrix3-extensions/pom.xml
    • -0
    • +1
    /harvester/history-gui/pom.xml
    • -0
    • +1
    /pom.xml
    • -39
    • +13
    /wayback/wayback-indexer/pom.xml
    • -1
    • +1
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/batch/UrlCanonicalizerFactory.java
    • -26
    • +110
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/CDXIndexer.java
    • -9
    • +6
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/CDXMapper.java
  16. … 3 more files in changeset.
1:44 pm

Rasmus Bohl Kristensen

Review changes https://sbforge.org/fisheye/cru/CR-NAS-393, changes to uber-jar set up and small pom fixes

Options
    • -3
    • +3
    /common/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
    • -2
    • +3
    /common/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java
    • -0
    • +105
    /hadoop-uber-jar/pom.xml
    • -5
    • +0
    /harvester/harvest-scheduler/pom.xml
    • -6
    • +0
    /harvester/harvester-core/pom.xml
    • -0
    • +1
    /harvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java
    • -2
    • +2
    /harvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java
    • -0
    • +152
    /harvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/MetadataCDXMapper.java
    • -16
    • +16
    /harvester/heritrix3/heritrix3-bundler/pom.xml
    • -1
    • +1
    /harvester/heritrix3/heritrix3-extensions/pom.xml
    • -0
    • +1
    /harvester/history-gui/pom.xml
    • -0
    • +1
    /pom.xml
    • -39
    • +0
    /wayback/wayback-indexer/pom.xml
    • -12
    • +9
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/CDXIndexer.java
    • -18
    • +4
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/CDXMapper.java
  16. … 4 more files in changeset.
9:04 am

Rasmus Bohl Kristensen

Thursday 19 Nov
2:20 pm

Peter Christiansen

Just save it for further improvement

Options
    • -0
    • +131
    /archive/archive-core/src/main/java/dk/netarkivet/archive/arcrepository/distribute/JMSBitmagArcRepositoryClient.java
Monday 16 Nov
11:25 am

csrster <csr@statsbiblioteket.dk>

Merge remote-tracking branch 'origin/bitmag' into bitmag

# Conflicts:

# common/common-core/src/main/java/dk/netarkivet/common/utils/warc/WarcRecordClient.java

Options
    • -2
    • +3
    /common/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
    • -1
    • +1
    /harvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java
    • -4
    • +4
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
11:19 am

csrster <csr@statsbiblioteket.dk>

Merge branch 'NARK-1987-httpclientpooling' into NARK-1962-FileResolverClient

# Conflicts:

# common/common-core/src/main/java/dk/netarkivet/common/utils/warc/WarcRecordClient.java

# common/common-core/src/test/java/dk/netarkivet/common/utils/warc/WarcRecordClientTester.java

Options
    • -78
    • +46
    /common/common-core/src/main/java/dk/netarkivet/common/utils/warc/WarcRecordClient.java
    • -2
    • +1
    /common/common-core/src/test/java/dk/netarkivet/common/utils/warc/WarcRecordClientTest.java
11:16 am

csrster <csr@statsbiblioteket.dk>

Merge branch 'bitmag' into NARK-1962-FileResolverClient

Options
    • -12
    • +9
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
Friday 13 Nov
9:46 am

Rasmus Bohl Kristensen

Ah the problem was that I for some reason had two different 'testdata' directories in netarchivesuite-docker-compose and only one of them was mounted in the container - my test files were in the wr...

Ah the problem was that I for some reason had two different 'testdata' directories in netarchivesuite-docker-compose and only one of them was mounted in the container - my test files were in the wrong one.

But maybe the test files should be added to git? One of the files is already added, so I'm not sure why the others aren't.

Thursday 12 Nov
4:36 pm

Rasmus Bohl Kristensen

This last test fails for me. A NullPointer is thrown at line 253 because getBitarchiveRecord(..) at line 251 returns null.

This last test fails for me. A NullPointer is thrown at line 253 because getBitarchiveRecord(..) at line 251 returns null.

4:08 pm

Rasmus Bohl Kristensen

4:08 pm

Rasmus Bohl Kristensen

Wednesday 11 Nov
9:09 am

Colin Rosenthal

9:09 am

Rasmus Bohl Kristensen

9:07 am

Colin Rosenthal

8:20 am

Colin Rosenthal

Note: this is now the correct way to initialise a FileResolver since we now have two different implementations.

Note: this is now the correct way to initialise a FileResolver since we now have two different implementations.

8:16 am

Colin Rosenthal

Note: this is going to cause some reworking in Rasmus Bohl Kristensen's issue NARK-1973

Note: this is going to cause some reworking in Rasmus Bohl Kristensen's issue NARK-1973

8:15 am

Colin Rosenthal

This is now fixed, but in the NARK-1987-httpclientpooling branch. General Comment: we have too many active issues/branches !!! https://sbforge.org/fisheye/static/pmwukm/2static/images/wiki/icons/e...

This is now fixed, but in the NARK-1987-httpclientpooling branch.

General Comment: we have too many active issues/branches !!!

8:04 am

Colin Rosenthal

8:02 am

Colin Rosenthal

Yes, do that. Making harvester-core depend on wayback classes looks wrong.

Yes, do that. Making harvester-core depend on wayback classes looks wrong.

8:01 am

Colin Rosenthal

This is going to have to be merged up with the new way of getting a FileResolver from the NARK-1962-FileResolverClient branch. This should be an extra JIRA task.

This is going to have to be merged up with the new way of getting a FileResolver from the NARK-1962-FileResolverClient branch. This should be an extra JIRA task.

7:55 am

Colin Rosenthal

Do you have a test with a metadata file and METADATA_DO_DEDUP=true ?

Do you have a test with a metadata file and METADATA_DO_DEDUP=true ?

Friday 06 Nov
11:37 am

csrster <csr@statsbiblioteket.dk>

Tidied up logic in client and tests

Options
    • -40
    • +30
    /common/common-core/src/main/java/dk/netarkivet/common/utils/warc/WarcRecordClient.java
    • -2
    • +1
    /common/common-core/src/test/java/dk/netarkivet/common/utils/warc/WarcRecordClientTest.java
    • -71
    • +30
    /common/common-core/src/test/java/dk/netarkivet/common/utils/warc/WarcRecordClientTester.java
9:13 am

Colin Rosenthal

well ... the problem is that the 3rd API needs a signal that the incoming stream is either a warc-record or an arc-record and this can be achieved by just giving it a filename with the appropriate ...

well ... the problem is that the 3rd API needs a signal that the incoming stream is either a warc-record or an arc-record and this can be achieved by just giving it a filename with the appropriate suffix in the constructor. But there needs to be a comment in the code to this effect [kfc]

Thursday 05 Nov
4:44 pm

Rasmus Bohl Kristensen

NARK-1973 - Fetching of Metadata/CDX
NARK-1973 - Fetching of Metadata/CDX
4:44 pm

Rasmus Bohl Kristensen

Issue here: https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1973

Issue here: https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1973

4:41 pm

Rasmus Bohl Kristensen

I was a bit unsure if it was okay to link the two modules this way. I figure an alternative could be to move the CDXMapper/-Indexer classes to the common-core module?

I was a bit unsure if it was okay to link the two modules this way. I figure an alternative could be to move the CDXMapper/-Indexer classes to the common-core module?

4:34 pm

Rasmus Bohl Kristensen

4:16 pm

Rasmus Bohl Kristensen

Merge branch 'NARK-1954-filenameharvester-reimplementation' into bitmag

Options
    • -4
    • +4
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
4:16 pm

Rasmus Bohl Kristensen

Setting fix from review https://sbforge.org/jira/browse/NARK-1954

Options
    • -2
    • +3
    /common/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
    • -1
    • +1
    /harvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java
    • -4
    • +1
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
    • -2
    • +2
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/FileNameHarvester.java
    • -1
    • +1
    /wayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/WaybackIndexer.java
4:03 pm

Rasmus Bohl Kristensen

Added setting for new job input/output dirs and more logging

Options
    • -2
    • +14
    /common/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
    • -19
    • +18
    /harvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java
    • -5
    • +5
    /harvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java