SuccessChanges

Summary

  1. Review changes https://sbforge.org/fisheye/cru/CR-NAS-393, changes to (commit: bf5e943440e3760f4e25662ffcf603c3a78c0b2e) (details)
  2. First attempt at a command-line metadata extraction job. (commit: 8183749da409edb2229dabb86b9e2fbaca8a182d) (details)
  3. Downgraded hadoop to stable 3.2.2 (commit: 2dd04f160af25e4c61fcdc14c697af66e40b3ed7) (details)
  4. Created an invoker-module to prevent the job from including all the (commit: fffd21b9e21b9572899722516c0c6e10a16db15c) (details)
  5. Create FileSystem with newInstance and close it afterwards. DO NOT CLOSE (commit: b006660cc04ac3f6c2442dc0d09b74b6c017c9c9) (details)
  6. Added an extra sanity check in the run.sh script. (commit: dae7ccb17d1e55a49039a7388ff60797959fc609) (details)
  7. Modified to support dynamic identification of the correct file-system (commit: dd68c05afd42a7a8beb6c10dde0a134b2a9b47a4) (details)
  8. Refactoring to make MetadataIndexingApplication closer to a reusable (commit: 48366f7a72262d6f1442b635b466a2b929b9bcd3) (details)
  9. Refactored to use login mechanism instead of doAs. (commit: dc8b3564c4d580f0a8078d58514296c11b01dbce) (details)
  10. Added harvester-core to uber jar (commit: 2214312e8db00065089f6ee1da3811cff20a181f) (details)
  11. Small guava pom change (commit: e25e3e3cc198dd3e350f4accd2ce268af0393687) (details)
  12. Corrected internal versions (commit: d5e904f8602621fb085dc9d52d2bf6a31992b8d9) (details)
  13. More fixed versions (commit: e7b9a3383dc1ff2f5b4d6cfbd0ec7a8e3943d7ac) (details)
  14. Follow up to own review comments (commit: 681d1147f7824aa7a18804c4b1d9475ce9c26c19) (details)
Commit bf5e943440e3760f4e25662ffcf603c3a78c0b2e by Rasmus Bohl Kristensen (rbkr)
Review changes https://sbforge.org/fisheye/cru/CR-NAS-393, changes to
uber-jar set up and small pom fixes
 (commit: bf5e943440e3760f4e25662ffcf603c3a78c0b2e)
The file was modifiedharvester/history-gui/pom.xml
The file was modifiedwayback/wayback-indexer/pom.xml
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/CDXMapper.java
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java
The file was modifiedpom.xml
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
The file was modifiedharvester/harvester-core/pom.xml
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/CDXIndexer.java
The file was modifiedwayback/wayback-indexer/src/test/java/dk/netarkivet/wayback/hadoop/CDXJobTest.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
The file was addedhadoop-uber-jar/pom.xml
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java
The file was addedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/MetadataCDXMapper.java
The file was modifiedharvester/harvest-scheduler/pom.xml
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/DedupIndexer.java
The file was modifiedwayback/wayback-indexer/src/test/java/dk/netarkivet/wayback/hadoop/CDXMapperTester.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java
The file was modifiedharvester/heritrix3/heritrix3-bundler/pom.xml
The file was modifiedharvester/heritrix3/heritrix3-extensions/pom.xml
Commit 8183749da409edb2229dabb86b9e2fbaca8a182d by Colin Rosenthal (csr)
First attempt at a command-line metadata extraction job.
 (commit: 8183749da409edb2229dabb86b9e2fbaca8a182d)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJob.java
The file was addedhadoop-uber-jar/src/main/resources/run.sh
The file was modifiedhadoop-uber-jar/pom.xml
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java
The file was modifiedpom.xml
The file was addedhadoop-uber-jar/src/main/java/MetadataIndexingApplication.java
Commit 2dd04f160af25e4c61fcdc14c697af66e40b3ed7 by Asger Askov Blekinge (abr)
Downgraded hadoop to stable 3.2.2
 (commit: 2dd04f160af25e4c61fcdc14c697af66e40b3ed7)
The file was modifiedharvester/heritrix3/heritrix3-controller/src/main/java/dk/netarkivet/harvester/heritrix3/controller/AbstractRestHeritrixController.java
The file was modifiedpom.xml
The file was removedhadoop-uber-jar/src/main/resources/run.sh
The file was modifiedhadoop-uber-jar/pom.xml
The file was modifiedhadoop-uber-jar/src/main/java/MetadataIndexingApplication.java
Commit fffd21b9e21b9572899722516c0c6e10a16db15c by Asger Askov Blekinge (abr)
Created an invoker-module to prevent the job from including all the
hadoop dependencies once again
 (commit: fffd21b9e21b9572899722516c0c6e10a16db15c)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
The file was addedhadoop-uber-jar-invoker/pom.xml
The file was addedhadoop-uber-jar-invoker/src/main/assembly/assembly.xml
The file was modifiedhadoop-uber-jar/src/main/java/MetadataIndexingApplication.java
The file was modifiedpom.xml
The file was addedhadoop-uber-jar-invoker/src/main/resources/run.sh
Commit b006660cc04ac3f6c2442dc0d09b74b6c017c9c9 by Asger Askov Blekinge (abr)
Create FileSystem with newInstance and close it afterwards. DO NOT CLOSE
FileSystems gotten with .get
 (commit: b006660cc04ac3f6c2442dc0d09b74b6c017c9c9)
The file was modifiedhadoop-uber-jar/src/main/java/MetadataIndexingApplication.java
The file was modifiedwayback/wayback-indexer/src/test/java/dk/netarkivet/wayback/hadoop/CDXJobTest.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java
Commit dae7ccb17d1e55a49039a7388ff60797959fc609 by Colin Rosenthal (csr)
Added an extra sanity check in the run.sh script.
 (commit: dae7ccb17d1e55a49039a7388ff60797959fc609)
The file was modifiedhadoop-uber-jar/src/main/java/MetadataIndexingApplication.java
The file was modifiedhadoop-uber-jar-invoker/src/main/assembly/assembly.xml
The file was modifiedhadoop-uber-jar-invoker/src/main/resources/run.sh
Commit dd68c05afd42a7a8beb6c10dde0a134b2a9b47a4 by Colin Rosenthal (csr)
Modified to support dynamic identification of the correct file-system
for hadoop input
 (commit: dd68c05afd42a7a8beb6c10dde0a134b2a9b47a4)
The file was modifiedhadoop-uber-jar/src/main/java/MetadataIndexingApplication.java
The file was modifiedhadoop-uber-jar-invoker/src/main/resources/run.sh
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java
Commit 48366f7a72262d6f1442b635b466a2b929b9bcd3 by Colin Rosenthal (csr)
Refactoring to make MetadataIndexingApplication closer to a reusable
real-world case.
 (commit: 48366f7a72262d6f1442b635b466a2b929b9bcd3)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java
The file was modifiedpom.xml
The file was addedhadoop-uber-jar-invoker/src/main/resources/input.txt
The file was modifiedhadoop-uber-jar-invoker/src/main/resources/run.sh
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
The file was modifiedcommon/common-core/src/main/resources/dk/netarkivet/common/settings.xml
The file was modifiedhadoop-uber-jar/src/main/java/MetadataIndexingApplication.java
Commit dc8b3564c4d580f0a8078d58514296c11b01dbce by Colin Rosenthal (csr)
Refactored to use login mechanism instead of doAs.
 (commit: dc8b3564c4d580f0a8078d58514296c11b01dbce)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java
The file was modifiedhadoop-uber-jar/src/main/java/MetadataIndexingApplication.java
Commit 2214312e8db00065089f6ee1da3811cff20a181f by Colin Rosenthal (csr)
Added harvester-core to uber jar
 (commit: 2214312e8db00065089f6ee1da3811cff20a181f)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java
The file was modifiedhadoop-uber-jar/pom.xml
Commit e25e3e3cc198dd3e350f4accd2ce268af0393687 by Rasmus Bohl Kristensen (rbkr)
Small guava pom change
 (commit: e25e3e3cc198dd3e350f4accd2ce268af0393687)
The file was modifiedhadoop-uber-jar/pom.xml
The file was modifiedcommon/common-core/pom.xml
The file was modifiedharvester/harvester-test/src/test/java/dk/netarkivet/harvester/indexserver/hadoop/GetMetadataMapperTester.java
Commit d5e904f8602621fb085dc9d52d2bf6a31992b8d9 by Colin Rosenthal (csr)
Corrected internal versions
 (commit: d5e904f8602621fb085dc9d52d2bf6a31992b8d9)
The file was modifiedhadoop-uber-jar-invoker/pom.xml
The file was modifiedhadoop-uber-jar/pom.xml
The file was modifiedharvester/heritrix3/heritrix3-extensions/pom.xml
Commit e7b9a3383dc1ff2f5b4d6cfbd0ec7a8e3943d7ac by Colin Rosenthal (csr)
More fixed versions
 (commit: e7b9a3383dc1ff2f5b4d6cfbd0ec7a8e3943d7ac)
The file was modifiedhadoop-uber-jar/pom.xml
Commit 681d1147f7824aa7a18804c4b1d9475ce9c26c19 by Colin Rosenthal (csr)
Follow up to own review comments
 (commit: 681d1147f7824aa7a18804c4b1d9475ce9c26c19)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java
The file was modifiedcommon/common-core/src/test/java/dk/netarkivet/common/utils/warc/WarcRecordClientTest.java
The file was modifiedcommon/common-core/src/test/java/dk/netarkivet/common/utils/warc/WarcRecordClientTester.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
The file was modifiedhadoop-uber-jar/pom.xml
The file was modifiedwayback/wayback-indexer/src/test/java/dk/netarkivet/wayback/hadoop/CDXJobTest.java