SuccessChanges

Summary

  1. Added some logging. (commit: 4f2a7081eba3dbf57dd74356caac68bf394bb4de) (details)
  2. Improved logging in PutFileEventHandler (commit: fa6ad587d2ca341a37f1c95a72cc699737b71052) (details)
  3. Added handling for IDENTIFY_TIMEOUT and correct handling of out-of-sync (commit: 0fe311646422afdbc3d46bebc0d550af56e558bd) (details)
  4. Changed expected appliaction set in SystemTest to match new (commit: 9ea5aefda0f8b97159d90e98203f545b7c7600f7) (details)
  5. Changed log level. (commit: 2fad6ab352d1f932669f1c5b8f3c7350aff55071) (details)
  6. Ensure jobs are closed to prevent threadleak in invoking java process (commit: bf6398619466adaf8f019aae7210544afc6d142c) (details)
  7. Ensure Filesystem objects are closed after use (commit: 802c4d77c7232bf263fc1b0a534c0ee2944b83b1) (details)
  8. Include exit code in IOFailure exception (commit: 75bc6fef31051e97fa06da8e58c28c3d508346b7) (details)
  9. HadoopJobTools logs if the job failes (commit: 22d8f391af675451ab3dbef953113f2d77d968a3) (details)
  10. Changed log level. (commit: 6d4214fb0e32b897d1bddc72add08d79a5ed0dde) (details)
  11. GetMetadataMapper and cacheFile report progress to prevent a (commit: 934d4bb20c17b3e7ed28636af4164857e0fb7705) (details)
  12. Hadoop 3.3.1 as used in test and prod clusters (commit: 1b0383b08a490ff22a913d9888c0e19ae8298dcf) (details)
  13. dedupIndexer can now send progress info to hadoop and thus hopefully (commit: 936944d28b6add356f10837d7f4b1f5f3f8efa39) (details)
  14. Merged commit (commit: 7a085bb6f6c16325f007b5ff3614731fe928968e) (details)
  15. Fixed error introduced during merge (commit: 51e3eaacfe50ff91354213a63907835f73599123) (details)
  16. Fixed error in test spec (commit: 00f781947cfe78334b9723ae55fcfbcdf952cd2e) (details)
  17. Fixed error in test spec (commit: bc7b7cc0f5d8b3cd6778e6ed71771e56c635a7f4) (details)
  18. Explicitly create cache file when caching hdfs (commit: 34952e98ab34e3985d9171f85ee4128d6e7f8d29) (details)
  19. Modified FileResolver to return empty if http response code is not 200. (commit: cbc51994639305fe8a36746c6ba4c00492b3173e) (details)
  20. Fixed bitmag getfileids and some cleanup (commit: 520e4de5e267e3af2d40fa6e78803c15a33df510) (details)
  21. Writing direct to hdfs. (commit: 2ab46b46fece97462f8000d14ddcf3017c77dd2c) (details)
  22. Added direct output streaming from hdfs (commit: 480e7411841e986bc48e2bc9fa7d5f759f5eead7) (details)
  23. Rewritten GetFileIDsAction to use a new handler for each call. (commit: d354945a9730fe2b394f1cc434afefa39d4cf69e) (details)
  24. Fixed some issues with holding large hadoop result sets in memory (commit: c5d6c5d38bb0cfe52a15493ccde9fa24a07e6a8d) (details)
Commit 4f2a7081eba3dbf57dd74356caac68bf394bb4de by Colin Rosenthal (csr)
Added some logging.
 (commit: 4f2a7081eba3dbf57dd74356caac68bf394bb4de)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java (diff)
Commit fa6ad587d2ca341a37f1c95a72cc699737b71052 by Colin Rosenthal (csr)
Improved logging in PutFileEventHandler
 (commit: fa6ad587d2ca341a37f1c95a72cc699737b71052)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/putfile/PutFileEventHandler.java (diff)
Commit 0fe311646422afdbc3d46bebc0d550af56e558bd by Colin Rosenthal (csr)
Added handling for IDENTIFY_TIMEOUT and correct handling of out-of-sync
messages.
 (commit: 0fe311646422afdbc3d46bebc0d550af56e558bd)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/putfile/PutFileEventHandler.java (diff)
Commit 9ea5aefda0f8b97159d90e98203f545b7c7600f7 by Colin Rosenthal (csr)
Changed expected appliaction set in SystemTest to match new
configuration
 (commit: 9ea5aefda0f8b97159d90e98203f545b7c7600f7)
The file was modifiedintegration-test/system-test/src/test/java/dk/netarkivet/systemtest/NASSystemUtil.java (diff)
Commit 2fad6ab352d1f932669f1c5b8f3c7350aff55071 by Colin Rosenthal (csr)
Changed log level.
 (commit: 2fad6ab352d1f932669f1c5b8f3c7350aff55071)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/harvester/harvesting/monitor/HarvestMonitor.java (diff)
Commit bf6398619466adaf8f019aae7210544afc6d142c by Asger Askov Blekinge (abr)
Ensure jobs are closed to prevent threadleak in invoking java process
 (commit: bf6398619466adaf8f019aae7210544afc6d142c)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobTool.java (diff)
Commit 802c4d77c7232bf263fc1b0a534c0ee2944b83b1 by Asger Askov Blekinge (abr)
Ensure Filesystem objects are closed after use
 (commit: 802c4d77c7232bf263fc1b0a534c0ee2944b83b1)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopFileUtils.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java (diff)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/CrawlLogExtractionMapper.java (diff)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/MetadataCDXMapper.java (diff)
Commit 75bc6fef31051e97fa06da8e58c28c3d508346b7 by Asger Askov Blekinge (abr)
Include exit code in IOFailure exception
 (commit: 75bc6fef31051e97fa06da8e58c28c3d508346b7)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJob.java (diff)
Commit 22d8f391af675451ab3dbef953113f2d77d968a3 by Asger Askov Blekinge (abr)
HadoopJobTools logs if the job failes
 (commit: 22d8f391af675451ab3dbef953113f2d77d968a3)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobTool.java (diff)
Commit 6d4214fb0e32b897d1bddc72add08d79a5ed0dde by Colin Rosenthal (csr)
Changed log level.
 (commit: 6d4214fb0e32b897d1bddc72add08d79a5ed0dde)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/BitmagUtils.java (diff)
Commit 934d4bb20c17b3e7ed28636af4164857e0fb7705 by Asger Askov Blekinge (abr)
GetMetadataMapper and cacheFile report progress to prevent a
task-timeout error
 (commit: 934d4bb20c17b3e7ed28636af4164857e0fb7705)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopFileUtils.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java (diff)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/CrawlLogExtractionMapper.java (diff)
Commit 1b0383b08a490ff22a913d9888c0e19ae8298dcf by Asger Askov Blekinge (abr)
Hadoop 3.3.1 as used in test and prod clusters
 (commit: 1b0383b08a490ff22a913d9888c0e19ae8298dcf)
The file was modifiedpom.xml (diff)
Commit 936944d28b6add356f10837d7f4b1f5f3f8efa39 by Asger Askov Blekinge (abr)
dedupIndexer can now send progress info to hadoop and thus hopefully
prevent timeouts
 (commit: 936944d28b6add356f10837d7f4b1f5f3f8efa39)
The file was addedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/ProgressableOutputStream.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/DedupIndexer.java (diff)
Commit 7a085bb6f6c16325f007b5ff3614731fe928968e by Colin Rosenthal (csr)
Merged commit
 (commit: 7a085bb6f6c16325f007b5ff3614731fe928968e)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/putfile/PutFileEventHandler.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java (diff)
Commit 51e3eaacfe50ff91354213a63907835f73599123 by Colin Rosenthal (csr)
Fixed error introduced during merge
 (commit: 51e3eaacfe50ff91354213a63907835f73599123)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/MetadataCDXMapper.java (diff)
Commit 00f781947cfe78334b9723ae55fcfbcdf952cd2e by Colin Rosenthal (csr)
Fixed error in test spec
 (commit: 00f781947cfe78334b9723ae55fcfbcdf952cd2e)
The file was modifiedintegration-test/system-test/src/test/java/dk/netarkivet/systemtest/NASSystemUtil.java (diff)
Commit bc7b7cc0f5d8b3cd6778e6ed71771e56c635a7f4 by Colin Rosenthal (csr)
Fixed error in test spec
 (commit: bc7b7cc0f5d8b3cd6778e6ed71771e56c635a7f4)
The file was modifiedintegration-test/system-test/src/test/java/dk/netarkivet/systemtest/NASSystemUtil.java (diff)
Commit 34952e98ab34e3985d9171f85ee4128d6e7f8d29 by Asger Askov Blekinge (abr)
Explicitly create cache file when caching hdfs
 (commit: 34952e98ab34e3985d9171f85ee4128d6e7f8d29)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopFileUtils.java (diff)
Commit cbc51994639305fe8a36746c6ba4c00492b3173e by Colin Rosenthal (csr)
Modified FileResolver to return empty if http response code is not 200.
 (commit: cbc51994639305fe8a36746c6ba4c00492b3173e)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/service/FileResolverRESTClient.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJob.java (diff)
Commit 520e4de5e267e3af2d40fa6e78803c15a33df510 by Colin Rosenthal (csr)
Fixed bitmag getfileids and some cleanup
 (commit: 520e4de5e267e3af2d40fa6e78803c15a33df510)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java (diff)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/FileNameHarvester.java (diff)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/WaybackIndexer.java (diff)
The file was modifiedcommon/common-core/src/main/resources/dk/netarkivet/common/settings.xml (diff)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/IndexerQueue.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/getfileids/GetFileIDsEventHandler.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/getfileids/GetFileIDsAction.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/service/FileResolverRESTClient.java (diff)
Commit 2ab46b46fece97462f8000d14ddcf3017c77dd2c by Colin Rosenthal (csr)
Writing direct to hdfs.
 (commit: 2ab46b46fece97462f8000d14ddcf3017c77dd2c)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java (diff)
The file was addedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/hadoop/CDXStrategy.java
Commit 480e7411841e986bc48e2bc9fa7d5f759f5eead7 by Colin Rosenthal (csr)
Added direct output streaming from hdfs
 (commit: 480e7411841e986bc48e2bc9fa7d5f759f5eead7)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java (diff)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/WaybackIndexer.java (diff)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobTool.java (diff)
Commit d354945a9730fe2b394f1cc434afefa39d4cf69e by Colin Rosenthal (csr)
Rewritten GetFileIDsAction to use a new handler for each call.
 (commit: d354945a9730fe2b394f1cc434afefa39d4cf69e)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/getfileids/GetFileIDsAction.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/getfileids/GetFileIDsEventHandler.java (diff)
Commit c5d6c5d38bb0cfe52a15493ccde9fa24a07e6a8d by Colin Rosenthal (csr)
Fixed some issues with holding large hadoop result sets in memory
 (commit: c5d6c5d38bb0cfe52a15493ccde9fa24a07e6a8d)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java (diff)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java (diff)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java (diff)
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java (diff)