SuccessChanges

Summary

  1. Added some logging. (commit: 4f2a7081eba3dbf57dd74356caac68bf394bb4de) (details)
  2. Improved logging in PutFileEventHandler (commit: fa6ad587d2ca341a37f1c95a72cc699737b71052) (details)
  3. Added handling for IDENTIFY_TIMEOUT and correct handling of out-of-sync (commit: 0fe311646422afdbc3d46bebc0d550af56e558bd) (details)
  4. Ensure jobs are closed to prevent threadleak in invoking java process (commit: bf6398619466adaf8f019aae7210544afc6d142c) (details)
  5. Ensure Filesystem objects are closed after use (commit: 802c4d77c7232bf263fc1b0a534c0ee2944b83b1) (details)
  6. Include exit code in IOFailure exception (commit: 75bc6fef31051e97fa06da8e58c28c3d508346b7) (details)
  7. HadoopJobTools logs if the job failes (commit: 22d8f391af675451ab3dbef953113f2d77d968a3) (details)
  8. Changed log level. (commit: 6d4214fb0e32b897d1bddc72add08d79a5ed0dde) (details)
  9. GetMetadataMapper and cacheFile report progress to prevent a (commit: 934d4bb20c17b3e7ed28636af4164857e0fb7705) (details)
  10. Merged commit (commit: 7a085bb6f6c16325f007b5ff3614731fe928968e) (details)
  11. Explicitly create cache file when caching hdfs (commit: 34952e98ab34e3985d9171f85ee4128d6e7f8d29) (details)
  12. Modified FileResolver to return empty if http response code is not 200. (commit: cbc51994639305fe8a36746c6ba4c00492b3173e) (details)
  13. Fixed bitmag getfileids and some cleanup (commit: 520e4de5e267e3af2d40fa6e78803c15a33df510) (details)
  14. Added direct output streaming from hdfs (commit: 480e7411841e986bc48e2bc9fa7d5f759f5eead7) (details)
  15. Rewritten GetFileIDsAction to use a new handler for each call. (commit: d354945a9730fe2b394f1cc434afefa39d4cf69e) (details)
  16. Fixed some issues with holding large hadoop result sets in memory (commit: c5d6c5d38bb0cfe52a15493ccde9fa24a07e6a8d) (details)
Commit 4f2a7081eba3dbf57dd74356caac68bf394bb4de by Colin Rosenthal (csr)
Added some logging.
 (commit: 4f2a7081eba3dbf57dd74356caac68bf394bb4de)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java
Commit fa6ad587d2ca341a37f1c95a72cc699737b71052 by Colin Rosenthal (csr)
Improved logging in PutFileEventHandler
 (commit: fa6ad587d2ca341a37f1c95a72cc699737b71052)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/putfile/PutFileEventHandler.java
Commit 0fe311646422afdbc3d46bebc0d550af56e558bd by Colin Rosenthal (csr)
Added handling for IDENTIFY_TIMEOUT and correct handling of out-of-sync
messages.
 (commit: 0fe311646422afdbc3d46bebc0d550af56e558bd)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/putfile/PutFileEventHandler.java
Commit bf6398619466adaf8f019aae7210544afc6d142c by Asger Askov Blekinge (abr)
Ensure jobs are closed to prevent threadleak in invoking java process
 (commit: bf6398619466adaf8f019aae7210544afc6d142c)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobTool.java
Commit 802c4d77c7232bf263fc1b0a534c0ee2944b83b1 by Asger Askov Blekinge (abr)
Ensure Filesystem objects are closed after use
 (commit: 802c4d77c7232bf263fc1b0a534c0ee2944b83b1)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopFileUtils.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/CrawlLogExtractionMapper.java
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/MetadataCDXMapper.java
Commit 75bc6fef31051e97fa06da8e58c28c3d508346b7 by Asger Askov Blekinge (abr)
Include exit code in IOFailure exception
 (commit: 75bc6fef31051e97fa06da8e58c28c3d508346b7)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJob.java
Commit 22d8f391af675451ab3dbef953113f2d77d968a3 by Asger Askov Blekinge (abr)
HadoopJobTools logs if the job failes
 (commit: 22d8f391af675451ab3dbef953113f2d77d968a3)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobTool.java
Commit 6d4214fb0e32b897d1bddc72add08d79a5ed0dde by Colin Rosenthal (csr)
Changed log level.
 (commit: 6d4214fb0e32b897d1bddc72add08d79a5ed0dde)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/BitmagUtils.java
Commit 934d4bb20c17b3e7ed28636af4164857e0fb7705 by Asger Askov Blekinge (abr)
GetMetadataMapper and cacheFile report progress to prevent a
task-timeout error
 (commit: 934d4bb20c17b3e7ed28636af4164857e0fb7705)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopFileUtils.java
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/hadoop/CrawlLogExtractionMapper.java
Commit 7a085bb6f6c16325f007b5ff3614731fe928968e by Colin Rosenthal (csr)
Merged commit
 (commit: 7a085bb6f6c16325f007b5ff3614731fe928968e)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/putfile/PutFileEventHandler.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/GetMetadataMapper.java
Commit 34952e98ab34e3985d9171f85ee4128d6e7f8d29 by Asger Askov Blekinge (abr)
Explicitly create cache file when caching hdfs
 (commit: 34952e98ab34e3985d9171f85ee4128d6e7f8d29)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopFileUtils.java
Commit cbc51994639305fe8a36746c6ba4c00492b3173e by Colin Rosenthal (csr)
Modified FileResolver to return empty if http response code is not 200.
 (commit: cbc51994639305fe8a36746c6ba4c00492b3173e)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/service/FileResolverRESTClient.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJob.java
Commit 520e4de5e267e3af2d40fa6e78803c15a33df510 by Colin Rosenthal (csr)
Fixed bitmag getfileids and some cleanup
 (commit: 520e4de5e267e3af2d40fa6e78803c15a33df510)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/CommonSettings.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/IndexerQueue.java
The file was modifiedcommon/common-core/src/main/resources/dk/netarkivet/common/settings.xml
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/getfileids/GetFileIDsEventHandler.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/service/FileResolverRESTClient.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/getfileids/GetFileIDsAction.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/FileNameHarvester.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/WaybackIndexer.java
Commit 480e7411841e986bc48e2bc9fa7d5f759f5eead7 by Colin Rosenthal (csr)
Added direct output streaming from hdfs
 (commit: 480e7411841e986bc48e2bc9fa7d5f759f5eead7)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/WaybackIndexer.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobTool.java
Commit d354945a9730fe2b394f1cc434afefa39d4cf69e by Colin Rosenthal (csr)
Rewritten GetFileIDsAction to use a new handler for each call.
 (commit: d354945a9730fe2b394f1cc434afefa39d4cf69e)
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/getfileids/GetFileIDsAction.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/distribute/bitrepository/action/getfileids/GetFileIDsEventHandler.java
Commit c5d6c5d38bb0cfe52a15493ccde9fa24a07e6a8d by Colin Rosenthal (csr)
Fixed some issues with holding large hadoop result sets in memory
 (commit: c5d6c5d38bb0cfe52a15493ccde9fa24a07e6a8d)
The file was modifiedwayback/wayback-indexer/src/main/java/dk/netarkivet/wayback/indexer/ArchiveFile.java
The file was modifiedcommon/common-core/src/main/java/dk/netarkivet/common/utils/hadoop/HadoopJobUtils.java
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/harvester/indexserver/RawMetadataCache.java
The file was modifiedharvester/harvester-core/src/main/java/dk/netarkivet/viewerproxy/webinterface/Reporting.java