Started 2 yr 10 mo ago

Not built Build NetarchiveSuite - common - core (06-Jul-2021 09:15:15)

Changes
  1. https://sbforge.org/jira/browse/NAS-2859 (commit: 5aa217c09975ae120744599f93eb99979658c293) (detail)
  2. Update OnbFreeSpaceProvider.java (commit: aed496e2550fe60a499b747c74dff74db965dd57) (detail)
  3. Update OnbFreeSpaceProvider.java (commit: f4b2bc107c333408155e76b78985e4ed2d395790) (detail)
  4. Update OnbFreeSpaceProvider.java (commit: 431ffbf03c51e065068eef0b8f70164a0d92f1a8) (detail)
  5. Updated version to a unique name (commit: e3b328a3cca883f6396a2955ef28bb0e2c7d2300) (detail)
  6. poms updated with Hadoop and basic settings for it added in (commit: 1360e402889d881aee7e6a964c7847f2a96de0b5) (detail)
  7. Small changes to settings (commit: a85a3ad520234cf8cedeeef058911ad47c304474) (detail)
  8. Can now at least work with local bitmag, it seems (commit: 91bdce496fa2ebe20ee0eeb8ce97528a868505f1) (detail)
  9. Indexing through hadoop instead of batch should now work for WARC files (commit: 982056795b045ed339987660c2d13dd7f41d1073) (detail)
  10. Changes from review (commit: a4871eadd0dcfd71fe3bda54170c5f911ae3da88) (detail)
  11. Refactored Bitrepository to a singleton. (commit: e494e8d0482fde2bc44ee840b4e675dac00a43d5) (detail)
  12. Bitrepository class changes (commit: 9a39a4b2aacdc9283f4c34b080693ca5a1ddfe92) (detail)
  13. Dependency fix to avoid logging loop and small logging changes (commit: a709b310c9f6af51c6312e3ed3fec73a912e2622) (detail)
  14. [maven-release-plugin] prepare release netarchivesuite-6.0 (commit: 68ab4244669d4e8d7847001c179f62cb019cacc1) (detail)
  15. [maven-release-plugin] prepare for next development iteration (commit: 16147d5dddfd034ad243da25c911a9fa1e4d53d3) (detail)
  16. [maven-release-plugin] rollback the release of netarchivesuite-6.0 (commit: b246f40d31190967ee84f51cf54e73def89c6737) (detail)
  17. [maven-release-plugin] prepare release netarchivesuite-6.0 (commit: e97567d8b0cf594e4ba5ac10d3d7d8449adc0cc0) (detail)
  18. [maven-release-plugin] prepare for next development iteration (commit: 974b3a9a687aca6fd6fba84828f84f0699ff8078) (detail)
  19. [maven-release-plugin] rollback the release of netarchivesuite-6.0 (commit: df51498918198325ccd2b687dde5c09872ddcc1b) (detail)
  20. [maven-release-plugin] prepare release netarchivesuite-6.0 (commit: 597dca6302626d2eb975dbd7b6e8f7bf8e4dfe17) (detail)
  21. [maven-release-plugin] prepare for next development iteration (commit: d7f4a80b29070e594c26a17a7087d4fff502b8ea) (detail)
  22. [maven-release-plugin] rollback the release of netarchivesuite-6.0 (commit: c8b3c3a9a215db98a3e07c327593f73b93d531ea) (detail)
  23. [maven-release-plugin] prepare release netarchivesuite-6.0 (commit: d71699a62c3a45657e4faf14519b97ba71593b80) (detail)
  24. [maven-release-plugin] prepare for next development iteration (commit: d2fae1846cd326ddf93451ef5d9f15ecbe4f6a73) (detail)
  25. [maven-release-plugin] rollback the release of netarchivesuite-6.0 (commit: 185f170ccde75314ccc6dfec0939d2390c459824) (detail)
  26. [maven-release-plugin] prepare release netarchivesuite-6.0 (commit: 3ca4bd51ae53845b7e6867d637188d4b866f17a6) (detail)
  27. [maven-release-plugin] prepare for next development iteration (commit: 3208775980a139629d0de3a5c1c9859f2ee21543) (detail)
  28. [maven-release-plugin] rollback the release of netarchivesuite-6.0 (commit: f67005308b68ec48d71838bc2efe4d7a9ee38a07) (detail)
  29. [maven-release-plugin] prepare release netarchivesuite-6.0 (commit: dff365105289bbe62558d6dafa673690caf7d153) (detail)
  30. [maven-release-plugin] prepare for next development iteration (commit: 6fcd724e7c8c86b5edb57b79e74e0c2a202101e1) (detail)
  31. WarcRecordClient.java andApacheClientReaderFactory.java in (commit: d71da3e20c6df7f04c793f8b55f0fc6a028286d1) (detail)
  32. WarcRecordClient get and getFile changed (commit: 2b5e8b16b240a009f9ae1ee456350d70aaa821ac) (detail)
  33. Mulig del-løsning (commit: c81836a89060b2194e91860f82947d17a27fe8dc) (detail)
  34. Fixed datafil og tilføjet lidt dokumentation (commit: 26339a4799be4c7ecff361b2be29861d8af2e2f8) (detail)
  35. Added an integration test for WarcRecordClient (commit: 64ac9178de086347e9335ba4b98316370ab699b8) (detail)
  36. WarcRecord fixes for WarcRecordClientTest and Tester (commit: 961cff06f0e4c6241fc550d476add1739b142f7f) (detail)
  37. Error fix (commit: 3cce599a2ccae8aa6ba1c26c75a33cd893ca6220) (detail)
  38. Made method for indexing with Hadoop that assumes direct access to input (commit: b227515615b8ca0bd8c8fe2fd34be189679549c8) (detail)
  39. latest from pc (commit: 9cb0209af9f385e3527307c1dd08c4a78b588c4a) (detail)
  40. Moved getWarc from constructor to get (commit: 4e00feebd87d28c227a323907ab83cdaca0e143d) (detail)
  41. Code-maturation for cdx-indexing (commit: c87a69b57c851c9d90e12982a344da40f614bc74) (detail)
  42. URI corrected to include filename Not yet robust for files not in gzip (commit: 3148c07477ae875b0e29021e146090e7487312f7) (detail)
  43. Hardcoded finName for testing (commit: dda3889ee6740e4306cf84f8e0b7bbee557ee8ee) (detail)
  44. Hardcoded finName for testing (commit: dee3eaab03e37446ffd1be94b1783193a68bba4d) (detail)
  45. Now passes integration test. (commit: 65fd5e068a37d7040f4ef741dd35347d00136ce5) (detail)
  46. Now returns correct record. (commit: 671d7f64ec0db29802bba897eddfdce56d026b49) (detail)
  47. Efter lidt cleanup (commit: dd9de321b98a8635ba8dd1948152469046b3985f) (detail)
  48. Efter endnu lidt cleanup, men før logs (commit: e7504646b6f369f362a96d9ddcf01e4df7266323) (detail)
  49. Added hadoop job for getting metadata lines from archive files and an (commit: 553e20659df3bb62c6d121d50c6effa3fc8947e9) (detail)
  50. latest update (commit: 5763a813e2ec0f0eb1e8ad50cefcafc4767f0455) (detail)
  51. Added filehandling for GetMetadataArchiveMapper and small touch ups (commit: 24aaecbc74299fa9fda9191dfe510977aa027b8f) (detail)
  52. added null response if http statuscode is not 200 (commit: 4804a0787d97a2a9c3802edb716d3d3c78753259) (detail)
  53. removed printlns and added logging for http exception (commit: db425bdcd25a943fdf3c59f741fb5eeb04da82b1) (detail)
  54. Small refactor of ArchiveFile/HadoopUtils, few touch ups and started on (commit: 0d880c6017572102b4bf24e60d87ea35a84e2470) (detail)
  55. added test methods for archive files and negative testing (commit: 085be39423a4881b64793f1328a73ba0286a3f3f) (detail)
  56. Added tests (commit: ec684feb1b69f99952e68dd4e366289b45710ed6) (detail)
  57. test corrections excludes .gz (commit: aca94d36b025e95458e0f718bfd83b9d6876545a) (detail)
  58. 'Start' of https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1970 (commit: d52a6bfda1ed72ad4fc125356ae274f18e0de8c6) (detail)
  59. Tiny settings change for NARK-1882 review (commit: 27a3d95258902008eda3d450c7261e3d694a4c10) (detail)
  60. Integration of Hadoop dedup indexing with GetMetadataArchiveMapper now (commit: ca2c62d474caf14d80e8e1e8f3970a4582e84672) (detail)
  61. Cleaned up a few things in RawMetadataCache and refactored HadoopUtils (commit: d10211a994d936309d076e87f0ae9699d99f385e) (detail)
  62. Squashed commit of the following: (commit: 8d9adc2b50d996dfaa544528b34a7b6b96947d1e) (detail)
  63. Added pattern configuration constants in GetMetadataMapper (commit: 9c130776d86bffa43170028cad724353348ec8dc) (detail)
  64. Cleanup aaording review (commit: 5b3c5fbb6202b4be581afacd06be50dbe3e0deb2) (detail)
  65. latest changes i getFile etc. (commit: 2c586ca6fd4c5b62a1df805f10f8bd1ac451b323) (detail)
  66. corrected (commit: 216184c3737c5b895f1eba74be84f5e4beab244e) (detail)
  67. A few final edits. (commit: e7fbf863417a855376671d375fbe0e4332b9f0ca) (detail)
  68. 'Initial' commit (commit: c4553748c54d383e3f082fc10cf29b2ca4688ab3) (detail)
  69. First commit on arc_record branch (commit: d27f60647703e6333b0baa83ece150e2b1c238a5) (detail)
  70. Review https://sbforge.org/fisheye/cru/CR-NAS-385 changes (commit: 553c4afcb7ddf654b26cc4c9afa3c4cdc7c79197) (detail)
  71. Added testing (commit: 779f5a0dc37edfd41a0dd30d1dc4dd2167c8e13b) (detail)
  72. added .arc test-files (commit: dfffbac038e84ceb95a38d42024b6eb2fd557fdf) (detail)
  73. Fixed dependency problem and added simple application class to run (commit: 016eb5f4073e02288b2acd38f940e0002db0642f) (detail)
  74. Fixed get .arc-record with positive offset (commit: 66c495b026566a13658e39cf40e28120e91a31b8) (detail)
  75. Javadoc added to few files https://sbforge.org/fisheye/cru/CR-NAS-387 (commit: 18336a02e91749175a3a19b4a9fafaa181af053c) (detail)
  76. More review changes https://sbforge.org/fisheye/cru/CR-NAS-387 (commit: 274b0fb8819d98558b17b3fae93c4885ebb6200c) (detail)
  77. minor changes tests (commit: fc4184449c23c26d5a014544479b6673b024164c) (detail)
  78. Initial functioning FileResolverRESTClient (commit: c60029793838e1421e0eb97c3ca8aee8d2b32149) (detail)
  79. Removed some old bitmag classes (commit: 4674413716dbf6fc8e322c47ab8c2fb89c259664) (detail)
  80. Improved handling of try/catch logic (commit: 7a7cda6012b1ba4c6e7d0a1832e5c22b5133bb65) (detail)
  81. Added some new tests and matured code ready for review (commit: b005ce2942cfab8a18e1389da311c32f2e5ac1ee) (detail)
  82. Removed more old bitmag classes, refactored parts of some classes for (commit: dbf8703610bf19bfe044cf7f6f59a10710fdf7b4) (detail)
  83. Fixed some old imports that made the compiler angry (commit: 48d011475ef9dc0480986dbab35e8266e3207f4f) (detail)
  84. Fixed up FileResolverRESTClient for review and refactored code to enable (commit: 0a31340c22213cb7707a5188ec83ded5143c22ce) (detail)
  85. Latest bug fixes on loop testing (commit: f41a6bc3cf46940f48d0dbfca3121cd68679342a) (detail)
  86. Undo of file-change permissions. (commit: 0019993d36df0e7d3be07594ded1edfe0c2e101b) (detail)
  87. Fixed handling of returning used client to pool (commit: cba38820408a11fa56fe247f7b1c9eab668c6ba2) (detail)
  88. Added cdx indexing for metadata files in CDXIndexer and proper testing (commit: 7000ae16f8936955299227260d601c5db7005b81) (detail)
  89. Got Hadoop replacement for ArchiveExtractCDXJob ready, refactored some (commit: 52c718231cc4eb4ba13f6161ace4f701aeb4b738) (detail)
  90. Added setting for new job input/output dirs and more logging (commit: 629996d5c70f12e916878cb48e1c234b932eedcd) (detail)
  91. Setting fix from review https://sbforge.org/jira/browse/NARK-1954 (commit: 5cb2bc46120e35bc9e1074ec1f135efd672d4b2a) (detail)
  92. Tidied up logic in client and tests (commit: 6996761e42e6f09685269eee0b656c2686cbc2d6) (detail)
  93. Review changes https://sbforge.org/fisheye/cru/CR-NAS-393, changes to (commit: bf5e943440e3760f4e25662ffcf603c3a78c0b2e) (detail)
  94. FileResolverRESTClient now sends collectionId as an extra query (commit: 4327720c7867c82e5f3533a23657a5cd16149eba) (detail)
  95. Fixed SimpleFileResolver, refactored how Hadoop jobs can be started, and (commit: 987c230dc013d45aaac9554df0392043965e56a0) (detail)
  96. Added collectionId parameter to WarcRecordClient (commit: 7aadc2501609e5b0ee956af0928ef4793af85ccb) (detail)
  97. Added exactfilename parameter to FileResolverRESTClient. (commit: c8272eac493023dd1aaa9ebef671a1e3a42c3742) (detail)
  98. Added settings for new job and finished last refactoring parts (commit: dcee3b48afde25b3ab1ac42fa65adc64f672d91e) (detail)
  99. Squashed commit of the following: (commit: ab9b8860ca1f5323ca20cabf8a23c7ee01009bc8) (detail)
  100. Small changes from review https://sbforge.org/fisheye/cru/CR-NAS-395 (commit: 3d6bc39bc70440065a715acfe11a76e0575c5ea3) (detail)
  101. Modified JMSBitmagArcRepositoryClient,  PutfileAction and (commit: 539f2aef1903a6745252d0f4b8787c1dd3c1c282) (detail)
  102. small changes in PutFileAction and PutFileEventHandler (commit: 5896c19b108a9e79aea9e0f9490cc0442370ddef) (detail)
  103. latest (commit: ef0643a9605af26ce8d66d58423e802a22574056) (detail)
  104. newest version with warcRecordClient updates (commit: 9f2b875dd6b7ea22925f7d081b9b425b1f2fe8df) (detail)
  105. Added a default value for setting useBitmagHadoopBackend (commit: d4d44145f82d3a6d425472d62eed5dd070672e8f) (detail)
  106. Squashed commit of the following: (commit: 9687194f6e849461945a3f75bdd3906f128d71c8) (detail)
  107. Fixed duplicate code. (commit: c9dd56cca78bda6cf6a436949298f6426eccf6d7) (detail)
  108. Removed old bitmag classes and remnants of it (commit: ad3aaf637af932564da7d15e83b75c02e9f7fb6f) (detail)
  109. Removed dead code (commit: 83446ca81aa7274e519a5e4c1a61b1dc57000fe4) (detail)
  110. Basic CR-NAS-399 changes (commit: 55db2873e040bf7fdca20bab5b2fa6c4870c2b48) (detail)
  111. Bit of refactoring and made SSL provider to work with https (commit: 8f03956481c07a5c8639a847a6a51a30a8827882) (detail)
  112. First attempt at a command-line metadata extraction job. (commit: 8183749da409edb2229dabb86b9e2fbaca8a182d) (detail)
  113. Changed how the SSLContext is built to avoid trusting self-signed certs (commit: 18e8f959abf99005115a5e2521286242e98db82f) (detail)
  114. Fixed the error with closed hadoop file system (commit: 87397cff475747f2efdba4befe07276742918bc7) (detail)
  115. Created an invoker-module to prevent the job from including all the (commit: fffd21b9e21b9572899722516c0c6e10a16db15c) (detail)
  116. Create FileSystem with newInstance and close it afterwards. DO NOT CLOSE (commit: b006660cc04ac3f6c2442dc0d09b74b6c017c9c9) (detail)
  117. Improved logging (commit: 6486dd243255f8bbf407a34207a76e5a4a27d331) (detail)
  118. Attempted improvement of remote file handling of failures. (commit: 6724f3707ca7351efe7cedd055695d534c974fbb) (detail)
  119. Modified to support dynamic identification of the correct file-system (commit: dd68c05afd42a7a8beb6c10dde0a134b2a9b47a4) (detail)
  120. Few clarification fixes to java doc (commit: 44da8bb849b06dc55392c01eaf7475563bc75313) (detail)
  121. Refactoring to make MetadataIndexingApplication closer to a reusable (commit: 48366f7a72262d6f1442b635b466a2b929b9bcd3) (detail)
  122. Refactored to use login mechanism instead of doAs. (commit: dc8b3564c4d580f0a8078d58514296c11b01dbce) (detail)
  123. Removed all unnecessary configuration overrides. (commit: e1c9cc6202e4f7da4fa340f9d9d84ab4f94ea957) (detail)
  124. Added default truststore settings (commit: d3522cfe5d45e6eeafeaf740119b5ef9bc7604d1) (detail)
  125. Added explicit jersey-server dep. to GUI. (commit: 06f48b57ed3030f29f067d2abae08d74bfab1f98) (detail)
  126. Set fallback to environment name for collection (commit: 152eca7f4a888cbcf93654634c732545ce19c938) (detail)
  127. Squashed commit of the following: (commit: 73e016f0e681e23ea9b50428fed236cbba4b706c) (detail)
  128. Tidying up for review. (commit: e68e482d4385d1ac0a0c8aac903055e1b23eaff3) (detail)
  129. Quick attempt to enable hadoop in GUI (commit: 47d462cd57f8647b2162a7f8b2cd3a97d5fc8b42) (detail)
  130. Improved logging on job creation (commit: ef7cc9ec201529ed39e1996abf4d939b9e91c518) (detail)
  131. Read hadoop truststore location from NAS settings (commit: 611487279c35ee59fc5ca71bf6c464d99f08be07) (detail)
  132. Pom Jersey fix (commit: d8f038ad17589937dc4fdfd363f138c80098e822) (detail)
  133. Stuff (commit: 12036689cc9f582449065b5b4072c59a7fcee60d) (detail)
  134. Small guava pom change (commit: e25e3e3cc198dd3e350f4accd2ce268af0393687) (detail)
  135. Follow-up from code review (commit: 100475c5b0ecb5b09e6400ca0a404a79c82d1504) (detail)
  136. Moved Kerberos logins (commit: d022a62c4c855acd8a042db1a240ea515a063d7f) (detail)
  137. Small job change for clarity in cluster job overview (commit: 7af86ebcd7c55acfb7a5244c9c85f5c52ca6764a) (detail)
  138. collectionID setting fix to always default to env name when unset (commit: a114fff89a4046d8084f41be0a25f599dd785575) (detail)
  139. Added hadoop-common as necessary (commit: 3cbe36d8da58ba1f65b23a4740b141a79886b53b) (detail)
  140. Minimal fix to test if ssl works correctly (commit: c1acc3688c43b627668e1edcb6d89035145d9a67) (detail)
  141. Unused imports and small line removals (commit: 84edae215c8f466a8002744284cb300e251d9882) (detail)
  142. Added an exclusion to prevent fatal runtime error. (commit: 6c2e5f7d039dd9bf50c0f1d2e67e57cc4f85778c) (detail)
  143. Follow up to own review comments (commit: 681d1147f7824aa7a18804c4b1d9475ce9c26c19) (detail)
  144. ny pom.xmm a intergaces og annotations (commit: 1fd4d1d17ef45ac49b8f1d91504b390ecb47a70e) (detail)
  145. Fixed circular dependency (commit: b9239bc6e51ea9fafa652287c25a3a1a76306c30) (detail)
  146. Removed requirement for RequiresFileResolver (commit: fdc67c18e38c74545fbda1b6d0912b08d871deca) (detail)
  147. Fixed tests and trying out @ignore for failing mappers (commit: cb5f59cb8465f2eea112989e4ed878851c79ff87) (detail)
  148. Changed version to 7.0-SNAPSHOT (commit: 617efc9a7b921f06f00bcf8aee6e104253242f84) (detail)
  149. Fixed FileNameHarvester throwing NullPointer on failed comms with Bitmag (commit: 191b3f4828a54bd30d183a745a9e05e7e85462c4) (detail)
  150. [maven-release-plugin] prepare release netarchivesuite-7.0 (commit: 90520aea9d5e13775630dfd51d5942015feb6732) (detail)
  151. [maven-release-plugin] prepare for next development iteration (commit: 9f8459ec50a9145c1922559b5ed7a407b550ca7b) (detail)
  152. Added a default retry handler to the http(s) client. (commit: 9205ce706e71716126189dd7ca8474738613c996) (detail)
  153. Made metadata cache directory configurable (commit: 70bf4d5757d431ba901d09081d02306d96459aa6) (detail)
  154. Added store retries. (commit: 3a463182078ff63092d68f49a210567f2b255ea1) (detail)
  155. Improved logging for upload failures. (commit: e0b51e738a1e2c2743b472b297405dd19541a090) (detail)
  156. Added a retry wait to bitmag uploads. (commit: 6d47283d160b1b8dbe11635b21f8d58d38679b3e) (detail)
  157. Set requestSentRetryEnabled to true in calls to (commit: a6926e367f5662441d852e3e4d0f69189a09130c) (detail)
  158. Set requestSentRetryEnabled to true in calls to (commit: eca2e6f8e8291f94a917a05ef81db7ae272719d3) (detail)
  159. Created a more aggressive retry handler. (commit: 90d20a892319d1ad816714fc2d722ed4653fe428) (detail)
  160. Added some logging to retry handler (commit: 4673e7b2c09d6517d567f588cc3b44ca17df1c94) (detail)
  161. Added some logging to retry handler ctor (commit: b575541e901a0432a89d4edd07861693ddc4952e) (detail)
  162. Hadoop memory and core allocation utility functions (commit: a14e8103d5588c5760a5e6e7b69bb4d665fa8730) (detail)
  163. Added utils for managing map-only uber-jobs (commit: 057052a055a46fa4ecb94e8398bc4b0279432a8f) (detail)
  164. Added rethrows for better error handling from hadoop. (commit: 42fe34dbfa04f34062e56c107c65a71869fd3c04) (detail)
  165. Set default hadoop queue names to "default" (commit: beb96074d734343d6314f5a6fe6f2114af9e999e) (detail)