Changes

Updated version to a unique name
 (commit: e3b328a3cca883f6396a2955ef28bb0e2c7d2300)

Small changes to settings
 (commit: a85a3ad520234cf8cedeeef058911ad47c304474)

Can now at least work with local bitmag, it seems
 (commit: 91bdce496fa2ebe20ee0eeb8ce97528a868505f1)

FileNameHarvester now grabs list of files directly from bitmag. Added
basic stuff for getting started with using hadoop jobs instead of batch
in ArchiveFile
 (commit: 6824e324e6ccd74a10740790c11968bd5803a312)

Indexing through hadoop instead of batch should now work for WARC files
 (commit: 982056795b045ed339987660c2d13dd7f41d1073)

Changes from review
 (commit: a4871eadd0dcfd71fe3bda54170c5f911ae3da88)

Fixed dependency conflict with hadoop-client package and finished
harvestRecentFilenames
 (commit: e8735f0b8fa3800b012a7928e369cf358b5248d8)

Refactored Bitrepository to a singleton.
 (commit: e494e8d0482fde2bc44ee840b4e675dac00a43d5)

Small logging changes
 (commit: c091c483521bf77ea95b7450dd0aee6449f5ccac)

Bitrepository class changes
 (commit: 9a39a4b2aacdc9283f4c34b080693ca5a1ddfe92)

Dependency fix to avoid logging loop and small logging changes
 (commit: a709b310c9f6af51c6312e3ed3fec73a912e2622)

Added some integration tests for indexing on hadoop
 (commit: 96dfe55073b04b9cab36dbc4607c7496c877c2fd)

Removed the test which was less like the anticipated prod architecture
 (commit: 1ac65bb88c41f4f60214f728145a0c39ad035d46)

Tidied up the hadoop/cdx integration test
 (commit: b8eaa110fb2b6f1b8e2530776aa5c8d3e901273c)

Added Readme file in empty directory
 (commit: c4da8b152e8fa7f10de0bd1fb48ed0868b7664eb)

Added Readme file in empty directory
 (commit: 96b73b68fa7e1301ab0328df8e0210d56a735c52)

Added a hdfs setting that seems relevant
 (commit: 60cd273673637ba210763dda8b107a7b96bef508)

Made method for indexing with Hadoop that assumes direct access to input
files
 (commit: b227515615b8ca0bd8c8fe2fd34be189679549c8)

Dedup indexing
 (commit: f5508c47ebc69bc946ff16ca87218c949a59508a)

Code-maturation for cdx-indexing
 (commit: c87a69b57c851c9d90e12982a344da40f614bc74)

Attempt to avoid double-indexing
 (commit: e1c23281deb8bd3055c9f85508c5076076f37c32)

Initial work on FileResolver
 (commit: d411beb865c7dace665ae2709953857503a37c27)

Added hadoop job for getting metadata lines from archive files and an
integration test to go with it
 (commit: 553e20659df3bb62c6d121d50c6effa3fc8947e9)

Added pattern-matching method to file-resolver
 (commit: 57b380f2f8c289544404aecbb81f6cef8a084274)

Small refactor of ArchiveFile/HadoopUtils, few touch ups and started on
metadata job
 (commit: 0d880c6017572102b4bf24e60d87ea35a84e2470)

Changed test to use paths relative to module root
 (commit: 0602fcb9458cfa655d0cb39f98d004e720bc9e42)

Added a conf flag to switch between standard indexing and dedup indexing
for metadata files.
 (commit: 2e2173b833c5c5ddd879e7548b96d875a06353a1)

'Start' of https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1970
 (commit: d52a6bfda1ed72ad4fc125356ae274f18e0de8c6)

Integration of Hadoop dedup indexing with GetMetadataArchiveMapper now
works - still needs few tweaks though
 (commit: ca2c62d474caf14d80e8e1e8f3970a4582e84672)

Cleaned up a few things in RawMetadataCache and refactored HadoopUtils
into two separate classes
 (commit: d10211a994d936309d076e87f0ae9699d99f385e)

Squashed commit of the following:
commit d10211a994d936309d076e87f0ae9699d99f385e Author: bohlski
<rbkr@kb.dk> Date:   Mon Sep 21 15:08:37 2020 +0200
    Cleaned up a few things in RawMetadataCache and refactored
HadoopUtils into two separate classes
commit ca2c62d474caf14d80e8e1e8f3970a4582e84672 Author: bohlski
<rbkr@kb.dk> Date:   Tue Sep 15 13:07:22 2020 +0200
    Integration of Hadoop dedup indexing with GetMetadataArchiveMapper
now works - still needs few tweaks though
commit d52a6bfda1ed72ad4fc125356ae274f18e0de8c6 Author: bohlski
<rbkr@kb.dk> Date:   Fri Sep 11 09:41:00 2020 +0200
    'Start' of
https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1970
commit 73ec57e3facac150ad9dccb85f46718a98456bc0 Merge: 0d880c601
57b380f2f Author: bohlski <rbkr@kb.dk> Date:   Thu Sep 3 13:37:28 2020
+0200
    Merge branch 'NARK-1882-hadoop-indexing' of
https://github.com/netarchivesuite/netarchivesuite into
NARK-1882-hadoop-indexing
commit 0d880c6017572102b4bf24e60d87ea35a84e2470 Author: bohlski
<rbkr@kb.dk> Date:   Thu Sep 3 13:37:24 2020 +0200
    Small refactor of ArchiveFile/HadoopUtils, few touch ups and started
on metadata job
commit 24aaecbc74299fa9fda9191dfe510977aa027b8f Author: bohlski
<rbkr@kb.dk> Date:   Tue Sep 1 14:30:03 2020 +0200
    Added filehandling for GetMetadataArchiveMapper and small touch ups
 (commit: 8d9adc2b50d996dfaa544528b34a7b6b96947d1e)

Added pattern configuration constants in GetMetadataMapper
 (commit: 9c130776d86bffa43170028cad724353348ec8dc)

Review https://sbforge.org/fisheye/cru/CR-NAS-385 changes
 (commit: 553c4afcb7ddf654b26cc4c9afa3c4cdc7c79197)

Small refactor and implemented harvestRecentFilenames
 (commit: 199355bf63aba632f2dcb23c66de7778b794e97e)

Removed more old bitmag classes, refactored parts of some classes for
easier later removal, and added old necessary functionality (e.g. utils)
to already existing classes
 (commit: dbf8703610bf19bfe044cf7f6f59a10710fdf7b4)

Fixed some old imports that made the compiler angry
 (commit: 48d011475ef9dc0480986dbab35e8266e3207f4f)

Fixed up FileResolverRESTClient for review and refactored code to enable
its use via factory method
 (commit: 0a31340c22213cb7707a5188ec83ded5143c22ce)

Added more logging to FileNameHarvester
 (commit: b7120d70f3adf93ca6a12368f30897084a8a6295)

Small refactor to make ArchiveFile's collectHadoopResults use
HadoopJobUtils' collectOutputLines method - fix for old todo
 (commit: 358b6977ce8fc98987a32327d57815bd30c0f34a)

Fixed bug with indexing threads sharing same filesystem instance
 (commit: d8c00a93115685b3357ec4202d7de727d76486fa)

Fixed bug with indexing threads sharing same filesystem instance
 (commit: e9969a932d57a7e301ec95365219e3a662c3b0c6)

Added cdx indexing for metadata files in CDXIndexer and proper testing
that works
 (commit: 7000ae16f8936955299227260d601c5db7005b81)

Got Hadoop replacement for ArchiveExtractCDXJob ready, refactored some
stuff, fixed old bugs in CDXMapper and added more tests for it
 (commit: 52c718231cc4eb4ba13f6161ace4f701aeb4b738)

Setting fix from review https://sbforge.org/jira/browse/NARK-1954
 (commit: 5cb2bc46120e35bc9e1074ec1f135efd672d4b2a)

Review changes https://sbforge.org/fisheye/cru/CR-NAS-393, changes to
uber-jar set up and small pom fixes
 (commit: bf5e943440e3760f4e25662ffcf603c3a78c0b2e)

Fixed SimpleFileResolver, refactored how Hadoop jobs can be started, and
implemented getCrawlLogLinesMatchingRegexp
 (commit: 987c230dc013d45aaac9554df0392043965e56a0)

Squashed commit of the following:
commit 92109a0ad5ddb9238d593fbce21023ae05804b16 Author: bohlski
<rbkr@kb.dk> Date:   Mon Dec 7 11:41:48 2020 +0100
    Small changes from review https://sbforge.org/fisheye/cru/CR-NAS-395
commit e052b35ecbe8d07c2a88e914d3202d863b57bf50 Author: bohlski
<rbkr@kb.dk> Date:   Wed Dec 2 15:05:48 2020 +0100
    Made small fix/cleanup in crawl log mapper and added more
documentation
commit dcee3b48afde25b3ab1ac42fa65adc64f672d91e Author: bohlski
<rbkr@kb.dk> Date:   Wed Dec 2 14:07:18 2020 +0100
    Added settings for new job and finished last refactoring parts
commit 987c230dc013d45aaac9554df0392043965e56a0 Author: bohlski
<rbkr@kb.dk> Date:   Mon Nov 30 11:56:25 2020 +0100
    Fixed SimpleFileResolver, refactored how Hadoop jobs can be started,
and implemented getCrawlLogLinesMatchingRegexp
 (commit: ab9b8860ca1f5323ca20cabf8a23c7ee01009bc8)

Removed old bitmag classes and remnants of it
 (commit: ad3aaf637af932564da7d15e83b75c02e9f7fb6f)

Bit of refactoring and made SSL provider to work with https
 (commit: 8f03956481c07a5c8639a847a6a51a30a8827882)

Create FileSystem with newInstance and close it afterwards. DO NOT CLOSE
FileSystems gotten with .get
 (commit: b006660cc04ac3f6c2442dc0d09b74b6c017c9c9)

Refactoring to make MetadataIndexingApplication closer to a reusable
real-world case.
 (commit: 48366f7a72262d6f1442b635b466a2b929b9bcd3)

Squashed commit of the following:
commit c0397d7d25f0848af27c5ba646cc6b0124afa9ff Merge: d3522cfe5
f0f4a71ed Author: bohlski <rbkr@kb.dk> Date:   Tue Feb 16 11:15:41 2021
+0100
    Merge branch 'bitmag' into NARK-2016-HTTPS-WRS-FileResolver
commit d3522cfe5d45e6eeafeaf740119b5ef9bc7604d1 Author: bohlski
<rbkr@kb.dk> Date:   Tue Feb 16 11:08:47 2021 +0100
    Added default truststore settings
commit 44da8bb849b06dc55392c01eaf7475563bc75313 Author: bohlski
<rbkr@kb.dk> Date:   Wed Feb 10 16:04:30 2021 +0100
    Few clarification fixes to java doc
commit 18e8f959abf99005115a5e2521286242e98db82f Author: bohlski
<rbkr@kb.dk> Date:   Tue Feb 2 16:46:49 2021 +0100
    Changed how the SSLContext is built to avoid trusting self-signed
certs
commit 8f03956481c07a5c8639a847a6a51a30a8827882 Author: bohlski
<rbkr@kb.dk> Date:   Tue Feb 2 11:02:29 2021 +0100
    Bit of refactoring and made SSL provider to work with https
 (commit: 73e016f0e681e23ea9b50428fed236cbba4b706c)

Moved Kerberos logins
 (commit: d022a62c4c855acd8a042db1a240ea515a063d7f)

collectionID setting fix to always default to env name when unset
 (commit: a114fff89a4046d8084f41be0a25f599dd785575)

Just some optimized imports and small stuff
 (commit: 48977bc912bcd2f4b9b44db06e65c04eb0281352)

Follow up to own review comments
 (commit: 681d1147f7824aa7a18804c4b1d9475ce9c26c19)