Changes

Updated version to a unique name
 (commit: e3b328a3cca883f6396a2955ef28bb0e2c7d2300)

Added hadoop job for getting metadata lines from archive files and an
integration test to go with it
 (commit: 553e20659df3bb62c6d121d50c6effa3fc8947e9)

Added filehandling for GetMetadataArchiveMapper and small touch ups
 (commit: 24aaecbc74299fa9fda9191dfe510977aa027b8f)

Small refactor of ArchiveFile/HadoopUtils, few touch ups and started on
metadata job
 (commit: 0d880c6017572102b4bf24e60d87ea35a84e2470)

'Start' of https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1970
 (commit: d52a6bfda1ed72ad4fc125356ae274f18e0de8c6)

Integration of Hadoop dedup indexing with GetMetadataArchiveMapper now
works - still needs few tweaks though
 (commit: ca2c62d474caf14d80e8e1e8f3970a4582e84672)

Cleaned up a few things in RawMetadataCache and refactored HadoopUtils
into two separate classes
 (commit: d10211a994d936309d076e87f0ae9699d99f385e)

Squashed commit of the following:
commit d10211a994d936309d076e87f0ae9699d99f385e Author: bohlski
<rbkr@kb.dk> Date:   Mon Sep 21 15:08:37 2020 +0200
    Cleaned up a few things in RawMetadataCache and refactored
HadoopUtils into two separate classes
commit ca2c62d474caf14d80e8e1e8f3970a4582e84672 Author: bohlski
<rbkr@kb.dk> Date:   Tue Sep 15 13:07:22 2020 +0200
    Integration of Hadoop dedup indexing with GetMetadataArchiveMapper
now works - still needs few tweaks though
commit d52a6bfda1ed72ad4fc125356ae274f18e0de8c6 Author: bohlski
<rbkr@kb.dk> Date:   Fri Sep 11 09:41:00 2020 +0200
    'Start' of
https://sbprojects.statsbiblioteket.dk/jira/browse/NARK-1970
commit 73ec57e3facac150ad9dccb85f46718a98456bc0 Merge: 0d880c601
57b380f2f Author: bohlski <rbkr@kb.dk> Date:   Thu Sep 3 13:37:28 2020
+0200
    Merge branch 'NARK-1882-hadoop-indexing' of
https://github.com/netarchivesuite/netarchivesuite into
NARK-1882-hadoop-indexing
commit 0d880c6017572102b4bf24e60d87ea35a84e2470 Author: bohlski
<rbkr@kb.dk> Date:   Thu Sep 3 13:37:24 2020 +0200
    Small refactor of ArchiveFile/HadoopUtils, few touch ups and started
on metadata job
commit 24aaecbc74299fa9fda9191dfe510977aa027b8f Author: bohlski
<rbkr@kb.dk> Date:   Tue Sep 1 14:30:03 2020 +0200
    Added filehandling for GetMetadataArchiveMapper and small touch ups
 (commit: 8d9adc2b50d996dfaa544528b34a7b6b96947d1e)

Added pattern configuration constants in GetMetadataMapper
 (commit: 9c130776d86bffa43170028cad724353348ec8dc)

Review https://sbforge.org/fisheye/cru/CR-NAS-385 changes
 (commit: 553c4afcb7ddf654b26cc4c9afa3c4cdc7c79197)

Fixed up FileResolverRESTClient for review and refactored code to enable
its use via factory method
 (commit: 0a31340c22213cb7707a5188ec83ded5143c22ce)

Added cdx indexing for metadata files in CDXIndexer and proper testing
that works
 (commit: 7000ae16f8936955299227260d601c5db7005b81)

Got Hadoop replacement for ArchiveExtractCDXJob ready, refactored some
stuff, fixed old bugs in CDXMapper and added more tests for it
 (commit: 52c718231cc4eb4ba13f6161ace4f701aeb4b738)

Added setting for new job input/output dirs and more logging
 (commit: 629996d5c70f12e916878cb48e1c234b932eedcd)

Setting fix from review https://sbforge.org/jira/browse/NARK-1954
 (commit: 5cb2bc46120e35bc9e1074ec1f135efd672d4b2a)

Review changes https://sbforge.org/fisheye/cru/CR-NAS-393, changes to
uber-jar set up and small pom fixes
 (commit: bf5e943440e3760f4e25662ffcf603c3a78c0b2e)

Fixed SimpleFileResolver, refactored how Hadoop jobs can be started, and
implemented getCrawlLogLinesMatchingRegexp
 (commit: 987c230dc013d45aaac9554df0392043965e56a0)

Added settings for new job and finished last refactoring parts
 (commit: dcee3b48afde25b3ab1ac42fa65adc64f672d91e)

Made small fix/cleanup in crawl log mapper and added more documentation
 (commit: e052b35ecbe8d07c2a88e914d3202d863b57bf50)

Squashed commit of the following:
commit 92109a0ad5ddb9238d593fbce21023ae05804b16 Author: bohlski
<rbkr@kb.dk> Date:   Mon Dec 7 11:41:48 2020 +0100
    Small changes from review https://sbforge.org/fisheye/cru/CR-NAS-395
commit e052b35ecbe8d07c2a88e914d3202d863b57bf50 Author: bohlski
<rbkr@kb.dk> Date:   Wed Dec 2 15:05:48 2020 +0100
    Made small fix/cleanup in crawl log mapper and added more
documentation
commit dcee3b48afde25b3ab1ac42fa65adc64f672d91e Author: bohlski
<rbkr@kb.dk> Date:   Wed Dec 2 14:07:18 2020 +0100
    Added settings for new job and finished last refactoring parts
commit 987c230dc013d45aaac9554df0392043965e56a0 Author: bohlski
<rbkr@kb.dk> Date:   Mon Nov 30 11:56:25 2020 +0100
    Fixed SimpleFileResolver, refactored how Hadoop jobs can be started,
and implemented getCrawlLogLinesMatchingRegexp
 (commit: ab9b8860ca1f5323ca20cabf8a23c7ee01009bc8)

Squashed commit of the following:
commit a738bebf1170c0527ccbbc2925c4e15515ed28ab Author: bohlski
<rbkr@kb.dk> Date:   Mon Jan 4 14:31:28 2021 +0100
    Changes from review https://sbforge.org/fisheye/cru/CR-NAS-396
commit 3e922d974c296333cd469778c91f6bad0174270d Author: bohlski
<rbkr@kb.dk> Date:   Wed Dec 9 11:15:02 2020 +0100
    Small addition to error message on invalid base url
commit 16706d0129abb1caed2a9bdaa6df6b7b087ed895 Author: bohlski
<rbkr@kb.dk> Date:   Tue Dec 8 16:36:59 2020 +0100
    Modified the bitmag arcrepositoryclient get() to now request records
through the warc record service.
commit 9a92eca3bdfd3b6546a9a8569433b270f4f938cf Author: bohlski
<rbkr@kb.dk> Date:   Mon Dec 7 15:00:49 2020 +0100
    Fixed missing settings documentation from NARK-1998
 (commit: 9687194f6e849461945a3f75bdd3906f128d71c8)

First attempt at a kill switch that returns an empty index for dedups
 (commit: 08d62e8104de4fe99d49b71b4b7933e41987bb56)

Second attempt using IndexReadyMessage
 (commit: 3199f61725d3badd01b8e26a1c0c295c7564cb09)

Added some logging
 (commit: aea04138a7ce030b1772456dee253c744b10453e)

Further attempt
 (commit: 701b2c647c674bc72091877fbd6ab2bd8e989ca9)

Further attempt using IndexReadyMessage
 (commit: ca9377f522312bdb2babb2c33e664becd1fbcf81)

Back to reply
 (commit: ddb5dd34fc6b691f0318b0965dc09b51f3da9f66)

Added a bit more logging.
 (commit: e4c67af253358e93ca41e108cfefff2072a6d9fa)

Removed potential error when requesting empty cache
 (commit: f6a7d91cbb5b2f3feaf42fc1b7e83ada5f2bb73a)

Clean-up
 (commit: f0f4a71edd0773f7fd35d30bfb5f80abe93057eb)

Refactoring to make MetadataIndexingApplication closer to a reusable
real-world case.
 (commit: 48366f7a72262d6f1442b635b466a2b929b9bcd3)

Initial version using fileresolver
 (commit: f212546a060f7271527d9d722308241d60e1b720)

Added explicit jersey-server dep. to GUI.
 (commit: 06f48b57ed3030f29f067d2abae08d74bfab1f98)

Added a necessary filtering stage to match only current collection
 (commit: 5501fca44675886868cd1672489a9000f4a15c97)

Tidying up for review.
 (commit: e68e482d4385d1ac0a0c8aac903055e1b23eaff3)

Forcing HadoopJobStrategy to use hdfs
 (commit: bafdb18e7a4b7667e765882e8a1205de182f3c91)

Forcing HadoopJobStrategy to use hdfs
 (commit: 6adbe99384b5754740a0cc4e6359ad3d1cc4e9ea)

Added harvester-core to uber jar
 (commit: 2214312e8db00065089f6ee1da3811cff20a181f)

Moved Kerberos logins
 (commit: d022a62c4c855acd8a042db1a240ea515a063d7f)

Small fixes and revert
 (commit: ba7b6362e8dc63a0711bdc4a89c957ebca15a6bd)

Readded Kerberos login to IndexRequestServer
 (commit: 9e06be74dd416fc3e0c6036fcc457796b4547e53)

collectionID setting fix to always default to env name when unset
 (commit: a114fff89a4046d8084f41be0a25f599dd785575)

Follow up to own review comments
 (commit: 681d1147f7824aa7a18804c4b1d9475ce9c26c19)