Status of the production sites
Our biggest issue during the last weeks of 2019 was a harvest problem between Christmas and New Year, that is to say, there were no uploads (neither from the last broad crawl for 2019 nor from the selective crawls) to the Copenhagen bit archive from 24 December 2019 to 1 January 2020. 453 jobs failed with upload errors. About 2400 files are missing on the server, but the system did not “tell” us, that there was no server space left. The minSpaceleft setup implemented before Christmas apparently did not work.
We will work on uploading the missing 2400 files when the broad crawl is finished. We started this broad crawl on 8 November 2019.
An electrical cut in Copenhagen last night probably has produced some loss, as the broad crawl was not finished.
News from our test environment: the newest snapshot NAS 5.7 IIPC Heritrix bundle is able to run on NAS 5.5 without problems.