Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.



Broad crawl:

  • Our second broad crawl 2018 is ongoing. We started with running step 1 with a limit of 10 MB/domain from May 20 to May 28 (552 jobs). We relaunched a run with a limit of 10 MB/domain. We have a problem with too many -50 return codes.

Event crawls:

  • The collection on the collective negotiations on pay is almost finished, as all unions have accepted the results.
  • We just launched a new event crawl on a political and cultural meeting on the island of Bornholm – called “Folkemødet”, which means “peoples meeting”. This a pilot project for the use of BCWeb.

All technical and practical obstacles for the use of BCWeb are “surmounted” – we have now two registered external users.

We have installed OpenWayback in our test environment, among others are we focusing on the replay of https-pages. It works for quite some sites, but not for all sites.




  •  Our Crawl of is almost finished, but still running. The austrian blogging platform wanted to shutdown by end of May, but they postponed the shutdown to end of June. This gives us extra time to finish our jobs.
  • As soon these jobs are finished we are upgrading to Nas 5.4 or 5.4.1 and preparing our domaincrawl
  • We have a request for crawling a website regularly. It's the website of Vienna They want to support us with resources. Our Management has also interest in offering such a service.