Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • BNF: Clara, Sara
  • ONB: Andreas
  • KB/DK - Copenhagen: Tue, Stephen, Anders, Kristian
  • KB/DK - Aarhus: Colin, Sabine, Knud Åge
  • BNE: Alicia, María, Manuel, José
  • KB/Sweden: Par, Thomas, Peter

...

Panel

Broad crawl

We finished our third broad crawl for 2019 (with a limit of 50 MB/step 1 and 16 GB/step 2) on 10 September. In 602 jobs we harvested a total of about 93 TB or 187 million objects. There are lots of sites blocking us, we will solve that by giving our new broad crawl harvesters new IP addresses and updating our throttling firewall rules. Simultaniously we ran the selective crawls connected to the broad crawls: Research databases, Municipalities and regions, Ministries and Government agencies, YouTube.

Now we are doing the “cleaning up” and improvements to prepare the next broad crawl

Selective crawls

Getting IP-validated access to content behind paywalls is still a big issue (to get in touch with the right person from the website owners).

Ongoing projects

  • Implementation of SolR Wayback
  • New user access procedure and form
  • Data mining/extractions from the archive: make sure with our legal department, that we follow all relevant laws
  • Visual QA of https-seeds: configuration of pc’s for reading Warc-files

BnF

Panel


ONB

Panel


BNE

Panel

Broad Crawl

We have started our annual broad crawl on September 23.

There are almost 2 million websites that we divide in sets of 500 domains per job with a limit of 150 MB/domain.

We use two specific networks (FTTH-Fiber to The Home) to make the broad crawl in order to leave the regular network for our selective collections

We have already collected 38% of the websites (26 TB of information) without important inconveniences

KB-Sweden

Panel

Broad crawl 

We have had lots of problems with the broad crawl we started a couple of weeks ago. 

Next meetings

  • November 5
  • December 3
  • January 7, 2020

...