|Our annual broad crawl is still underway. It began on 8th October and must finish before the end of December. We have already crawled more than 68.52 TB. The crawl is taking more time than last year because of several technical problems.|
One problem came from an unresponsive Heritrix process: the Heritix process was no longer reachable even by the HarvestController. The communication port was blocked and new instances of Heritrix were created but stopped instantly. Bert, from the IT team, has activated a new monitoring script that will kill the HarvestController in case of a hung Heritrix process.
The second source of problems came from the infrastructure: the hardware resources of the disks were saturated causing too high latency rates for both read and write operations, which meant we had to reduce the number of threads which slows down the crawl overall.
Other infrastructure problems:
- A CPU on a physical machine failed. The physical machine was removed from the park and all the virtual machines (VM) were moved to another physical machine but the network connexion was lost for four VMs during the moving. Consequently several jobs failed. Moreover we had to replace a virtual hard disk on a VM, and due to a failure to copy the deduplication index several jobs were launched and failed.
- Due to an oversight the maximum number of files which can be opened by a process in the same time was not changed increased for the broad crawl and several jobs were launched and failed.
But the most important problem comes from the broker. Several times during the crawl, the broker crashed leading to the failure of all active jobs of the broad crawl and even of those of the selective crawls. So far it has been impossible to find the reason for the crashes.The saturated hard disks with their latencies may be responsible for it. Bert will investigate further.