IndexRequestServer does not keep track of ongoing index requests, hence is vulnerable to crashes.

When our production engineers ran robustness tests, they tried activating a few harvest definitions, let the jobs request a deduplication index, then killed and restarted the IndexServerApplication.

The harvesters kept waiting for a reply to the index request (and won't stop before hitting their timeout) while the IndexServer had obviously forgotten of all previous requests.

This is due to the fact that IndexRequestServer is implemented in a rather crude way, as it starts a thread for each request.

We should make an implementation that keeps track of index requests based on the subset of jobs that is requested, and which would be able to restart processing these requests when restarting the application.

Also harvesters should actively check for index generation completion instead of passively waiting for an answer. An even better solution would be not to submit a job before the relevant index has been generated, because it is a prerequisite to the job successful completion.