librarian: stop the DOI search from hanging on a chunk-count mismatch

search_bot conflated MAXTHREADS into two jobs at once - how many chunk files to
read (files 0..MAXTHREADS-1) AND how many producer sentinels to wait for - so
the two had to match exactly. Set too low it silently skipped trailing chunks;
set too high (or with any chunk missing/unreadable) a producer crashed before
emitting its sentinel, the consumers' count never reached the threshold, and
search_for_doi hung on join() forever. The idle-timeout failsafe that was meant
to break a starved consumer was dead code: `if empty_counter > 5: ... elif
empty_counter > 10: break` - >10 implies >5, so the elif never ran.

Fix, three layers:
* auto-discover the chunk files present (discover_chunk_files: <n>_chunk.txt in
  numeric order) instead of range(0, MAXTHREADS). All files are read regardless
  of count, and no producer is ever pointed at a missing file;
* the sentinel threshold is now the number of producers actually started, so it
  can't drift from what's emitted;
* producers emit their sentinel in a finally, so even a crash (missing/unreadable
  chunk) can't starve the count; and the idle backstop is reordered so it can
  actually fire (>EMPTY_LIMIT seconds) as a last resort.

MAXTHREADS is deprecated and unused (kept only so old env files don't break);
docs/env updated to say chunk files are auto-discovered.

For the reported case (MAXTHREADS=40, files 0..43): before, files 40-43 were
silently never searched, and any run that referenced a missing chunk hung
forever. After, all 44 are searched and it always terminates.

Verified in a pytest-only venv (tests/unit/test_search_bot.py): DOI in a
trailing chunk is found; an unreadable chunk still terminates; empty dir returns
at once; discovery is numeric-sorted. Full unit job 27 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docker/env/librarian.env.example

@@ -12,10 +12,14 @@ CONJURER_MAIN_BOT=http://BOT_VM_IP:5000
 # Crossref polite-pool contact (or put credentials in netrc under "crossref").
 CONJURER_CROSSREF_MAILTO=you@example.com
 
-# Local DOI chunk database (mounted volume): expects 0_chunk.txt .. N_chunk.txt
+# Local DOI chunk database (mounted volume): expects 0_chunk.txt .. N_chunk.txt.
+# The chunk files are auto-discovered, so ALL of them are searched no matter how
+# many there are - just drop them in this directory.
 CONJURER_LIBRARIAN_DB_PATH=/doi/
-CONJURER_LIBRARIAN_MAXTHREADS=41
 CONJURER_LIBRARIAN_CHUNK=_chunk.txt
+# DEPRECATED and unused: chunk files are now auto-discovered. It used to have to
+# equal the file count exactly or the search would skip files / hang forever.
+# CONJURER_LIBRARIAN_MAXTHREADS=41
 
 # Runtime JSON state dir (mounted, persistent): cr_results/rr_results/
 # not_in_db/s_results are seeded here on first run.