Librarian: bound the DOI search work queue to stop OOM-killing the pod

The pod restarted spontaneously mid-search (no liveness probe is set, so
it was the kernel OOM-killer against the 1Gi limit). Cause: search_bot
built its work queue with maxsize 35_500_000. The producers stream the
WHOLE DOI database (tens of millions of lines across chunks) into it while
a few consumers drain, so the queue could buffer gigabytes of lines -
blowing the 1Gi container and taking the whole in-flight search with it.

Bound the queue (default 100k lines, env CONJURER_LIBRARIAN_WORKQ_SIZE),
so producers backpressure to consumers and RAM stays in the low MB.
Because a bounded queue means a producer can now block on a FULL queue,
make the producer's put timeout-poll the TERM sentinel, so a full queue
whose consumers have already finished (all DOIs found) can never deadlock
it. New test pins that: tiny queue + target on line 1 + thousands of
trailing decoys still terminates and finds the target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests/unit/test_search_bot.py

@@ -131,3 +131,22 @@ def test_discover_chunk_files_sorted_numerically(tmp_path, monkeypatch):
 
     # Numeric order (10 after 2, not lexicographic), and non-chunk files ignored.
     assert found == ["0_chunk.txt", "1_chunk.txt", "2_chunk.txt", "10_chunk.txt"]
+
+
+def test_bounded_queue_does_not_deadlock_on_early_termination(tmp_path, monkeypatch):
+    # The OOM fix bounds the work queue. That means a producer can block on a
+    # FULL queue - and if the consumers have already finished (all DOIs found)
+    # it must notice the TERM sentinel instead of hanging forever. Tiny queue +
+    # target on the first line + thousands of trailing decoys the producer still
+    # holds is exactly that situation.
+    monkeypatch.setattr(search_bot, "DATABASE_PATH", str(tmp_path) + "/")
+    monkeypatch.setattr(search_bot, "WORK_Q_SIZE", 3)  # force the producer to block
+    target = "10.1234/found.on.line.one"
+    lines = [target + "\n"] + [f"10.0000/decoy-{i}\n" for i in range(5000)]
+    (tmp_path / "0_chunk.txt").write_text("".join(lines), encoding="utf-8")
+
+    finished, result = _run_bounded([(target, "DATA")], timeout=20)
+
+    assert finished, "a full bounded queue deadlocked the producer on early termination"
+    hit = [r for r in result if r["DOI"] == target and r["exists"]]
+    assert hit, "the target on the first line should have been found"