Librarian: bound the DOI search work queue to stop OOM-killing the pod

The pod restarted spontaneously mid-search (no liveness probe is set, so
it was the kernel OOM-killer against the 1Gi limit). Cause: search_bot
built its work queue with maxsize 35_500_000. The producers stream the
WHOLE DOI database (tens of millions of lines across chunks) into it while
a few consumers drain, so the queue could buffer gigabytes of lines -
blowing the 1Gi container and taking the whole in-flight search with it.

Bound the queue (default 100k lines, env CONJURER_LIBRARIAN_WORKQ_SIZE),
so producers backpressure to consumers and RAM stays in the low MB.
Because a bounded queue means a producer can now block on a FULL queue,
make the producer's put timeout-poll the TERM sentinel, so a full queue
whose consumers have already finished (all DOIs found) can never deadlock
it. New test pins that: tiny queue + target on line 1 + thousands of
trailing decoys still terminates and finds the target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

conjurer_librarian/search_bot.py

@@ -21,7 +21,7 @@ Global Variables:
 # TODO: Wpiemdolić to wszystko w klasę z loggerem przysłanym z góry
 import os
 import re
-from queue import Empty, Queue
+from queue import Empty, Full, Queue
 from threading import Thread
 import time
 q = Queue()
@@ -44,7 +44,13 @@ CHUNK = os.getenv("CONJURER_LIBRARIAN_CHUNK", "_chunk.txt")
 MAXTHREADS = int(os.getenv("CONJURER_LIBRARIAN_MAXTHREADS", "0"))
 
 _sentinel = object()
-WORK_Q_SIZE = 35500000
+# BOUNDED work queue. The producers stream the WHOLE DOI database (potentially
+# tens of millions of lines across chunks) into this queue; the previous cap of
+# 35_500_000 items was effectively unbounded (~3.5 GB of buffered lines), which
+# OOM-killed the 1 GiB container mid-search. A small bound makes the producers
+# backpressure to the consumers, keeping RAM to a few MB. The producer put below
+# stays responsive to the TERM sentinel so a full queue can never deadlock it.
+WORK_Q_SIZE = int(os.getenv("CONJURER_LIBRARIAN_WORKQ_SIZE", "100000"))
 # Idle backstop: after this many consecutive empty seconds a consumer assumes
 # the producers are done (or dead) and exits, so the search can never hang even
 # if a sentinel were somehow lost. The primary, correct termination is still the
@@ -107,7 +113,26 @@ def producer(out_q, control_q, filename, _logger):
                     print(f"EOF {filename}")
                     break
 
-                out_q.put(line)
+                # Backpressure-safe put onto the BOUNDED queue: wait for room,
+                # but keep polling the TERM sentinel so a full queue whose
+                # consumers have already finished can never deadlock us here.
+                stopped = False
+                while True:
+                    try:
+                        out_q.put(line, timeout=1)
+                        break
+                    except Full:
+                        try:
+                            if control_q.get(block=False) is _sentinel:
+                                control_q.put(_sentinel)
+                                stopped = True
+                                break
+                        except Empty:
+                            pass
+                if stopped:
+                    print("TERM signal received (queue full)")
+                    break
+
                 try:
                     check = control_q.get(block=False)
                 except Empty: