Two hygiene fixes on top of the work-queue OOM bound:
Result dumps: cr_results / rr_results / s_results.json were write-only
(nothing reads them) yet accumulated EVERY search forever and json.load'd
the whole growing file on each write - unbounded RAM and PVC growth, and
for a deep search the raw cr_results dump is hundreds of MB. They are now
off by default (CONJURER_LIBRARIAN_DEBUG_DUMPS) and, when enabled, are
overwritten with just the latest search - never loaded or accumulated.
not_in_db.json is untouched: it's a real queue the scraper drains.
Search logging: search_bot logged via print(), including a per-line
carriage-return progress line that flooded stdout / the log file with
millions of entries - fine for a desktop app, unreadable and bloating in
a container. All of it is now proper logging at DEBUG (with coarse
per-500k-line progress), so a normal run is quiet. The librarian log
level is configurable (CONJURER_LIBRARIAN_LOG_LEVEL, default INFO) and a
stdout handler is added so stays useful now that the
search no longer prints straight to stdout. Set DEBUG for full verbosity.
Also: make test_result_delivery_contract hermetic (point the durable
spool at a temp dir so it can't pollute or be poisoned by the real
result_inbox/ between runs) and gitignore the runtime spool dirs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diagnostic coverage for the 'search vanished' report. Proves the bot side
of result delivery is correct end to end (right shape reaches IN_COMM_Q;
empty result still delivered; wrong api-key -> 401 vanish; uuid mismatch
-> orphaned away from the querent), which isolates a SYSTEMATIC vanish to
transport: the librarian being unable to reach the bot's /conjurer at all
(CONJURER_MAIN_BOT). The bot Service is NodePort with no pinned nodePort
while the librarian hardcodes :32442 - and being in-cluster it should use
the Service DNS http://bot:5000 instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gitea