Librarian: drop write-only result dumps + tame search logging
CI / compile (pull_request) Successful in 9s
CI / unit (pull_request) Successful in 23s
CI / integration (pull_request) Successful in 27s
build / build (push) Successful in 33s
CI / compile (push) Successful in 13s
CI / unit (push) Successful in 33s
CI / integration (push) Successful in 27s

Two hygiene fixes on top of the work-queue OOM bound:

Result dumps: cr_results / rr_results / s_results.json were write-only
(nothing reads them) yet accumulated EVERY search forever and json.load'd
the whole growing file on each write - unbounded RAM and PVC growth, and
for a deep search the raw cr_results dump is hundreds of MB. They are now
off by default (CONJURER_LIBRARIAN_DEBUG_DUMPS) and, when enabled, are
overwritten with just the latest search - never loaded or accumulated.
not_in_db.json is untouched: it's a real queue the scraper drains.

Search logging: search_bot logged via print(), including a per-line
carriage-return progress line that flooded stdout / the log file with
millions of entries - fine for a desktop app, unreadable and bloating in
a container. All of it is now proper logging at DEBUG (with coarse
per-500k-line progress), so a normal run is quiet. The librarian log
level is configurable (CONJURER_LIBRARIAN_LOG_LEVEL, default INFO) and a
stdout handler is added so  stays useful now that the
search no longer prints straight to stdout. Set DEBUG for full verbosity.

Also: make test_result_delivery_contract hermetic (point the durable
spool at a temp dir so it can't pollute or be poisoned by the real
result_inbox/ between runs) and gitignore the runtime spool dirs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit was merged in pull request #16.
This commit is contained in:
2026-08-02 20:19:49 +02:00
parent 40605b959f
commit c5643aa28f
4 changed files with 68 additions and 65 deletions
+5
View File
@@ -202,3 +202,8 @@ not_in_db.json
rr_results.json
s_results.json
*.bak.DS_Store
# Durable result-delivery spool (runtime, per-deploy)
result_inbox/
delivered_uuids/
outbox/