Librarian: drop write-only result dumps + tame search logging
CI / compile (pull_request) Successful in 9s
CI / unit (pull_request) Successful in 23s
CI / integration (pull_request) Successful in 27s
build / build (push) Successful in 33s
CI / compile (push) Successful in 13s
CI / unit (push) Successful in 33s
CI / integration (push) Successful in 27s
CI / compile (pull_request) Successful in 9s
CI / unit (pull_request) Successful in 23s
CI / integration (pull_request) Successful in 27s
build / build (push) Successful in 33s
CI / compile (push) Successful in 13s
CI / unit (push) Successful in 33s
CI / integration (push) Successful in 27s
Two hygiene fixes on top of the work-queue OOM bound: Result dumps: cr_results / rr_results / s_results.json were write-only (nothing reads them) yet accumulated EVERY search forever and json.load'd the whole growing file on each write - unbounded RAM and PVC growth, and for a deep search the raw cr_results dump is hundreds of MB. They are now off by default (CONJURER_LIBRARIAN_DEBUG_DUMPS) and, when enabled, are overwritten with just the latest search - never loaded or accumulated. not_in_db.json is untouched: it's a real queue the scraper drains. Search logging: search_bot logged via print(), including a per-line carriage-return progress line that flooded stdout / the log file with millions of entries - fine for a desktop app, unreadable and bloating in a container. All of it is now proper logging at DEBUG (with coarse per-500k-line progress), so a normal run is quiet. The librarian log level is configurable (CONJURER_LIBRARIAN_LOG_LEVEL, default INFO) and a stdout handler is added so stays useful now that the search no longer prints straight to stdout. Set DEBUG for full verbosity. Also: make test_result_delivery_contract hermetic (point the durable spool at a temp dir so it can't pollute or be poisoned by the real result_inbox/ between runs) and gitignore the runtime spool dirs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit was merged in pull request #16.
This commit is contained in:
@@ -202,3 +202,8 @@ not_in_db.json
|
||||
rr_results.json
|
||||
s_results.json
|
||||
*.bak.DS_Store
|
||||
|
||||
# Durable result-delivery spool (runtime, per-deploy)
|
||||
result_inbox/
|
||||
delivered_uuids/
|
||||
outbox/
|
||||
|
||||
Reference in New Issue
Block a user