Two hygiene fixes on top of the work-queue OOM bound:
Result dumps: cr_results / rr_results / s_results.json were write-only
(nothing reads them) yet accumulated EVERY search forever and json.load'd
the whole growing file on each write - unbounded RAM and PVC growth, and
for a deep search the raw cr_results dump is hundreds of MB. They are now
off by default (CONJURER_LIBRARIAN_DEBUG_DUMPS) and, when enabled, are
overwritten with just the latest search - never loaded or accumulated.
not_in_db.json is untouched: it's a real queue the scraper drains.
Search logging: search_bot logged via print(), including a per-line
carriage-return progress line that flooded stdout / the log file with
millions of entries - fine for a desktop app, unreadable and bloating in
a container. All of it is now proper logging at DEBUG (with coarse
per-500k-line progress), so a normal run is quiet. The librarian log
level is configurable (CONJURER_LIBRARIAN_LOG_LEVEL, default INFO) and a
stdout handler is added so stays useful now that the
search no longer prints straight to stdout. Set DEBUG for full verbosity.
Also: make test_result_delivery_contract hermetic (point the durable
spool at a temp dir so it can't pollute or be poisoned by the real
result_inbox/ between runs) and gitignore the runtime spool dirs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pod restarted spontaneously mid-search (no liveness probe is set, so
it was the kernel OOM-killer against the 1Gi limit). Cause: search_bot
built its work queue with maxsize 35_500_000. The producers stream the
WHOLE DOI database (tens of millions of lines across chunks) into it while
a few consumers drain, so the queue could buffer gigabytes of lines -
blowing the 1Gi container and taking the whole in-flight search with it.
Bound the queue (default 100k lines, env CONJURER_LIBRARIAN_WORKQ_SIZE),
so producers backpressure to consumers and RAM stays in the low MB.
Because a bounded queue means a producer can now block on a FULL queue,
make the producer's put timeout-poll the TERM sentinel, so a full queue
whose consumers have already finished (all DOIs found) can never deadlock
it. New test pins that: tiny queue + target on line 1 + thousands of
trailing decoys still terminates and finds the target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Librarian.__init__ reads the Crossref contact from CONJURER_CROSSREF_MAILTO,
then tries to override it from a 'crossref' netrc entry. When no netrc is
mounted (the normal container setup - default /root/.netrc) the read raises
FileNotFoundError and it logged 'Crossref credentials missing in netrc ...'
on EVERY search, even though the env var was set and used. Pure noise.
Only warn when there is genuinely no contact from either source (env unset
AND netrc unreadable) - which is also the case that then raises. When the
env var is set, a missing netrc is expected and logged at debug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An 8h search result must survive a transient bot outage, an api/address
misroute, or a restart of either side. Make the librarian->bot result
path durably at-least-once with idempotent rendering:
Shared: durable_queue.DiskQueue - a dependency-free, atomically-written,
one-file-per-key disk queue (unit-tested), shared by both images
(added to Dockerfile.librarian; the bot already COPYs *.py).
Librarian (sender): finished results go to a persistent OUTBOX before
sending; delivery retries with backoff; an entry is removed only on a
positive ACK; a resender thread keeps flushing the OUTBOX, so a result
survives a bot outage AND a librarian restart (OUTBOX is on the state
volume) - it simply keeps trying until acked.
Bot (receiver): /conjurer is now idempotent and durable - each result is
persisted to an INBOX before acking and only queued if its uuid was not
already delivered (dropped as a duplicate) or already pending. Once the
cog actually renders it, mark_delivered() records the uuid and clears the
inbox, so the librarian's resends become no-ops. On startup the bot
replays any accepted-but-unrendered result from the INBOX, so a bot crash
mid-flight doesn't lose it. Pongs stay ephemeral.
Together: the librarian keeps a result until the bot confirms it; the bot
keeps it until it is on screen; duplicates never double-render. Combined
with the deploy return-path fix, an expensive result no longer vanishes.
Tests: unit test_durable_queue; integration test_librarian_outbox
(retry/backoff, resend survives outage) and test_result_durable_delivery
(persist, dedup pending, dedup delivered, replay, pong not persisted).
Suite: 55 unit + 39 integration green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two refinements to the librarian health/delivery story, matching how it
actually behaves under load:
1. Busy-aware ping (case b - broken return path). A ping arriving while
the worker is grinding a search no longer queues behind it (which made
a healthy-but-busy librarian time out and look dead). The librarian
tracks worker_busy and, when set, pongs back IMMEDIATELY without
touching the queue. Being busy is fine - you can keep piling searches
on. The ping still travels the librarian->bot return path, so it keeps
catching the one thing it must: a disrupted/incompatible return path
where queries vanish. Idle pings still go through the internal queue.
2. Per-query watchdog (case a - finished but result lost). The librarian
now tracks every search uuid's lifecycle (queued -> processing ->
gone) in active_queries, exposed via a new POST /query_status. After
dispatching a search the bot records it in self.pending; watch_pending
polls /query_status for each. While the librarian still knows the uuid
the search is progressing - left alone. The moment a uuid VANISHES
there while still pending on the bot, its result was computed but never
delivered: after a grace window (to rule out an in-flight result) the
bot posts a notice to the channel - but ONLY then. A normally delivered
result is popped from self.pending by check_data_q and never flagged.
Hardening: the worker's search body is now wrapped in try/except/finally
so a crashing search can't kill the worker thread (which would freeze the
queue), and worker_busy / active_queries are always cleared. The grace
logic lives in a dependency-free librarian_watchdog.pending_verdict so it
is unit-testable without discord/pdf libs. /ping and /query_status are
plain (sync) views so they run without flask[async].
Tests: unit test_librarian_watchdog (verdict transitions); integration
test_librarian_query_lifecycle (query_status known/unknown + auth,
idle-ping-queues, busy-ping-pongs-directly). Suite: 28 integration + 48
unit green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The librarian health check was a plain GET to '/', which only proved
Flask was listening - not that the service could actually take a query,
run it through its internal queue+worker, and answer back. So the cog
could load against a librarian whose worker was wedged or that couldn't
reach the bot on the return leg.
Replace it with a ping that travels the SAME path a real search does, on
both sides:
bot: QueryControl -> OUT_COMM_Q -> scan_queue -> awaiting_q
librarian: POST /ping -> librarian_queue -> worker pulls it off
(no Crossref/DOI search) -> pongs back with the same uuid
bot: /conjurer -> incoming_q -> scan_incoming matches uuid, wakes waiter
The cog enables only when that whole loop closes within 3s. This also
proves the librarian->bot return path, which a GET never did.
Safety: uuid is random per ping; the wait and POST are both bounded so
startup can't stall; a pong that finds no waiter is dropped (never
orphaned into IN_COMM_Q, which would make the cog post a bogus 'no
results' message); and a ping whose pong never returns is swept out of
awaiting_q after PING_TTL_SECONDS so nothing leaks. All awaiting_q writes
stay within scan_queue (append) and scan_incoming (remove) - no locks,
no cross-thread mutation.
Integration tests cover: OK round-trip, timeout when accepted-but-no-pong,
unreachable, non-200, orphan-pong-dropped, and that real results still
reach IN_COMM_Q. Suite: 24 integration + 41 unit green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Seven confirmed defects from the code audit, each small and low-risk.
* ai_functions.get_random_cyclic_message: random.randint(0, len(CYCLIC_WORDS))
is inclusive -> could return len -> IndexError. Now randrange(len) + guard on
an empty CYCLIC_WORDS.
* librarian_commands.get_image_sadox: random.randrange(0, len(res)-1) never
picked the last comic and raised ValueError('empty range') on a single file.
Now randrange(len) + an empty-dir guard.
* ai_commands image generation: every DALL-E error branch replied but did not
return, so control fell through to `if response:` with response unbound ->
UnboundLocalError right after the friendly message. Each branch now returns;
response is pre-initialised; and PermissionDeniedError no longer passes a
(message, text) tuple as a single arg.
* search_bot DOI match: `item["DOI"] in data` was a substring test, so a DOI
that is a prefix of a longer one (10.1/1 vs 10.1/12) produced a false 'exists'
hit. Now matches the line's first whitespace token exactly, via an O(1) dict
index built once per consumer (also removes the O(queried-DOIs) per-line scan
- a real win for large databases).
* communication_subroutine.scan_incoming: matched records were never removed
from awaiting_q, so it grew unbounded over uptime and a reused UUID could
re-match a stale record. Matched records are now dropped after dispatch.
* communication_subroutine.id3: (resp.headers.get("icy-name") or "").title()
guards against a stream that omits headers (was AttributeError on None,
500-ing the /prepped_tracks "next" handler).
* betoniarka.scan_tracks: waits for the radio logs to exist instead of dying
with FileNotFoundError on a fresh deploy (which silently killed the
now-playing forwarder until a restart).
Verified: tests/unit/test_search_bot.py gains exact-match and trailing-metadata
cases; full unit job 43 passed. Remaining observations (image-gen stale
/home/pi fallback paths + dead FileNotFoundError-after-OSError branch; tailer
still vulnerable to mid-run log rotation; DOI-first-token assumption) noted for
follow-up - none are crashes on the normal path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two field-reported robustness gaps on top of the hang fix.
1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised
UnicodeDecodeError from readline() - which is a ValueError, so the earlier
`except OSError` did NOT catch it. The finally-sentinel meant no hang, but the
producer died mid-file with a loud traceback and every DOI after the bad byte
went unsearched. Now chunks are opened with errors="replace" (bad bytes become
U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF,
and the producer's except is broadened from OSError to Exception so no per-file
error can ever crash the thread - it's logged and the sentinel still fires.
2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly
what goes out - target URL, uuid, DOI count and the DOI list - so the librarian
log plainly shows a result was sent and what was in it. And a failed POST is no
longer fatal: a RequestException used to propagate out of the worker loop and
kill the thread, stalling every future query until restart; it's now caught and
logged, and a non-200 from the bot is logged as a warning.
Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96
byte before a valid DOI and asserting that DOI is still found (file read to
completion, not aborted). All 5 search_bot unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
search_bot conflated MAXTHREADS into two jobs at once - how many chunk files to
read (files 0..MAXTHREADS-1) AND how many producer sentinels to wait for - so
the two had to match exactly. Set too low it silently skipped trailing chunks;
set too high (or with any chunk missing/unreadable) a producer crashed before
emitting its sentinel, the consumers' count never reached the threshold, and
search_for_doi hung on join() forever. The idle-timeout failsafe that was meant
to break a starved consumer was dead code: `if empty_counter > 5: ... elif
empty_counter > 10: break` - >10 implies >5, so the elif never ran.
Fix, three layers:
* auto-discover the chunk files present (discover_chunk_files: <n>_chunk.txt in
numeric order) instead of range(0, MAXTHREADS). All files are read regardless
of count, and no producer is ever pointed at a missing file;
* the sentinel threshold is now the number of producers actually started, so it
can't drift from what's emitted;
* producers emit their sentinel in a finally, so even a crash (missing/unreadable
chunk) can't starve the count; and the idle backstop is reordered so it can
actually fire (>EMPTY_LIMIT seconds) as a last resort.
MAXTHREADS is deprecated and unused (kept only so old env files don't break);
docs/env updated to say chunk files are auto-discovered.
For the reported case (MAXTHREADS=40, files 0..43): before, files 40-43 were
silently never searched, and any run that referenced a missing chunk hung
forever. After, all 44 are searched and it always terminates.
Verified in a pytest-only venv (tests/unit/test_search_bot.py): DOI in a
trailing chunk is found; an unreadable chunk still terminates; empty dir returns
at once; discovery is numeric-sorted. Full unit job 27 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(Re-applied cleanly on top of main: PR #18 was rebase-merged, so the branch
that became #19 conflicted on the already-landed commits. This carries ONLY
the librarian state fix - the sole content difference between that branch
and main - so nothing else is touched or lost.)
The worker threads open cr_results/rr_results/not_in_db/s_results.json in
place ('r+'), crashing with FileNotFoundError in a fresh container. New
conjurer_librarian/lib_paths.py resolves all four under a persistent dir
(CONJURER_LIBRARIAN_STATE_DIR, default /lib_temp_files) and seeds missing
ones with '{}' on import; shared by conjurer_librarian.py and scrape_bot.py
(no circular import). ndb_database/database initialised to {} before load so
a corrupt persisted file degrades to empty instead of NameError. search_bot
/search_bot2 open the DOI DB 'r' not 'r+' so /doi can stay read-only. Docker:
STATE_DIR env + VOLUME, compose mounts /srv/librarian/state:/lib_temp_files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Containerises the three services (each intended for its own Proxmox VM)
and adds the code changes needed to run cleanly on Linux/Docker.
Code fixes:
- constants.py: CONJURER_DATA_DIR roots all writable bot state under one
mounted volume (per-variable overrides still win; native Pi unaffected)
- conjurer_librarian/search_bot.py + scrape_bot.py: the hardcoded Windows
DOI database path (C:\Database\chunks\) is now CONJURER_LIBRARIAN_DB_PATH,
with CONJURER_LIBRARIAN_MAXTHREADS / _CHUNK also env-overridable
Docker:
- docker/Dockerfile.{bot,librarian,musician} + compose.{bot,librarian,musician}.yaml
- docker/env/*.env.example (force-added; real *.env stays gitignored)
- docker/entrypoint.bot.sh seeds default JSON state into /data only when
absent, so preserved history is never overwritten
- .dockerignore
- docs/deployment/DOCKER_PROXMOX.md: step-by-step runbook incl. preserving
the existing command/conversation history and cross-VM auth
The bot image uses the vendored yt_dlp/spotify_dl forks (they win on
sys.path over the pip packages), dropping the old sed patching.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two scaffold workflows (Python application / Python package) failed on
every PR: they installed deps from a non-existent requirements.txt, ran
flake8/pytest over the vendored yt_dlp fork (new syntax under the 3.8/3.9
matrix), and collected ad-hoc root scripts — notably test_ai.py, which is
an invalid pasted object dump (not Python).
- Remove python-app.yml / python-package.yml and the junk root scripts
(test.py, test_ai.py, test_time.py)
- Add .github/workflows/ci.yml with three PR-check jobs:
* compile — py_compile every first-party .py (no deps)
* unit — pytest on pure logic (conanjurer_functions, constants)
* integration — boot the Flask services and assert the X-Conjurer-Api-Key
auth contract (communication_subroutine + conjurer_musician)
- Add tests/ suite, pytest.ini (testpaths=tests) and conftest.py (sys.path)
Fixes surfaced by the compile gate / needed for the integration job:
- conjurer_librarian/search_bot.py + search_bot2.py: f-string reused the
same quote ({item["exists"]}) -> SyntaxError on Python < 3.12
- conjurer_musician/media_search_functions.py: made import-safe
(env-overridable paths, lazy DB load / mkdir) so the service can be
imported and tested off the Pi
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* New requirement
* SMall fixes for tunnel
* First step towards cogization
* Tmp chng
* Testing of new functions in conjurer_musician.py and radio_conjurer.liq
* Fixes!
* Fix fix
* Mostly working ?
* Requests
* Fixes
* Fix
* Radio requests
* Radio hardkor
* Fix
* Fixes
* Change log to print
* Clearing fetch failed
* Small fix
* sx
* Fix fix fix
* test
* el fixo stupido
* test
* Maybe fix ?
* FIX!
* Fix fix fix
* FIx of volume
* Formatting and additional logging for bugfix
* Formatting and additional logging for bugfix
* Change of log location
* fixing_librarian_result
* Hopefully final fix that will be needed
* Temporary
* Glebokie gardlo
* Fix in scrape_bot
* Fix that was needed.
* Scraper settings
* Small fix in scraping
* FIxing bad naming convention
* Log message fix
* Test of deep search
* Fix in deep search
* Refactoring
* WHat and idiot coded that....
Ooops. That was me. A month ago.
* Refactoring continued
* Bugfix
* FIx
* Not needed after fix
aaa
aa
Bugfixing et masse
bbb
asas
aas
asas
aa
as
qq
as
as
as
as
as
aaa
asa
nn
aa
sda
88
asa
sasa
adsa
a
as
asdas
Final version before tests
Fix1
aa
aa
fx
dx
:)
aa
Hopefully last fix
aaa
aaa
bbb
bb
dd
aa
sdae
aa
bb
aa
kk
a
Fix
fx
aa
deploy
fx
asas
bb
as
fx
fx
aa
aa
mess
kk
A
fix
aa
aa
Viwe fix
a
test
a
aa
FX
asdasdasd
:)
asa
asdasda
aa
:)
asa
aa
:)
:)
sa
as
aa
aa
aa
aaa
as
11
aa
11
xxx
dd
ds
aa
a1
aa
Test
Suppress
Move
Installer
aaa
Serwis
`12`
aa
xd
asasa
xd
aa
asa
aaa
:)
Hejka
a
xd
aa
aaas
555
aaa
asa
aa
11
aaa
aaa
as
sa
async
asda
00
123
123
aa
as
ss
as
a
sa
aa
1
RES
T
Hejko
aa
11
AA
test
Test
gitea
root