Diagnostic coverage for the 'search vanished' report. Proves the bot side
of result delivery is correct end to end (right shape reaches IN_COMM_Q;
empty result still delivered; wrong api-key -> 401 vanish; uuid mismatch
-> orphaned away from the querent), which isolates a SYSTEMATIC vanish to
transport: the librarian being unable to reach the bot's /conjurer at all
(CONJURER_MAIN_BOT). The bot Service is NodePort with no pinned nodePort
while the librarian hardcodes :32442 - and being in-cluster it should use
the Service DNS http://bot:5000 instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two refinements to the librarian health/delivery story, matching how it
actually behaves under load:
1. Busy-aware ping (case b - broken return path). A ping arriving while
the worker is grinding a search no longer queues behind it (which made
a healthy-but-busy librarian time out and look dead). The librarian
tracks worker_busy and, when set, pongs back IMMEDIATELY without
touching the queue. Being busy is fine - you can keep piling searches
on. The ping still travels the librarian->bot return path, so it keeps
catching the one thing it must: a disrupted/incompatible return path
where queries vanish. Idle pings still go through the internal queue.
2. Per-query watchdog (case a - finished but result lost). The librarian
now tracks every search uuid's lifecycle (queued -> processing ->
gone) in active_queries, exposed via a new POST /query_status. After
dispatching a search the bot records it in self.pending; watch_pending
polls /query_status for each. While the librarian still knows the uuid
the search is progressing - left alone. The moment a uuid VANISHES
there while still pending on the bot, its result was computed but never
delivered: after a grace window (to rule out an in-flight result) the
bot posts a notice to the channel - but ONLY then. A normally delivered
result is popped from self.pending by check_data_q and never flagged.
Hardening: the worker's search body is now wrapped in try/except/finally
so a crashing search can't kill the worker thread (which would freeze the
queue), and worker_busy / active_queries are always cleared. The grace
logic lives in a dependency-free librarian_watchdog.pending_verdict so it
is unit-testable without discord/pdf libs. /ping and /query_status are
plain (sync) views so they run without flask[async].
Tests: unit test_librarian_watchdog (verdict transitions); integration
test_librarian_query_lifecycle (query_status known/unknown + auth,
idle-ping-queues, busy-ping-pongs-directly). Suite: 28 integration + 48
unit green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The librarian health check was a plain GET to '/', which only proved
Flask was listening - not that the service could actually take a query,
run it through its internal queue+worker, and answer back. So the cog
could load against a librarian whose worker was wedged or that couldn't
reach the bot on the return leg.
Replace it with a ping that travels the SAME path a real search does, on
both sides:
bot: QueryControl -> OUT_COMM_Q -> scan_queue -> awaiting_q
librarian: POST /ping -> librarian_queue -> worker pulls it off
(no Crossref/DOI search) -> pongs back with the same uuid
bot: /conjurer -> incoming_q -> scan_incoming matches uuid, wakes waiter
The cog enables only when that whole loop closes within 3s. This also
proves the librarian->bot return path, which a GET never did.
Safety: uuid is random per ping; the wait and POST are both bounded so
startup can't stall; a pong that finds no waiter is dropped (never
orphaned into IN_COMM_Q, which would make the cog post a bogus 'no
results' message); and a ping whose pong never returns is swept out of
awaiting_q after PING_TTL_SECONDS so nothing leaks. All awaiting_q writes
stay within scan_queue (append) and scan_incoming (remove) - no locks,
no cross-thread mutation.
Integration tests cover: OK round-trip, timeout when accepted-but-no-pong,
unreachable, non-200, orphan-pong-dropped, and that real results still
reach IN_COMM_Q. Suite: 24 integration + 41 unit green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a service group's cog stays disabled the log now says WHY:
_service_health returns the actual connection error (ConnectionError
= refused/down, ConnectTimeout = firewall/slow, gaierror = DNS/wrong
host) instead of a bare 'unreachable', and on_ready logs the resolved
FILE/LIBRARIAN/RADIO addresses so an env-var that never reached the
process (address falls back to the 192.168.1.15:5000 default) is
obvious at a glance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Seven confirmed defects from the code audit, each small and low-risk.
* ai_functions.get_random_cyclic_message: random.randint(0, len(CYCLIC_WORDS))
is inclusive -> could return len -> IndexError. Now randrange(len) + guard on
an empty CYCLIC_WORDS.
* librarian_commands.get_image_sadox: random.randrange(0, len(res)-1) never
picked the last comic and raised ValueError('empty range') on a single file.
Now randrange(len) + an empty-dir guard.
* ai_commands image generation: every DALL-E error branch replied but did not
return, so control fell through to `if response:` with response unbound ->
UnboundLocalError right after the friendly message. Each branch now returns;
response is pre-initialised; and PermissionDeniedError no longer passes a
(message, text) tuple as a single arg.
* search_bot DOI match: `item["DOI"] in data` was a substring test, so a DOI
that is a prefix of a longer one (10.1/1 vs 10.1/12) produced a false 'exists'
hit. Now matches the line's first whitespace token exactly, via an O(1) dict
index built once per consumer (also removes the O(queried-DOIs) per-line scan
- a real win for large databases).
* communication_subroutine.scan_incoming: matched records were never removed
from awaiting_q, so it grew unbounded over uptime and a reused UUID could
re-match a stale record. Matched records are now dropped after dispatch.
* communication_subroutine.id3: (resp.headers.get("icy-name") or "").title()
guards against a stream that omits headers (was AttributeError on None,
500-ing the /prepped_tracks "next" handler).
* betoniarka.scan_tracks: waits for the radio logs to exist instead of dying
with FileNotFoundError on a fresh deploy (which silently killed the
now-playing forwarder until a restart).
Verified: tests/unit/test_search_bot.py gains exact-match and trailing-metadata
cases; full unit job 43 passed. Remaining observations (image-gen stale
/home/pi fallback paths + dead FileNotFoundError-after-OSError branch; tailer
still vulnerable to mid-run log rotation; DOI-first-token assumption) noted for
follow-up - none are crashes on the normal path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fits the mythology pillar of the persona (Slavic/Norse/Celtic, Old Norse
phrases). New always-loaded cog oracle_commands:
* $runy [pytanie] draws three Elder Futhark runes (past/present/future, with
upright/reversed orientation - the 8 symmetric runes are never reversed) and
asks the ACTIVE AI backend to read the spread in Conjurer's voice. If the AI
is down it still shows the drawn runes with their own meanings, so the command
always answers.
* $runa_dnia gives one rune, deterministic per user per day (sha256 seed), so
it's stable if asked repeatedly - no AI call, no state file.
The full 24-rune Futhark, the draw logic and the reversal rules are pure and
unit-tested (distinct draw, non-invertible never reversed, per-day stability,
meaning fallback).
Also fixes a pre-existing unit-job breakage: test_bar_commands and
test_lore_commands each stubbed `discord` with different completeness and
shared sys.modules, so once both landed on main the one lacking `discord.ext.tasks`
shadowed the one needing it and collection failed order-dependently. A new
tests/unit/conftest.py stubs discord once, completely, before any test module -
the per-file stubs then skip. Full unit job: 41 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The conversation memory file grows forever (every chat appends a user+assistant
pair), so startup load gets slower and the disk fills. New always-loaded cog
lore_commands turns that growth into content: a background task summarises the
oldest slice into one in-character "legend" via the ACTIVE AI backend, replaces
those old messages with the summary (bounding the file, keeping continuity for
the next startup's context), archives it to legendy.json, and announces it on
Safety: the compaction transforms (build_transcript, apply_compaction) are pure
and unit-tested. The file rewrite is re-read -> back up -> atomic write with no
await in between, so a handle_response append that lands while the summary is
being generated can neither be lost (it's in the preserved tail) nor corrupt
the file (single-threaded, no interleave). A .bak is kept. Scope note: this
bounds the on-disk file (startup/disk); the in-RAM MESSAGE_TABLE is a separate
concern left untouched to avoid yanking context from a live conversation.
Commands: $zapisz_legende (Vykidailo) forces a compaction now; $legendy recalls
a random past legend. All thresholds env-overridable (CONJURER_MEMORY_COMPACT_*,
CONJURER_LEGENDS_CHANNEL). constants gains LEGENDS_FILE + config + seed; bot.py
registers the cog.
Verified: tests/unit/test_lore_commands.py covers prefix-replace/tail-keep,
preservation of appends made during summarisation, and transcript formatting +
head/tail truncation. Unit job 32 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The most in-character capability the bot has: the persona is literally a 200kg
bartender who mixes strong drinks with intriguing names. New always-loaded cog
bar_commands:
* $nalej [motyw] asks the ACTIVE AI backend (whatever $gadaj_teraz selects) to
invent one themed cocktail in Conjurer's voice - persona reused from
GPT_SETTINGS[0] as a system message, instructions as the user turn, via
handle_response request_type NONE so it never pollutes the bar's conversation
memory. Empty motyw = a surprise; "radio"/"pod muzykę" themes the drink on the
track currently playing (PREPPED_TRACKS["now_playing"]).
* every drink is appended to menu.json (new seeded state file, CONJURER_MENU_FILE
overridable) with name/theme/author/timestamp/full text - emergent bar lore.
* $menu lists the invented drinks and pours one at random from the archive.
Text-only for now; a DALL-E drink image is an easy follow-up (the render path
already exists in ai_commands, OpenAI-only).
constants gains MENU_FILE (next to pamiec.json by default) + its seed; bot.py
registers bar_commands as a core cog. Verified: tests/unit/test_bar_commands.py
covers name extraction (markers/markdown/fallback) and the menu round-trip
incl. corrupt-file tolerance; unit job 32 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two field-reported robustness gaps on top of the hang fix.
1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised
UnicodeDecodeError from readline() - which is a ValueError, so the earlier
`except OSError` did NOT catch it. The finally-sentinel meant no hang, but the
producer died mid-file with a loud traceback and every DOI after the bad byte
went unsearched. Now chunks are opened with errors="replace" (bad bytes become
U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF,
and the producer's except is broadened from OSError to Exception so no per-file
error can ever crash the thread - it's logged and the sentinel still fires.
2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly
what goes out - target URL, uuid, DOI count and the DOI list - so the librarian
log plainly shows a result was sent and what was in it. And a failed POST is no
longer fatal: a RequestException used to propagate out of the worker loop and
kill the thread, stalling every future query until restart; it's now caught and
logged, and a non-200 from the bot is logged as a warning.
Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96
byte before a valid DOI and asserting that DOI is still found (file read to
completion, not aborted). All 5 search_bot unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_musician_auth.py still probed /clear_pr_pls on the musician, but that
endpoint moved to betoniarka during the split - the musician now 404s it, so
all three tests failed 404 != 401/200. This was pre-existing debt, unrelated to
the AI/share/bridge work; it just kept the integration job red.
Split the coverage to match the current architecture:
* test_musician_auth.py exercises the same auth contract (no key -> 401, key ->
200, key unset -> open) against /get_share_list, an authenticated endpoint the
musician still serves, with a valid body so the permitted case is a clean 200
rather than a 400;
* new test_betoniarka_auth.py covers /clear_pr_pls where it now lives, pointing
PRIORITY_PLAYLIST_PATH at a tmp file so the authorised case can truncate it,
and checks /ping stays open;
* conftest.py adds conjurer_betoniarka to sys.path so the service imports.
Verified in a clean venv (pytest flask waitress requests), matching the CI
integration job: 13 passed, up from 3 failed / 6 passed. Unit suite unaffected
(23 passed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two connected features.
1) AI query interface (via the comm layer). communication_subroutine gains an
AI_QUERY_Q, a submit_ai_query() in-process entry point, and an authed
POST /ai_query endpoint ({prompt, channel_id, request_type?, username?}). The
prompt is queued and answered asynchronously by a new tasks.loop worker in the
always-loaded AI cog (Events), which calls handle_response - so it runs on
whichever backend $gadaj_teraz currently selects (GPT or Claude) - and posts the
answer to the requested channel, chunked to Discord's limit. request_type "NONE"
(default) is a clean one-shot: no persona system prompt, no memory write. The
worker starts before the OpenAI guard in cog_load, so it also runs on a
Claude-only box; cog_unload cancels it.
2) Librarian AI review. New command $wyszukaj_z_recenzja mirrors
$wyszukaj_linki_do_dokumentow but sets ai_review=True on the QueryControl, which
rides the round-trip and is matched back by UUID. When the hits return,
check_data_q sends the raw list as before, then - if flagged - hands the same
list (already in Crossref-relevance order) plus the search phrase to the AI
queue for a weighted re-rank and per-source review, delivered to the same
channel. QueryControl gains an ai_review flag (default False, so the orphan path
and all existing callers are unaffected).
Confirmed separately (and noted in the docs): the DOI list the AI receives is
pre-sorted by Crossref relevance - the librarian pipeline only filters (drops
title-less items) and splits (in-db / not-in-db), never re-sorts, and relies on
insertion-ordered dicts (Py 3.7+).
Verified: /ai_query auth (401/200/400/open), submit_ai_query and the queued
dict shape, and the QueryControl flag - via a Flask test client and
tests/integration/test_ai_query_endpoint.py (5 tests, all pass; integration
suite 11 passed, the 3 failures are the pre-existing /clear_pr_pls musician
tests fixed on a separate branch). Full first-party compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
search_bot conflated MAXTHREADS into two jobs at once - how many chunk files to
read (files 0..MAXTHREADS-1) AND how many producer sentinels to wait for - so
the two had to match exactly. Set too low it silently skipped trailing chunks;
set too high (or with any chunk missing/unreadable) a producer crashed before
emitting its sentinel, the consumers' count never reached the threshold, and
search_for_doi hung on join() forever. The idle-timeout failsafe that was meant
to break a starved consumer was dead code: `if empty_counter > 5: ... elif
empty_counter > 10: break` - >10 implies >5, so the elif never ran.
Fix, three layers:
* auto-discover the chunk files present (discover_chunk_files: <n>_chunk.txt in
numeric order) instead of range(0, MAXTHREADS). All files are read regardless
of count, and no producer is ever pointed at a missing file;
* the sentinel threshold is now the number of producers actually started, so it
can't drift from what's emitted;
* producers emit their sentinel in a finally, so even a crash (missing/unreadable
chunk) can't starve the count; and the idle backstop is reordered so it can
actually fire (>EMPTY_LIMIT seconds) as a last resort.
MAXTHREADS is deprecated and unused (kept only so old env files don't break);
docs/env updated to say chunk files are auto-discovered.
For the reported case (MAXTHREADS=40, files 0..43): before, files 40-43 were
silently never searched, and any run that referenced a missing chunk hung
forever. After, all 44 are searched and it always terminates.
Verified in a pytest-only venv (tests/unit/test_search_bot.py): DOI in a
trailing chunk is found; an unreadable chunk still terminates; empty dir returns
at once; discovery is numeric-sorted. Full unit job 27 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reading radio_conjurer.liq shows request.playlist is not a Liquidsoap playlist
at all - it is a drop box. queue_processing() runs every 60s, reads every line,
pushes each into request.queue() as a URI, then deletes the file and recreates
it empty. That changes what writing to it means, so the mode is now named and
documented for what it is.
BRIDGE_MODE=dropbox (with "playlist" kept as an alias) appends the exact
translated path to that file. Because the lines are pushed as URIs it is an
exact hand-off - no keyword search - and it involves neither betoniarka nor the
bot, just the file and Liquidsoap. The liq also puts requests_queue first in the
fallback and does not apply the check_next replay guard to it, so a request
interrupts the rotation and plays even if the track ran recently; both are now
documented rather than left to be discovered.
Fixes found while wiring this up:
* the existence check was unconditional while the library mount was documented
as optional, so a drop-box-only setup could never queue anything. It is now
BRIDGE_VERIFY_FILE_EXISTS (auto|true|false), defaulting to checking only when
the library is actually visible;
* a missing parent dir was silently created, which would swallow requests into
the container's own filesystem when the radio's data dir was not mounted. It
is now a hard error naming the likely cause.
Verified: alias resolves; append/drain/append cycle against a simulation of the
liq's read-remove-recreate; both misconfigurations raise instead of silently
succeeding; drop-box-only path works with the library absent; api mode
unchanged, still prepending the sentinel that wyszukaj() discards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Star a track in Navidrome and it lands in betoniarka's request queue.
EXTERNAL component: imports nothing from Conjurer and bypasses the Discord bot
and its API entirely. It speaks only to Navidrome's Subsonic API and to the
radio operator, so the bot can be down and this keeps working.
Not shipped as a .ndp plugin, deliberately. Navidrome's plugin capabilities are
MetadataAgent, Scrobbler, Lyrics, SonicSimilarity, TaskWorker, Lifecycle,
SchedulerCallback and WebSocketCallback - there is no UI extension capability
(a plugin cannot add a button) and no star/love event to react to. The only
plugin-shaped alternative, Scrobbler, sees every track played, which is a
firehose rather than a one-click request. So the trigger is Navidrome's own
star control, which also works from any Subsonic client including phones.
The README documents this with sources.
Both sides index the same library under different mount points, so every path
is translated between the two roots; paths reported relative to Navidrome's
library are handled as well as absolute ones.
Two delivery modes: "api" (default) posts to /request_radio_file and needs no
shared filesystem, at the cost of the radio re-finding the track by keywords;
"playlist" appends the exact translated path and is exact but needs the radio's
data dir mounted. The api path prepends a sentinel token because betoniarka's
wyszukaj() drops lista_slow[0], where the Discord command word normally sits.
Verified: path translation for both absolute and relative forms; keyword
extraction (no regex metacharacters, extension dropped, Polish characters
intact); and an end-to-end check feeding the generated keywords through
betoniarka's real wyszukaj() against a library seeded with near-miss traps
(same artist, live version of the same title, same album name under another
artist) - it resolved to the exact intended file.
Docker/compose/systemd install paths documented. Not run against a live
Navidrome or radio - no instance reachable from here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Python half of the short-lived share links lived in the repo; the Apache
config and the cron entries that make it work were hand-placed on the host, so
the feature could not be rebuilt from a checkout. This adds the missing half.
Scripts (defaults unchanged, so existing bare-metal cron keeps working):
* scan_shares.py / revoke_shares.py take their paths from CONJURER_SHARE_*
instead of hardcoding the Pi layout, and create their parent dirs;
* the revoke TTL is now CONJURER_SHARE_TTL_SECONDS. Its --help claimed "2min"
while the code used a hardcoded 3600 - the help text now reports the real,
configured value.
New share service:
* docker/Dockerfile.share - Apache + the two jobs, reusing the same scripts
rather than forking copies;
* docker/share-vhost.conf.tpl - the previously undocumented Apache half.
Three settings are load-bearing and commented as such: +FollowSymLinks (the
shares ARE symlinks), -Indexes (a listing would expose every live token), and
a deny rule for dotfiles (revoke_shares.py keeps .downloads.json - a map of
every live token - inside the served directory);
* docker/entrypoint.share.sh - renders the vhost, seeds the index on first run,
then runs scanner/revoker in sleep loops beside Apache (no cron, so their
output shows up in docker logs);
* compose + env example, incl. the two volumes that MUST be shared with the
musician (it creates the links and reads the index).
docs/deployment/FILE_SHARING.md documents the mechanism, both deployment routes
(docker and existing host Apache), the Discord command, why each Apache setting
matters, verification commands, and the known limitations - notably that a link
nobody ever downloads is never revoked, since the TTL starts at first download.
Verified: scanner and revoker exercised end-to-end against temp dirs (index
built; token recorded from a combined-format log line and the symlink unlinked).
Compose file not validated - no docker on this machine.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
file_search_functions was the only bot-side service client that hardcoded the
musician's LAN address and sent no auth header. Both musician endpoints it
calls (/get_share_list, /get_share_links) run _authorize_request(), so the
whole file-share feature 401'd on any deployment that set CONJURER_API_KEY -
which the deployment guides instruct you to do - and the hardcoded IP made a
musician on another host (the Proxmox/docker split) unreachable regardless.
Use FILE_SERVICE_ADDRESS + service_headers() like music_functions does, add
the two endpoint paths to constants alongside the existing ones, and set an
explicit timeout so a wedged musician can't hang the calling cog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lets CI be triggered on demand from the Actions tab or `gh workflow run`,
instead of pushing a commit just to exercise the pipeline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Quality-of-life: checking which AI backend is live no longer requires
switching to it. Bare `$gadaj_teraz` replies with the active config and the
selectable ones; that read-only path is open to everyone, while switching
stays Vykidailo-gated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the two deployment guides to cover the new env vars and the runtime
switch: ANTHROPIC_API_KEY / netrc machine 'anthropic', CONJURER_AI_CONFIG
(gpt|claude), the $gadaj_teraz command, and the note that DALL-E + personal
assistants stay on OpenAI regardless of the switch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the bot's AI chat pipeline (ai_functions.handle_response) to talk to
either OpenAI or the Anthropic Messages API, chosen by one active-config
switch. Behaviour on the default "gpt" config is unchanged.
constants.py:
* guarded `import anthropic` + CLAUDECLIENT (mirrors OPENAICLIENT), netrc
machine 'anthropic' / ANTHROPIC_API_KEY;
* CLAUDE_LATEST_MODEL / CLAUDE_CHEAP_MODEL (opus-4-8 / haiku-4-5);
* AI_CONFIGS + DEFAULT_AI_CONFIG loaded from an optional 3rd element of
system_gpt_settings.json (backward compatible - a 2-element file falls
back to built-in defaults, active "gpt"). Single switch: CONJURER_AI_CONFIG
env > settings "active" > "gpt".
ai_functions.py:
* provider_generate() dispatches to OpenAI (unchanged openai_call) or the new
_anthropic_call() (splits system out, alternating messages, max_tokens,
temperature omitted - Opus 4.8 rejects sampling params);
* AIError normalises both SDKs' exceptions into one category set so
handle_response keeps its single set of in-character error replies;
* select_model() reads the active config; legacy "gpt-4o" default auto-maps
to the active provider's model so the switch actually changes the backend;
* set_active_ai_config()/list_ai_configs() with best-effort persistence back
into system_gpt_settings.json index 2.
ai_commands.py:
* $gadaj_teraz <config> hybrid command (Vykidailo-gated) switches backend at
runtime;
* graceful guards when OPENAICLIENT is None: personal assistants (OpenAI
Assistants API) and DALL-E image gen degrade instead of crashing, so a
Claude-only deployment boots.
system_gpt_settings.json: add the configs block (gpt/claude/_template) as the
collection point for future backends. requirements_bot.txt: add anthropic.
bot.env.example: ANTHROPIC_API_KEY + CONJURER_AI_CONFIG. Unit tests cover the
message splitter, model selection, config listing, and error mapping.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(Re-applied cleanly on top of main: PR #18 was rebase-merged, so the branch
that became #19 conflicted on the already-landed commits. This carries ONLY
the librarian state fix - the sole content difference between that branch
and main - so nothing else is touched or lost.)
The worker threads open cr_results/rr_results/not_in_db/s_results.json in
place ('r+'), crashing with FileNotFoundError in a fresh container. New
conjurer_librarian/lib_paths.py resolves all four under a persistent dir
(CONJURER_LIBRARIAN_STATE_DIR, default /lib_temp_files) and seeds missing
ones with '{}' on import; shared by conjurer_librarian.py and scrape_bot.py
(no circular import). ndb_database/database initialised to {} before load so
a corrupt persisted file degrades to empty instead of NameError. search_bot
/search_bot2 open the DOI DB 'r' not 'r+' so /doi can stay read-only. Docker:
STATE_DIR env + VOLUME, compose mounts /srv/librarian/state:/lib_temp_files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Permissions post-mortem that motivated this: the musician wrote radio
playlists AS ROOT onto a ROOT-OWNED network share which liquidsoap then
read AS USER 'radio' - chown fails on such shares by design (root squash /
uid mapping), so the radio came up and died on the playlists. The fix is
structural: the playlist WRITER now lives in the same container as the
READER, as the same user, on a local volume. No shared partition, no
chown, no uid mapping.
New: conjurer_betoniarka/betoniarka.py - runs inside the radio container
(started by the entrypoint as user 'radio', port 5005):
- library scan -> all_playlist/hit playlists with LOCAL container paths
(start + every 24h + authenticated GET /rescan)
- bot-facing radio API moved from the musician: /add_to_priority,
/create_priority_playlist, /request_radio_file, /clear_pr_pls,
plus GET /ping (health) and /stream (web page)
- radio_log/persistence tailer forwarding play events to the bot's
/prepped_tracks with the shared API key (bot-unreachable = logged, not fatal)
Musician: pure Discord music player now - keeps /mp3, /update_mp3,
/get_music and the file-share endpoints; all radio playlist writing, radio
paths/env and the tailer removed.
Bot: new CONJURER_RADIO_SERVICE (defaults to CONJURER_FILE_SERVICE so
un-split deployments keep working); radio_commands targets it; separate
'radio' health-gate group on betoniarka /ping (musician group now covers
music_commands + file_search_commands only).
Docker: betoniarka baked into the radio image (python3 + flask/waitress/
requests from Debian debs), port 5005 exposed, entrypoint starts it via
setpriv as 'radio'; data-volume chown is now best-effort with a loud
warning (keep the volume local); docs get the post-mortem + wiring.
Verified: py_compile everything; functional stub tests - rescan writes
local-path playlists, wyszukaj scores and appends to priority, auth
401/ok, tailer forward carries the API key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Liquidsoap refuses to start as root (init: security exit). Instead of
the settings.init.allow_root override, the image now has a dedicated
'radio' user (audio + pulse-access groups) and the entrypoint drops
privileges via setpriv after doing its root-only work (volume seeding,
icecast config render/start, pulse start, chown of the data volume).
The icecast secret is made group-readable (640 root:radio) because the
script parses it directly.
- The opam root moved from /root/.opam to /opt/opam so the liquidsoap
binary AND its stdlib .liq files are readable by the radio user.
NOTE: this invalidates the cached opam build layer - next build
recompiles liquidsoap (~15-20 min).
- pulseaudio (PULSE_MODE=internal) now starts with
--disallow-module-loading: system.pa startup modules still load, only
later client-requested loads are blocked, and the system-mode warning
goes away. The 'forcibly disabling SHM mode' notice is inherent to
system mode and harmless (documented).
Keeps the user's libcurl4-gnutls-dev build-dep fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review of the first radio-container cut:
- radio_conjurer.liq: replace the cargo-culted /home/pi/Conjurer and
/home/pi/MediaFolder paths with the container layout -
/srv/betoniarka/data (playlists, script.params, persistence/radio logs),
/srv/betoniarka/music (library + emergency track) and
/srv/betoniarka/secrets/icecast_credentials.json;
interactive.persistent now uses an absolute path;
output.icecast targets host="localhost" (icecast lives in the container)
- Icecast2 now runs INSIDE the radio container: entrypoint renders
/etc/icecast2/icecast.xml from docker/icecast.xml.tpl, filling
source/admin/relay passwords from the secret provisioned at install time
(same pattern as the bot's netrc); optional admin_password/relay_password
fields default to password; icecast starts as its unprivileged user;
port 8000 exposed for listeners; extra_hosts hack removed
- compose.radio.yaml: same-path mounts for /srv/betoniarka/{data,music,secrets}
- ffmpeg.pref removed from the repo (the pin only mattered against the
RPi OS repo; bookworm ships the right FFmpeg 5.x natively) - verdict
preserved in the docs and the legacy installer note updated
- docs: updated radio section (volumes, secret provisioning, wire-up)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dockerises the radio_conjurer.liq environment on Debian bookworm:
- Dockerfile.radio builds Liquidsoap through opam with build args for
OPAM_VERSION (static binary), OCAML_VERSION (default 4.14.2; 2.1.x needs
OCaml 4.x - prod ran 4.13.0, pass it for parity) and
LIQUIDSOAP_VERSION (default 2.1.4, matching the script)
- LIQ_OPAM_PACKAGES installs exactly the features the script uses:
mad+lame (mp3), cry (icecast), taglib (tags/replaygain), pulseaudio
(mic/out), samplerate, inotify (watch-reload), ffmpeg
- ffmpeg.pref verdict: the Pi pin forced Debian's FFmpeg 5.x family
(libavcodec59...) over RPi OS repo builds because the ocaml-ffmpeg
bindings are compiled against those sonames; bookworm ships them
natively so the container needs no pin (documented, not copied)
- pulse-access/audio group memberships baked in (were manual on the Pi);
PULSE_MODE=internal runs a system-wide pulse with a null sink so the
unmodified script works on a headless VM (mic = silence), host mode
mounts the real socket, none for edited scripts
- mounts mirror the script's hardcoded paths (/home/pi/Conjurer,
/home/pi/MediaFolder/mp3) so radio_conjurer.liq runs unmodified;
entrypoint seeds script+params on first run (never overwrites), creates
missing playlists, seeds placeholder icecast credentials (loud warning)
and generates a silent emergency mp3 when the single() file is missing
- compose.radio.yaml: extra_hosts maps the script's host="radio" to
ICECAST_HOST_IP; exposes 54321 (bot's RADIO_HARBOR /skip) and 9999
- docs: radio section with build args, pulse modes and the ffmpeg.pref /
groups verdicts
Part 2 of this PR (splitting bot<->radio comms out of the musician into a
radio+betoniarka container) follows on this branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cog did its own netrc read from a hardcoded /home/pi/.netrc at import
time, crashing on any other host. Now:
- constants.py: ASSEMBLYAI_API_KEY resolved like every other token
(env ASSEMBLYAI_API_KEY -> netrc machine 'assemblyai' at
CONJURER_NETRC_FILE); new TRANSCRIPTS_PATH (env
CONJURER_TRANSCRIPTS_PATH, defaults next to the log file, created by the
runtime layout)
- voice_recognition_commands.py: drop the hardcoded netrc read and
transcript dir; when the key is missing raise a clear RuntimeError so
the guarded loader disables ONLY this cog with a readable reason
- bot.env.example: document ASSEMBLYAI_API_KEY
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
log_handler belongs to Client.run() (which configures logging and then
calls start()); passing it to start() raised TypeError at launch. This was
inherited from the never-run dockerised variant's thin_client.py. We
configure our own handlers, so simply drop the kwarg.
Also set logger.propagate=False: a dependency calls logging.basicConfig(),
so every 'discord' record was printed twice (our format + root's).
Verified with a strict-signature stub client (start(token, *, reconnect)).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The restart-loop with only 'Starting discord bot' in the logs was caused by
gather(return_exceptions=True) eating the client.start() exception: a failed
login looked like a clean exit and docker just restarted the container.
bot.py:
- _run_bot: catch discord.LoginFailure / PrivilegedIntentsRequired with
CRITICAL messages telling exactly what to fix (token entry / Developer
Portal intents), full traceback for anything else, then re-raise
- main() returns an exit code; every task exception is re-logged as the
LAST line in docker logs ('Task died: ...'); process exits 1 on failure
- log the token source (env vs netrc) and its length at startup to catch
truncated/wrong-field netrc entries
- _run_comm_subroutine: comm crash is logged loudly but no longer relies on
gather to surface it (bot keeps running - Discord side is independent)
communication_subroutine.py (crash found by the new stub tests):
- comm_subroutine used 'logger' before its local assignment
(UnboundLocalError killed the whole comm layer at every startup)
- re-comment the flask_debug thread: it binds the same host:port as
waitress, so running both dies with 'address in use'
Verified via stub-injected client: LoginFailure -> CRITICAL + exit 1,
PrivilegedIntentsRequired -> CRITICAL + exit 1, unknown exception -> full
traceback + exit 1, clean shutdown -> exit 0; comm layer no longer crashes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Container/native startup died silently on any missing file/service. Now:
constants.py - self-healing runtime layout at import:
- create missing dirs (log dir, LOGSTORE, GRAPHICS_PATH, MUSIC_FOLDER)
- seed missing state files from the repo templates shipped next to
constants.py (settings/system_gpt/pamiec/pamiec_muzyki/accident_log),
falling back to safe empty JSON; existing files are NEVER overwritten
bot.py:
- log to stdout too, so 'docker logs' finally shows the crash reason
- missing Discord token = loud sys.exit with mount/env instructions
(was: silent return -> container crash-loop with empty logs)
- every cog loads independently (one broken cog = skipped with traceback,
bot continues)
- musician/librarian cogs are health-gated: enabled only when the service
answers HTTP; a watchdog re-checks every 5 min and enables them the
moment the service comes alive (no restart needed); tree re-synced
- on_ready reconnects no longer re-load extensions
requirements_conan.txt + Dockerfile.bot: aiomcrcon (Python <=3.11 only)
moved to best-effort extras so the 3.13 image builds clean and the
conanjurer cog stays dormant without it.
DOCKER_PROXMOX.md: startup model (core vs gated cogs) + crash-loop
troubleshooting incl. the 'disappearing files' checklist (nothing in the
stack deletes host files; bind mount = live state).
Verified: fresh-volume seeding creates dirs+templates, existing files
untouched, missing-token exits with FATAL message, health-gating logic
(stub-based runpy tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gitea
Michal Tuszowski