(Re-applied cleanly on top of main: PR #18 was rebase-merged, so the branch
that became #19 conflicted on the already-landed commits. This carries ONLY
the librarian state fix - the sole content difference between that branch
and main - so nothing else is touched or lost.)
The worker threads open cr_results/rr_results/not_in_db/s_results.json in
place ('r+'), crashing with FileNotFoundError in a fresh container. New
conjurer_librarian/lib_paths.py resolves all four under a persistent dir
(CONJURER_LIBRARIAN_STATE_DIR, default /lib_temp_files) and seeds missing
ones with '{}' on import; shared by conjurer_librarian.py and scrape_bot.py
(no circular import). ndb_database/database initialised to {} before load so
a corrupt persisted file degrades to empty instead of NameError. search_bot
/search_bot2 open the DOI DB 'r' not 'r+' so /doi can stay read-only. Docker:
STATE_DIR env + VOLUME, compose mounts /srv/librarian/state:/lib_temp_files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Permissions post-mortem that motivated this: the musician wrote radio
playlists AS ROOT onto a ROOT-OWNED network share which liquidsoap then
read AS USER 'radio' - chown fails on such shares by design (root squash /
uid mapping), so the radio came up and died on the playlists. The fix is
structural: the playlist WRITER now lives in the same container as the
READER, as the same user, on a local volume. No shared partition, no
chown, no uid mapping.
New: conjurer_betoniarka/betoniarka.py - runs inside the radio container
(started by the entrypoint as user 'radio', port 5005):
- library scan -> all_playlist/hit playlists with LOCAL container paths
(start + every 24h + authenticated GET /rescan)
- bot-facing radio API moved from the musician: /add_to_priority,
/create_priority_playlist, /request_radio_file, /clear_pr_pls,
plus GET /ping (health) and /stream (web page)
- radio_log/persistence tailer forwarding play events to the bot's
/prepped_tracks with the shared API key (bot-unreachable = logged, not fatal)
Musician: pure Discord music player now - keeps /mp3, /update_mp3,
/get_music and the file-share endpoints; all radio playlist writing, radio
paths/env and the tailer removed.
Bot: new CONJURER_RADIO_SERVICE (defaults to CONJURER_FILE_SERVICE so
un-split deployments keep working); radio_commands targets it; separate
'radio' health-gate group on betoniarka /ping (musician group now covers
music_commands + file_search_commands only).
Docker: betoniarka baked into the radio image (python3 + flask/waitress/
requests from Debian debs), port 5005 exposed, entrypoint starts it via
setpriv as 'radio'; data-volume chown is now best-effort with a loud
warning (keep the volume local); docs get the post-mortem + wiring.
Verified: py_compile everything; functional stub tests - rescan writes
local-path playlists, wyszukaj scores and appends to priority, auth
401/ok, tailer forward carries the API key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Liquidsoap refuses to start as root (init: security exit). Instead of
the settings.init.allow_root override, the image now has a dedicated
'radio' user (audio + pulse-access groups) and the entrypoint drops
privileges via setpriv after doing its root-only work (volume seeding,
icecast config render/start, pulse start, chown of the data volume).
The icecast secret is made group-readable (640 root:radio) because the
script parses it directly.
- The opam root moved from /root/.opam to /opt/opam so the liquidsoap
binary AND its stdlib .liq files are readable by the radio user.
NOTE: this invalidates the cached opam build layer - next build
recompiles liquidsoap (~15-20 min).
- pulseaudio (PULSE_MODE=internal) now starts with
--disallow-module-loading: system.pa startup modules still load, only
later client-requested loads are blocked, and the system-mode warning
goes away. The 'forcibly disabling SHM mode' notice is inherent to
system mode and harmless (documented).
Keeps the user's libcurl4-gnutls-dev build-dep fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review of the first radio-container cut:
- radio_conjurer.liq: replace the cargo-culted /home/pi/Conjurer and
/home/pi/MediaFolder paths with the container layout -
/srv/betoniarka/data (playlists, script.params, persistence/radio logs),
/srv/betoniarka/music (library + emergency track) and
/srv/betoniarka/secrets/icecast_credentials.json;
interactive.persistent now uses an absolute path;
output.icecast targets host="localhost" (icecast lives in the container)
- Icecast2 now runs INSIDE the radio container: entrypoint renders
/etc/icecast2/icecast.xml from docker/icecast.xml.tpl, filling
source/admin/relay passwords from the secret provisioned at install time
(same pattern as the bot's netrc); optional admin_password/relay_password
fields default to password; icecast starts as its unprivileged user;
port 8000 exposed for listeners; extra_hosts hack removed
- compose.radio.yaml: same-path mounts for /srv/betoniarka/{data,music,secrets}
- ffmpeg.pref removed from the repo (the pin only mattered against the
RPi OS repo; bookworm ships the right FFmpeg 5.x natively) - verdict
preserved in the docs and the legacy installer note updated
- docs: updated radio section (volumes, secret provisioning, wire-up)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dockerises the radio_conjurer.liq environment on Debian bookworm:
- Dockerfile.radio builds Liquidsoap through opam with build args for
OPAM_VERSION (static binary), OCAML_VERSION (default 4.14.2; 2.1.x needs
OCaml 4.x - prod ran 4.13.0, pass it for parity) and
LIQUIDSOAP_VERSION (default 2.1.4, matching the script)
- LIQ_OPAM_PACKAGES installs exactly the features the script uses:
mad+lame (mp3), cry (icecast), taglib (tags/replaygain), pulseaudio
(mic/out), samplerate, inotify (watch-reload), ffmpeg
- ffmpeg.pref verdict: the Pi pin forced Debian's FFmpeg 5.x family
(libavcodec59...) over RPi OS repo builds because the ocaml-ffmpeg
bindings are compiled against those sonames; bookworm ships them
natively so the container needs no pin (documented, not copied)
- pulse-access/audio group memberships baked in (were manual on the Pi);
PULSE_MODE=internal runs a system-wide pulse with a null sink so the
unmodified script works on a headless VM (mic = silence), host mode
mounts the real socket, none for edited scripts
- mounts mirror the script's hardcoded paths (/home/pi/Conjurer,
/home/pi/MediaFolder/mp3) so radio_conjurer.liq runs unmodified;
entrypoint seeds script+params on first run (never overwrites), creates
missing playlists, seeds placeholder icecast credentials (loud warning)
and generates a silent emergency mp3 when the single() file is missing
- compose.radio.yaml: extra_hosts maps the script's host="radio" to
ICECAST_HOST_IP; exposes 54321 (bot's RADIO_HARBOR /skip) and 9999
- docs: radio section with build args, pulse modes and the ffmpeg.pref /
groups verdicts
Part 2 of this PR (splitting bot<->radio comms out of the musician into a
radio+betoniarka container) follows on this branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cog did its own netrc read from a hardcoded /home/pi/.netrc at import
time, crashing on any other host. Now:
- constants.py: ASSEMBLYAI_API_KEY resolved like every other token
(env ASSEMBLYAI_API_KEY -> netrc machine 'assemblyai' at
CONJURER_NETRC_FILE); new TRANSCRIPTS_PATH (env
CONJURER_TRANSCRIPTS_PATH, defaults next to the log file, created by the
runtime layout)
- voice_recognition_commands.py: drop the hardcoded netrc read and
transcript dir; when the key is missing raise a clear RuntimeError so
the guarded loader disables ONLY this cog with a readable reason
- bot.env.example: document ASSEMBLYAI_API_KEY
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Container/native startup died silently on any missing file/service. Now:
constants.py - self-healing runtime layout at import:
- create missing dirs (log dir, LOGSTORE, GRAPHICS_PATH, MUSIC_FOLDER)
- seed missing state files from the repo templates shipped next to
constants.py (settings/system_gpt/pamiec/pamiec_muzyki/accident_log),
falling back to safe empty JSON; existing files are NEVER overwritten
bot.py:
- log to stdout too, so 'docker logs' finally shows the crash reason
- missing Discord token = loud sys.exit with mount/env instructions
(was: silent return -> container crash-loop with empty logs)
- every cog loads independently (one broken cog = skipped with traceback,
bot continues)
- musician/librarian cogs are health-gated: enabled only when the service
answers HTTP; a watchdog re-checks every 5 min and enables them the
moment the service comes alive (no restart needed); tree re-synced
- on_ready reconnects no longer re-load extensions
requirements_conan.txt + Dockerfile.bot: aiomcrcon (Python <=3.11 only)
moved to best-effort extras so the 3.13 image builds clean and the
conanjurer cog stays dormant without it.
DOCKER_PROXMOX.md: startup model (core vs gated cogs) + crash-loop
troubleshooting incl. the 'disappearing files' checklist (nothing in the
stack deletes host files; bind mount = live state).
Verified: fresh-volume seeding creates dirs+templates, existing files
untouched, missing-token exits with FATAL message, health-gating logic
(stub-based runpy tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- docker/entrypoint.musician.sh: on a fresh /data volume, create the
managed playlists and touch radio_log.log/persistence.log empty so the
track-forwarding thread waits instead of crashing when Liquidsoap runs
elsewhere; never overwrites preserved files
- Dockerfile.musician: wire the entrypoint, default CONJURER_MUSICIAN_BASE
=/data (persist playlists/logs) and CONJURER_STREAM_TEMPLATE=/app/stream.html;
declare the /data volume
- DOCKER_PROXMOX.md: expand the musician section (two volumes, preserving
existing playlists, radio-log coupling, stream template)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Containerises the three services (each intended for its own Proxmox VM)
and adds the code changes needed to run cleanly on Linux/Docker.
Code fixes:
- constants.py: CONJURER_DATA_DIR roots all writable bot state under one
mounted volume (per-variable overrides still win; native Pi unaffected)
- conjurer_librarian/search_bot.py + scrape_bot.py: the hardcoded Windows
DOI database path (C:\Database\chunks\) is now CONJURER_LIBRARIAN_DB_PATH,
with CONJURER_LIBRARIAN_MAXTHREADS / _CHUNK also env-overridable
Docker:
- docker/Dockerfile.{bot,librarian,musician} + compose.{bot,librarian,musician}.yaml
- docker/env/*.env.example (force-added; real *.env stays gitignored)
- docker/entrypoint.bot.sh seeds default JSON state into /data only when
absent, so preserved history is never overwritten
- .dockerignore
- docs/deployment/DOCKER_PROXMOX.md: step-by-step runbook incl. preserving
the existing command/conversation history and cross-VM auth
The bot image uses the vendored yt_dlp/spotify_dl forks (they win on
sys.path over the pip packages), dropping the old sed patching.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Michal Tuszowski
gitea