FILE_SHARING.md explains how the feature works and assumes you already know
the node story; there was nothing describing what a fresh box has to provide.
SHARE_NODE_SETUP.md fills that: the four host directories, the uid-33
readability requirement on the media library, deploying the musician+share
stack (Portainer or CLI), forwarding /share from the reverse proxy, and the
musician-side CONJURER_SHARE_* wiring - each with its verification command.
Calls out the three couplings that actually break it in practice: the media
must be mounted at the SAME container path on both containers (index paths and
symlinks are absolute), the link/index dirs must be shared, and
CONJURER_SHARE_BASE_URL must match the real public URL - it is not set in the
bundled stack file and silently falls back to czernobog.pl.
Also records what the node does NOT need: no host Apache, no cron, no Python,
no hand-placed scripts - Apache, the vhost, scan_shares.py and
revoke_shares.py are all baked into the image.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two connected features.
1) AI query interface (via the comm layer). communication_subroutine gains an
AI_QUERY_Q, a submit_ai_query() in-process entry point, and an authed
POST /ai_query endpoint ({prompt, channel_id, request_type?, username?}). The
prompt is queued and answered asynchronously by a new tasks.loop worker in the
always-loaded AI cog (Events), which calls handle_response - so it runs on
whichever backend $gadaj_teraz currently selects (GPT or Claude) - and posts the
answer to the requested channel, chunked to Discord's limit. request_type "NONE"
(default) is a clean one-shot: no persona system prompt, no memory write. The
worker starts before the OpenAI guard in cog_load, so it also runs on a
Claude-only box; cog_unload cancels it.
2) Librarian AI review. New command $wyszukaj_z_recenzja mirrors
$wyszukaj_linki_do_dokumentow but sets ai_review=True on the QueryControl, which
rides the round-trip and is matched back by UUID. When the hits return,
check_data_q sends the raw list as before, then - if flagged - hands the same
list (already in Crossref-relevance order) plus the search phrase to the AI
queue for a weighted re-rank and per-source review, delivered to the same
channel. QueryControl gains an ai_review flag (default False, so the orphan path
and all existing callers are unaffected).
Confirmed separately (and noted in the docs): the DOI list the AI receives is
pre-sorted by Crossref relevance - the librarian pipeline only filters (drops
title-less items) and splits (in-db / not-in-db), never re-sorts, and relies on
insertion-ordered dicts (Py 3.7+).
Verified: /ai_query auth (401/200/400/open), submit_ai_query and the queued
dict shape, and the QueryControl flag - via a Flask test client and
tests/integration/test_ai_query_endpoint.py (5 tests, all pass; integration
suite 11 passed, the 3 failures are the pre-existing /clear_pr_pls musician
tests fixed on a separate branch). Full first-party compile clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
search_bot conflated MAXTHREADS into two jobs at once - how many chunk files to
read (files 0..MAXTHREADS-1) AND how many producer sentinels to wait for - so
the two had to match exactly. Set too low it silently skipped trailing chunks;
set too high (or with any chunk missing/unreadable) a producer crashed before
emitting its sentinel, the consumers' count never reached the threshold, and
search_for_doi hung on join() forever. The idle-timeout failsafe that was meant
to break a starved consumer was dead code: `if empty_counter > 5: ... elif
empty_counter > 10: break` - >10 implies >5, so the elif never ran.
Fix, three layers:
* auto-discover the chunk files present (discover_chunk_files: <n>_chunk.txt in
numeric order) instead of range(0, MAXTHREADS). All files are read regardless
of count, and no producer is ever pointed at a missing file;
* the sentinel threshold is now the number of producers actually started, so it
can't drift from what's emitted;
* producers emit their sentinel in a finally, so even a crash (missing/unreadable
chunk) can't starve the count; and the idle backstop is reordered so it can
actually fire (>EMPTY_LIMIT seconds) as a last resort.
MAXTHREADS is deprecated and unused (kept only so old env files don't break);
docs/env updated to say chunk files are auto-discovered.
For the reported case (MAXTHREADS=40, files 0..43): before, files 40-43 were
silently never searched, and any run that referenced a missing chunk hung
forever. After, all 44 are searched and it always terminates.
Verified in a pytest-only venv (tests/unit/test_search_bot.py): DOI in a
trailing chunk is found; an unreadable chunk still terminates; empty dir returns
at once; discovery is numeric-sorted. Full unit job 27 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Python half of the short-lived share links lived in the repo; the Apache
config and the cron entries that make it work were hand-placed on the host, so
the feature could not be rebuilt from a checkout. This adds the missing half.
Scripts (defaults unchanged, so existing bare-metal cron keeps working):
* scan_shares.py / revoke_shares.py take their paths from CONJURER_SHARE_*
instead of hardcoding the Pi layout, and create their parent dirs;
* the revoke TTL is now CONJURER_SHARE_TTL_SECONDS. Its --help claimed "2min"
while the code used a hardcoded 3600 - the help text now reports the real,
configured value.
New share service:
* docker/Dockerfile.share - Apache + the two jobs, reusing the same scripts
rather than forking copies;
* docker/share-vhost.conf.tpl - the previously undocumented Apache half.
Three settings are load-bearing and commented as such: +FollowSymLinks (the
shares ARE symlinks), -Indexes (a listing would expose every live token), and
a deny rule for dotfiles (revoke_shares.py keeps .downloads.json - a map of
every live token - inside the served directory);
* docker/entrypoint.share.sh - renders the vhost, seeds the index on first run,
then runs scanner/revoker in sleep loops beside Apache (no cron, so their
output shows up in docker logs);
* compose + env example, incl. the two volumes that MUST be shared with the
musician (it creates the links and reads the index).
docs/deployment/FILE_SHARING.md documents the mechanism, both deployment routes
(docker and existing host Apache), the Discord command, why each Apache setting
matters, verification commands, and the known limitations - notably that a link
nobody ever downloads is never revoked, since the TTL starts at first download.
Verified: scanner and revoker exercised end-to-end against temp dirs (index
built; token recorded from a combined-format log line and the symlink unlinked).
Compose file not validated - no docker on this machine.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the two deployment guides to cover the new env vars and the runtime
switch: ANTHROPIC_API_KEY / netrc machine 'anthropic', CONJURER_AI_CONFIG
(gpt|claude), the $gadaj_teraz command, and the note that DALL-E + personal
assistants stay on OpenAI regardless of the switch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Permissions post-mortem that motivated this: the musician wrote radio
playlists AS ROOT onto a ROOT-OWNED network share which liquidsoap then
read AS USER 'radio' - chown fails on such shares by design (root squash /
uid mapping), so the radio came up and died on the playlists. The fix is
structural: the playlist WRITER now lives in the same container as the
READER, as the same user, on a local volume. No shared partition, no
chown, no uid mapping.
New: conjurer_betoniarka/betoniarka.py - runs inside the radio container
(started by the entrypoint as user 'radio', port 5005):
- library scan -> all_playlist/hit playlists with LOCAL container paths
(start + every 24h + authenticated GET /rescan)
- bot-facing radio API moved from the musician: /add_to_priority,
/create_priority_playlist, /request_radio_file, /clear_pr_pls,
plus GET /ping (health) and /stream (web page)
- radio_log/persistence tailer forwarding play events to the bot's
/prepped_tracks with the shared API key (bot-unreachable = logged, not fatal)
Musician: pure Discord music player now - keeps /mp3, /update_mp3,
/get_music and the file-share endpoints; all radio playlist writing, radio
paths/env and the tailer removed.
Bot: new CONJURER_RADIO_SERVICE (defaults to CONJURER_FILE_SERVICE so
un-split deployments keep working); radio_commands targets it; separate
'radio' health-gate group on betoniarka /ping (musician group now covers
music_commands + file_search_commands only).
Docker: betoniarka baked into the radio image (python3 + flask/waitress/
requests from Debian debs), port 5005 exposed, entrypoint starts it via
setpriv as 'radio'; data-volume chown is now best-effort with a loud
warning (keep the volume local); docs get the post-mortem + wiring.
Verified: py_compile everything; functional stub tests - rescan writes
local-path playlists, wyszukaj scores and appends to priority, auth
401/ok, tailer forward carries the API key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Liquidsoap refuses to start as root (init: security exit). Instead of
the settings.init.allow_root override, the image now has a dedicated
'radio' user (audio + pulse-access groups) and the entrypoint drops
privileges via setpriv after doing its root-only work (volume seeding,
icecast config render/start, pulse start, chown of the data volume).
The icecast secret is made group-readable (640 root:radio) because the
script parses it directly.
- The opam root moved from /root/.opam to /opt/opam so the liquidsoap
binary AND its stdlib .liq files are readable by the radio user.
NOTE: this invalidates the cached opam build layer - next build
recompiles liquidsoap (~15-20 min).
- pulseaudio (PULSE_MODE=internal) now starts with
--disallow-module-loading: system.pa startup modules still load, only
later client-requested loads are blocked, and the system-mode warning
goes away. The 'forcibly disabling SHM mode' notice is inherent to
system mode and harmless (documented).
Keeps the user's libcurl4-gnutls-dev build-dep fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review of the first radio-container cut:
- radio_conjurer.liq: replace the cargo-culted /home/pi/Conjurer and
/home/pi/MediaFolder paths with the container layout -
/srv/betoniarka/data (playlists, script.params, persistence/radio logs),
/srv/betoniarka/music (library + emergency track) and
/srv/betoniarka/secrets/icecast_credentials.json;
interactive.persistent now uses an absolute path;
output.icecast targets host="localhost" (icecast lives in the container)
- Icecast2 now runs INSIDE the radio container: entrypoint renders
/etc/icecast2/icecast.xml from docker/icecast.xml.tpl, filling
source/admin/relay passwords from the secret provisioned at install time
(same pattern as the bot's netrc); optional admin_password/relay_password
fields default to password; icecast starts as its unprivileged user;
port 8000 exposed for listeners; extra_hosts hack removed
- compose.radio.yaml: same-path mounts for /srv/betoniarka/{data,music,secrets}
- ffmpeg.pref removed from the repo (the pin only mattered against the
RPi OS repo; bookworm ships the right FFmpeg 5.x natively) - verdict
preserved in the docs and the legacy installer note updated
- docs: updated radio section (volumes, secret provisioning, wire-up)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dockerises the radio_conjurer.liq environment on Debian bookworm:
- Dockerfile.radio builds Liquidsoap through opam with build args for
OPAM_VERSION (static binary), OCAML_VERSION (default 4.14.2; 2.1.x needs
OCaml 4.x - prod ran 4.13.0, pass it for parity) and
LIQUIDSOAP_VERSION (default 2.1.4, matching the script)
- LIQ_OPAM_PACKAGES installs exactly the features the script uses:
mad+lame (mp3), cry (icecast), taglib (tags/replaygain), pulseaudio
(mic/out), samplerate, inotify (watch-reload), ffmpeg
- ffmpeg.pref verdict: the Pi pin forced Debian's FFmpeg 5.x family
(libavcodec59...) over RPi OS repo builds because the ocaml-ffmpeg
bindings are compiled against those sonames; bookworm ships them
natively so the container needs no pin (documented, not copied)
- pulse-access/audio group memberships baked in (were manual on the Pi);
PULSE_MODE=internal runs a system-wide pulse with a null sink so the
unmodified script works on a headless VM (mic = silence), host mode
mounts the real socket, none for edited scripts
- mounts mirror the script's hardcoded paths (/home/pi/Conjurer,
/home/pi/MediaFolder/mp3) so radio_conjurer.liq runs unmodified;
entrypoint seeds script+params on first run (never overwrites), creates
missing playlists, seeds placeholder icecast credentials (loud warning)
and generates a silent emergency mp3 when the single() file is missing
- compose.radio.yaml: extra_hosts maps the script's host="radio" to
ICECAST_HOST_IP; exposes 54321 (bot's RADIO_HARBOR /skip) and 9999
- docs: radio section with build args, pulse modes and the ffmpeg.pref /
groups verdicts
Part 2 of this PR (splitting bot<->radio comms out of the musician into a
radio+betoniarka container) follows on this branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Container/native startup died silently on any missing file/service. Now:
constants.py - self-healing runtime layout at import:
- create missing dirs (log dir, LOGSTORE, GRAPHICS_PATH, MUSIC_FOLDER)
- seed missing state files from the repo templates shipped next to
constants.py (settings/system_gpt/pamiec/pamiec_muzyki/accident_log),
falling back to safe empty JSON; existing files are NEVER overwritten
bot.py:
- log to stdout too, so 'docker logs' finally shows the crash reason
- missing Discord token = loud sys.exit with mount/env instructions
(was: silent return -> container crash-loop with empty logs)
- every cog loads independently (one broken cog = skipped with traceback,
bot continues)
- musician/librarian cogs are health-gated: enabled only when the service
answers HTTP; a watchdog re-checks every 5 min and enables them the
moment the service comes alive (no restart needed); tree re-synced
- on_ready reconnects no longer re-load extensions
requirements_conan.txt + Dockerfile.bot: aiomcrcon (Python <=3.11 only)
moved to best-effort extras so the 3.13 image builds clean and the
conanjurer cog stays dormant without it.
DOCKER_PROXMOX.md: startup model (core vs gated cogs) + crash-loop
troubleshooting incl. the 'disappearing files' checklist (nothing in the
stack deletes host files; bind mount = live state).
Verified: fresh-volume seeding creates dirs+templates, existing files
untouched, missing-token exits with FATAL message, health-gating logic
(stub-based runpy tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- docker/entrypoint.musician.sh: on a fresh /data volume, create the
managed playlists and touch radio_log.log/persistence.log empty so the
track-forwarding thread waits instead of crashing when Liquidsoap runs
elsewhere; never overwrites preserved files
- Dockerfile.musician: wire the entrypoint, default CONJURER_MUSICIAN_BASE
=/data (persist playlists/logs) and CONJURER_STREAM_TEMPLATE=/app/stream.html;
declare the /data volume
- DOCKER_PROXMOX.md: expand the musician section (two volumes, preserving
existing playlists, radio-log coupling, stream template)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Containerises the three services (each intended for its own Proxmox VM)
and adds the code changes needed to run cleanly on Linux/Docker.
Code fixes:
- constants.py: CONJURER_DATA_DIR roots all writable bot state under one
mounted volume (per-variable overrides still win; native Pi unaffected)
- conjurer_librarian/search_bot.py + scrape_bot.py: the hardcoded Windows
DOI database path (C:\Database\chunks\) is now CONJURER_LIBRARIAN_DB_PATH,
with CONJURER_LIBRARIAN_MAXTHREADS / _CHUNK also env-overridable
Docker:
- docker/Dockerfile.{bot,librarian,musician} + compose.{bot,librarian,musician}.yaml
- docker/env/*.env.example (force-added; real *.env stays gitignored)
- docker/entrypoint.bot.sh seeds default JSON state into /data only when
absent, so preserved history is never overwritten
- .dockerignore
- docs/deployment/DOCKER_PROXMOX.md: step-by-step runbook incl. preserving
the existing command/conversation history and cross-VM auth
The bot image uses the vendored yt_dlp/spotify_dl forks (they win on
sys.path over the pip packages), dropping the old sed patching.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gitea
Michal Tuszowski