conjurer/docs/deployment/DOCKER_PROXMOX.md

# Conjurer on Docker / Proxmox

Runbook for running Conjurer as Docker containers across Proxmox VMs:

| Component | VM | Container | Port | Image |
|-----------|----|-----------|------|-------|
| **Main bot** | VM-bot | `conjurer-bot` | 5000 (Flask comm) | `docker/Dockerfile.bot` |
| **Librarian** | VM-librarian | `conjurer-librarian` | 5001 | `docker/Dockerfile.librarian` |
| **Musician** | VM-musician | `conjurer-musician` | 5000 (+ radio) | `docker/Dockerfile.musician` |
| **Share** (optional) | VM-musician or its own | `conjurer-share` | 8081 → 80 | `docker/Dockerfile.share` |

The share service publishes short-lived file links over Apache and is documented
separately: [SHARE_NODE_SETUP.md](SHARE_NODE_SETUP.md) to stand a node up from
zero, [FILE_SHARING.md](FILE_SHARING.md) for how the feature works. It shares two
volumes with the musician, so set it up after the musician is running.

The three talk to each other over HTTP on the Proxmox LAN. Direction of calls:

```
   bot  --(/mp3,/get_music,/add_to_priority,...)-->  musician
   bot  --(/query)-------------------------------->  librarian
musician --(/prepped_tracks)-------------------->    bot
librarian --(/conjurer results)----------------->    bot
  *    --(/ai_query)----------------------------->    bot   (see 1c-ter)
```

Everything is configured through `CONJURER_*` environment variables (see the
`docker/env/*.env.example` files). Nothing is hardcoded to a host path anymore.

---

## 0. Prerequisites (per VM)

Create a small Linux VM in Proxmox (Debian 12 / Ubuntu 22.04+ is fine), then
install Docker:

```bash
sudo apt-get update && sudo apt-get install -y ca-certificates curl git
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker "$USER"   # log out/in afterwards
```

Clone the repo on each VM (they build from it):

```bash
sudo mkdir -p /opt && cd /opt
git clone https://github.com/migatu/conjurer.git
cd conjurer
```

> All `docker compose` commands below are run **from the repo root** (`/opt/conjurer`),
> because the compose files use `context: ..` relative to `docker/`.

---

## 1. Main bot (VM-bot)

### 1a. Prepare host directories

```bash
sudo mkdir -p /srv/conjurer/data /srv/conjurer/secrets
```

### 1b. Preserve existing command history  ⭐

The bot's conversation memory and settings live in JSON files. Copy them from
your current deployment (e.g. the Pi's `/home/pi/Conjurer/`) into the data
volume so the history carries over:

```bash
# run on the Pi, or scp the files across, then place them here:
sudo cp pamiec.json            /srv/conjurer/data/   # AI conversation history
sudo cp pamiec_muzyki.json     /srv/conjurer/data/   # music-DJ memory
sudo cp settings.json          /srv/conjurer/data/   # word/cyclic reactions
sudo cp system_gpt_settings.json /srv/conjurer/data/
sudo cp accident_log.json      /srv/conjurer/data/   # if present
```

If you skip this, the container starts with the (empty) template files baked
into the image and history begins fresh.

### 1c. Tokens

Drop your existing netrc (the one with `discord`, `openai`, `spotipy`,
`youtube` entries) into the secrets dir:

```bash
sudo cp ~/.netrc /srv/conjurer/secrets/.netrc
sudo chmod 600 /srv/conjurer/secrets/.netrc
```

(Alternatively skip netrc and set `DISCORD_TOKEN` / `OPENAI_API_KEY` in the env
file — those take precedence.)

For the **Claude backend** add an `anthropic` machine to the same netrc (or set
`ANTHROPIC_API_KEY` in `bot.env`):

```
machine anthropic
    password sk-ant-...
```

You only need this if you actually switch the bot to Claude — see 1c-bis.

### 1c-bis. AI backend switch (GPT ↔ Claude)

The bot's AI chat runs on one backend at a time, chosen by a single switch:

- **At startup**, set `CONJURER_AI_CONFIG` in `bot.env` — `gpt` (default) or
  `claude`. Leave it unset to use the `"active"` key in
  `system_gpt_settings.json` (falls back to `gpt`).
- **At runtime**, `$gadaj_teraz <config>` (Vykidailo only) flips the backend live
  and writes the choice back into `system_gpt_settings.json`.

The provider configs (which backend, which models) are collected in
`system_gpt_settings.json` under the third list element — add further AIs there.
`claude` needs `ANTHROPIC_API_KEY`; `gpt` needs `OPENAI_API_KEY`. Image
generation (`imaginuje sobie:`) and personal assistants stay on OpenAI whatever
the switch says (Anthropic has no equivalent) and degrade quietly if OpenAI is
not configured, so a Claude-only box still boots.

### 1c-ter. AI query interface + librarian AI review

The bot exposes a queued AI interface on its comm layer: `POST /ai_query` with
`{"prompt": ..., "channel_id": <discord channel id>, "request_type"?: "NONE"}`
(same `X-Conjurer-Api-Key` auth as the other endpoints). The prompt is queued and
answered asynchronously by whatever backend `$gadaj_teraz` currently selects
(GPT or Claude), and the answer is posted to `channel_id`. `request_type: "NONE"`
(the default) keeps it a clean one-shot that doesn't touch the bar's
conversation memory.

The first consumer of this is the librarian command **`$wyszukaj_z_recenzja`**:
it works like `$wyszukaj_linki_do_dokumentow`, but when the DOI hits come back
the list (already sorted by Crossref relevance) plus the search phrase are handed
to the AI for a weighted-relevance re-rank and a short source review, delivered
to the same channel right after the raw results. No extra config — it uses the
active AI backend.

### 1d. Configure and launch

```bash
cp docker/env/bot.env.example docker/env/bot.env
# edit docker/env/bot.env: set CONJURER_FILE_SERVICE / _LIBRARIAN_SERVICE to the
# other VMs' IPs, and CONJURER_API_KEY (same value on all three) if you want auth.

docker compose -f docker/compose.bot.yaml up -d --build
docker logs -f conjurer-bot
```

Look for `Extension loaded: …` for each cog and `All systems: operational`.

### 1e. Startup model: core cogs vs service-gated cogs

The bot **always** starts with the cogs that depend on nothing but itself
(administration, AI, other, latex, voice, conanjurer). Cogs that need a
sibling service are **health-gated**:

| Group | Cogs | Enabled when |
|-------|------|--------------|
| musician | `music_commands`, `radio_commands`, `file_search_commands` | `GET {FILE_SERVICE}/mp3` answers |
| librarian | `librarian_commands` | librarian answers HTTP at all |

When a service is down its cogs stay disabled (commands simply don't exist)
and the log says so. A watchdog re-checks every 5 minutes and enables the
cogs the moment the service starts answering — no bot restart needed.
A single broken cog (missing pip package, bad import) is skipped with a full
traceback in the log; it never takes the whole bot down.

### 1f. Troubleshooting a crash-looping container

`docker logs conjurer-bot` now shows the real reason (the bot logs to stdout
as well as the rotating file). The most common cases:

- **`FATAL: Discord token missing`** — the secrets mount is missing/empty or
  `CONJURER_NETRC_FILE` points elsewhere. Check:
  `docker inspect -f '{{json .Mounts}}' conjurer-bot | jq` and
  `docker exec conjurer-bot ls -la /secrets/` (after a manual
  `docker run … sleep infinity` if it crash-loops too fast).
- **Missing state files** — not fatal anymore: missing dirs are created and
  missing JSON state is seeded from the repo templates baked into the image
  (existing files are never overwritten). Fix the mount at your leisure.

**About files "disappearing" from `/srv/conjurer/...`:** nothing in this stack
deletes host files — the entrypoint and the bot only ever *create* missing
files. With a bind mount, `/srv/conjurer/data` **is** the live state (not an
installation staging area): don't delete it after a successful install.
If files vanished, the usual suspects are `docker compose down -v` (only
affects *named* volumes, not binds), a re-provisioned VM, or copying the files
to a different path than the one in the compose `volumes:` line — verify with
`docker inspect -f '{{json .Mounts}}' conjurer-bot`.

---

## 2. Librarian (VM-librarian)

### 2a. Mount the DOI database

The librarian checks keyword hits from Crossref against a local database of
DOI chunk files (`0_chunk.txt … N_chunk.txt`). Put that database on the VM and
point the volume at it:

```bash
sudo mkdir -p /srv/librarian/doi /srv/librarian/secrets
# copy/replicate your chunk files into /srv/librarian/doi/
```

> The old Windows path `C:\Database\chunks\` is now `CONJURER_LIBRARIAN_DB_PATH`
> (defaults to `/doi/` in the container); `CONJURER_LIBRARIAN_CHUNK`
> (`_chunk.txt`) sets the suffix. Chunk files (`0_chunk.txt … N_chunk.txt`) are
> auto-discovered and all searched, so just drop them in — put as many as you
> like. (`CONJURER_LIBRARIAN_MAXTHREADS` is deprecated and ignored: it used to
> have to equal the file count exactly or the search would skip files or hang.)

### 2b. Configure and launch

```bash
cp docker/env/librarian.env.example docker/env/librarian.env
# edit: CONJURER_MAIN_BOT=http://BOT_VM_IP:5000, CONJURER_CROSSREF_MAILTO, CONJURER_API_KEY

docker compose -f docker/compose.librarian.yaml up -d --build
docker logs -f conjurer-librarian
```

---

## 3. Musician (VM-musician)

The web service is container-ready (`docker/Dockerfile.musician` +
`compose.musician.yaml`). It containerises the **Flask file/playlist service
only** — the Liquidsoap radio (`radio_conjurer.liq`), `script.params` and any
Samba/NFS share tooling are separate and typically stay on the host or a
dedicated setup (you'll adapt those yourself).

### 3a. Two volumes: the library and the writable state

| Mount | Container path | Holds |
|-------|---------------|-------|
| `/srv/musician/music` | `/music` (`CONJURER_MUSIC_FOLDER`) | your mp3 library (indexed/served) |
| `/srv/musician/data` | `/data` (`CONJURER_MUSICIAN_BASE`) | playlists, logs, `radio_log.log`/`persistence.log` |

```bash
sudo mkdir -p /srv/musician/music /srv/musician/data
# point /srv/musician/music at (or copy in) your mp3s
```

### 3b. Preserve existing playlists/state (optional)

If you already run the musician, copy its working playlists into the data
volume so nothing is regenerated from scratch:

```bash
sudo cp all_playlist.playlist hit.playlist request.playlist \
        priority_queue.playlist playlist.json  /srv/musician/data/ 2>/dev/null || true
```

The container's entrypoint (`docker/entrypoint.musician.sh`) creates any
missing playlists and touches `radio_log.log` / `persistence.log` empty so the
track-forwarding thread waits instead of crashing when the radio runs
elsewhere. It never overwrites files you copied in.

### 3c. Configure and launch

```bash
cp docker/env/musician.env.example docker/env/musician.env
# edit: CONJURER_MAIN_BOT=http://BOT_VM_IP:5000 and CONJURER_API_KEY (match the bot)

docker compose -f docker/compose.musician.yaml up -d --build
docker logs -f conjurer-musician
```

### 3d. Radio coupling (if you keep Liquidsoap separate)

The musician only forwards "now playing" to the bot by tailing the radio's
`radio_log.log` / `persistence.log`. To wire them up, have Liquidsoap write
those two files into the same `/srv/musician/data` directory (or set
`CONJURER_RADIO_LOG` / `CONJURER_PERSISTENCE_LOG` to wherever it writes). The
`/stream` page template is served from the baked-in `/app/stream.html`
(override with `CONJURER_STREAM_TEMPLATE` if you customise it).

### 3e. Radio (Liquidsoap) container

`docker/Dockerfile.radio` builds the full Liquidsoap environment through
**opam**, with the OCaml/opam/Liquidsoap versions selectable at build time:

| Build arg | Default | Notes |
|-----------|---------|-------|
| `LIQUIDSOAP_VERSION` | `2.1.4` | what `radio_conjurer.liq` targets |
| `OCAML_VERSION` | `4.14.2` | 2.1.x needs OCaml **4.x** (never 5); prod ran 4.13.0 — pass it for exact parity |
| `OPAM_VERSION` | `2.1.5` | static binary from GitHub releases |
| `LIQ_OPAM_PACKAGES` | `mad lame cry taglib pulseaudio samplerate inotify ffmpeg` | exactly the features the script uses (mp3 in/out, icecast, tags/replaygain, pulse mic/out, watch-reload) |

The container runs **both Liquidsoap and Icecast** — the stream is served
from this one container (`output.icecast(host="localhost", …)` in the
script). Volume layout (same paths inside and outside):

| Host & container path | Holds |
|---|---|
| `/srv/betoniarka/data` | playlists, `script.params`, `persistence.log`, `radio_log.log`, the seeded `radio_conjurer.liq` |
| `/srv/betoniarka/music` | the mp3 library |
| `/srv/betoniarka/secrets` | `icecast_credentials.json` — **provision at install**, like the other services' secrets |

```bash
sudo mkdir -p /srv/betoniarka/data /srv/betoniarka/music /srv/betoniarka/secrets
# provision the icecast secret (same pattern as the bot's netrc):
echo '{ "password" : "SOURCE_PW", "admin_password" : "ADMIN_PW" }' \
  | sudo tee /srv/betoniarka/secrets/icecast_credentials.json
sudo chmod 600 /srv/betoniarka/secrets/icecast_credentials.json

docker compose -f docker/compose.radio.yaml up -d --build
docker logs -f conjurer-radio
```

At startup the entrypoint renders `/etc/icecast2/icecast.xml` from
`docker/icecast.xml.tpl`, filling the source/admin/relay passwords from the
secret (`admin_password`/`relay_password` are optional and default to
`password`), and starts icecast as its unprivileged user. If the secret is
missing, a `CHANGE_ME` placeholder is seeded with a loud warning — the stack
boots but the stream stays locked until you fix it.

The script + `script.params` are seeded into the data volume on first run
and never overwritten (live edits survive rebuilds). Missing playlists are
created empty and a silent emergency-fallback mp3 is generated if the
`single()` file is absent, so the script always boots.

Wire-up: listeners tune to `http://RADIO_VM_IP:8000/mp3-stream`; the bot
points at `CONJURER_RADIO_HARBOR=http://RADIO_VM_IP:54321` (harbor `/skip`
lives in the script itself) and `CONJURER_RADIO_SERVICE=http://RADIO_VM_IP:5005`
(the betoniarka API below).

### 3f. Betoniarka - the radio operator (and why the split)

**Permissions post-mortem.** The pre-split layout had three actors fighting
over the same files: the musician wrote radio playlists **as root**, the
radio expected them **as user `radio`**, and both met on a **root-owned
network share** where `chown` fails by design (root squash / uid mapping).
Every component worked; the combination could not.

**The fix is structural**: the process that *writes* the radio playlists now
lives in the same container as the process that *watches* them, running as
the same `radio` user on a **local** volume. No network share, no chown, no
uid mapping - the class of problem is gone, not patched.

`conjurer_betoniarka/betoniarka.py` runs inside the radio container
(started by the entrypoint as user `radio`, port **5005**) and owns:
- the library scan → `all_playlist.playlist` / `hit.playlist` (local paths,
  the same ones Liquidsoap resolves), on start + every 24h + `GET /rescan`
- the bot-facing radio API: `/add_to_priority`, `/create_priority_playlist`,
  `/request_radio_file`, `/clear_pr_pls` (+ `GET /ping` for health gating,
  `/stream` for the web page)
- tailing `radio_log.log`/`persistence.log` and forwarding play events to
  the bot's `/prepped_tracks` (with the shared API key)

The **musician** is now a pure Discord music player: `/mp3`, `/update_mp3`,
`/get_music` and the file-share endpoints. It no longer writes any radio
files and needs no shared partition with the radio VM. The music library
can still be replicated/mounted on both VMs (read-only on the radio side is
fine) - playlists reference the *radio VM's local* paths, generated locally.

Bot wiring after the split (env on the bot):
```
CONJURER_FILE_SERVICE=http://MUSICIAN_VM_IP:5000   # Discord music
CONJURER_RADIO_SERVICE=http://RADIO_VM_IP:5005     # betoniarka (radio cmds)
CONJURER_RADIO_HARBOR=http://RADIO_VM_IP:54321     # liquidsoap /skip
```
`CONJURER_RADIO_SERVICE` defaults to `CONJURER_FILE_SERVICE`, so an
un-split deployment keeps working unchanged. The bot health-gates
`radio_commands` on betoniarka's `/ping` (separate from the musician group,
which now covers only `music_commands` + `file_search_commands`).

**Privileges:** Liquidsoap refuses to run as root (`init: security exit`),
so the main process runs as the dedicated **`radio`** user (member of
`audio`/`pulse-access`) — no `settings.init.allow_root` override. The
entrypoint (root) seeds volumes, renders the icecast config, starts
icecast/pulse, chowns `/srv/betoniarka/data` to `radio` and drops
privileges via `setpriv`. This is also why the opam switch lives in
`/opt/opam` instead of `/root/.opam` (the binary and the liquidsoap stdlib
must be readable by `radio`).

**PulseAudio** (`PULSE_MODE` env):
- `internal` (default) — a system-wide pulse daemon runs inside the container
  with a **null sink** (`docker/pulse-system.pa`): no sound hardware needed,
  `output.pulseaudio()` plays into the void and the `input.pulseaudio()` mic
  path reads silence (which `blank.strip` already gates out). Right choice
  for a headless Proxmox VM. Started with `--disallow-module-loading`
  (startup modules from `system.pa` still load; only later client-requested
  loads are blocked). The `forcibly disabling SHM mode` notice is inherent
  to system mode and harmless.
- `host` — mount the host's pulse socket and set `PULSE_SERVER`, for a VM
  with real audio hardware (the Pi's Lexicon Lambda setup).
- `none` — you removed the pulse in/out from the script.

**Verdicts on the old Pi setup quirks** (asked during containerisation):
- `ffmpeg.pref` (Pin-Priority 1001 on the `libavcodec59/libavformat59/…`
  family): it **did have a purpose** — the opam-built ocaml-ffmpeg bindings
  for 2.1.x are compiled against Debian bookworm's FFmpeg 5.x sonames, and
  the pin forced those over conflicting Raspberry Pi OS repo builds (even as
  a downgrade). In this single-repo container the same versions come
  naturally, so the pin is redundant — **the file has been removed from the
  repo** (this note preserves the knowledge).
- adding root to `pulse-access`/`audio` groups: needed on the Pi for the
  system-wide pulse socket and ALSA device access. The image bakes the same
  memberships in (`usermod -aG audio,pulse-access root`) — harmless with the
  internal null sink, required for `PULSE_MODE=host`.

---

## 4. Networking & auth

- Open the ports between VMs on the Proxmox LAN: **bot 5000**, **librarian 5001**,
  **musician 5000** (+ radio harbor 54321 if used). A simple `ufw allow from
  <lan-subnet>` per port is enough; do not expose them to the internet.
- **Auth:** set the same `CONJURER_API_KEY` in all three `*.env` files. Then
  every internal call carries `X-Conjurer-Api-Key` and each service rejects
  requests without it (HTTP 401). Leave it empty everywhere to disable auth
  (fully backward compatible). ⚠️ Setting it on only one side breaks the link.
- The addresses point at each other by VM IP (or a DNS name). Set:
  - bot: `CONJURER_FILE_SERVICE`, `CONJURER_RADIO_HARBOR`, `CONJURER_LIBRARIAN_SERVICE`
  - librarian & musician: `CONJURER_MAIN_BOT`

---

## 5. Verify

```bash
# bot is up and reachable from another VM:
curl http://BOT_VM_IP:5000/conjurer            # -> "ALIVE"
# librarian answers:
curl http://LIBRARIAN_VM_IP:5001/ -I           # service reachable
docker ps            # all containers "Up"
docker logs conjurer-bot --tail 50
```

In Discord, exercise a command that round-trips through a service (e.g. a music
search that hits the musician, or a librarian query) to confirm the wiring and
the API key.

---

## 6. Updates

```bash
cd /opt/conjurer && git pull
docker compose -f docker/compose.bot.yaml up -d --build   # rebuild + restart
```

Data in `/srv/.../data` and `/doi` / `/music` volumes survives rebuilds, so
history and databases persist across updates.

---

## 7. Rollback / coexistence

- The native (Raspberry Pi / systemd) deployment is unaffected — none of the
  defaults changed; the container behaviour is opt-in via `CONJURER_DATA_DIR`
  and the other env vars. You can run both during migration.
- To roll back a VM: `docker compose -f docker/compose.<svc>.yaml down` and
  restart the previous deployment. The JSON state in `/srv/.../data` is plain
  files you can copy back to the Pi if needed.

---

## Notes on the images

- **Bot** uses the vendored `yt_dlp/` and `spotify_dl/` forks (they win over the
  pip packages because `/app` is first on `sys.path`), so your patches stay
  active without the old `sed` hacks from `install_main_bot.sh`.
- **Tectonic** (LaTeX `$latex` command) is installed best-effort; if the build
  step fails the bot still runs, just without LaTeX. Remove that layer from
  `Dockerfile.bot` if you don't need it.
- Voice needs `ffmpeg` + `libopus0` (both in the image). No microphone/pyaudio
  is required — voice is received over Discord and transcribed via AssemblyAI.