conjurer/docs/deployment/DOCKER_PROXMOX.md

# Conjurer on Docker / Proxmox

Runbook for running Conjurer as Docker containers across Proxmox VMs:

| Component | VM | Container | Port | Image |
|-----------|----|-----------|------|-------|
| **Main bot** | VM-bot | `conjurer-bot` | 5000 (Flask comm) | `docker/Dockerfile.bot` |
| **Librarian** | VM-librarian | `conjurer-librarian` | 5001 | `docker/Dockerfile.librarian` |
| **Musician** | VM-musician | `conjurer-musician` | 5000 (+ radio) | `docker/Dockerfile.musician` |

The three talk to each other over HTTP on the Proxmox LAN. Direction of calls:

```
   bot  --(/mp3,/get_music,/add_to_priority,...)-->  musician
   bot  --(/query)-------------------------------->  librarian
musician --(/prepped_tracks)-------------------->    bot
librarian --(/conjurer results)----------------->    bot
```

Everything is configured through `CONJURER_*` environment variables (see the
`docker/env/*.env.example` files). Nothing is hardcoded to a host path anymore.

---

## 0. Prerequisites (per VM)

Create a small Linux VM in Proxmox (Debian 12 / Ubuntu 22.04+ is fine), then
install Docker:

```bash
sudo apt-get update && sudo apt-get install -y ca-certificates curl git
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker "$USER"   # log out/in afterwards
```

Clone the repo on each VM (they build from it):

```bash
sudo mkdir -p /opt && cd /opt
git clone https://github.com/migatu/conjurer.git
cd conjurer
```

> All `docker compose` commands below are run **from the repo root** (`/opt/conjurer`),
> because the compose files use `context: ..` relative to `docker/`.

---

## 1. Main bot (VM-bot)

### 1a. Prepare host directories

```bash
sudo mkdir -p /srv/conjurer/data /srv/conjurer/secrets
```

### 1b. Preserve existing command history  ⭐

The bot's conversation memory and settings live in JSON files. Copy them from
your current deployment (e.g. the Pi's `/home/pi/Conjurer/`) into the data
volume so the history carries over:

```bash
# run on the Pi, or scp the files across, then place them here:
sudo cp pamiec.json            /srv/conjurer/data/   # AI conversation history
sudo cp pamiec_muzyki.json     /srv/conjurer/data/   # music-DJ memory
sudo cp settings.json          /srv/conjurer/data/   # word/cyclic reactions
sudo cp system_gpt_settings.json /srv/conjurer/data/
sudo cp accident_log.json      /srv/conjurer/data/   # if present
```

If you skip this, the container starts with the (empty) template files baked
into the image and history begins fresh.

### 1c. Tokens

Drop your existing netrc (the one with `discord`, `openai`, `spotipy`,
`youtube` entries) into the secrets dir:

```bash
sudo cp ~/.netrc /srv/conjurer/secrets/.netrc
sudo chmod 600 /srv/conjurer/secrets/.netrc
```

(Alternatively skip netrc and set `DISCORD_TOKEN` / `OPENAI_API_KEY` in the env
file — those take precedence.)

### 1d. Configure and launch

```bash
cp docker/env/bot.env.example docker/env/bot.env
# edit docker/env/bot.env: set CONJURER_FILE_SERVICE / _LIBRARIAN_SERVICE to the
# other VMs' IPs, and CONJURER_API_KEY (same value on all three) if you want auth.

docker compose -f docker/compose.bot.yaml up -d --build
docker logs -f conjurer-bot
```

Look for `Extension loaded: …` for each cog and `All systems: operational`.

### 1e. Startup model: core cogs vs service-gated cogs

The bot **always** starts with the cogs that depend on nothing but itself
(administration, AI, other, latex, voice, conanjurer). Cogs that need a
sibling service are **health-gated**:

| Group | Cogs | Enabled when |
|-------|------|--------------|
| musician | `music_commands`, `radio_commands`, `file_search_commands` | `GET {FILE_SERVICE}/mp3` answers |
| librarian | `librarian_commands` | librarian answers HTTP at all |

When a service is down its cogs stay disabled (commands simply don't exist)
and the log says so. A watchdog re-checks every 5 minutes and enables the
cogs the moment the service starts answering — no bot restart needed.
A single broken cog (missing pip package, bad import) is skipped with a full
traceback in the log; it never takes the whole bot down.

### 1f. Troubleshooting a crash-looping container

`docker logs conjurer-bot` now shows the real reason (the bot logs to stdout
as well as the rotating file). The most common cases:

- **`FATAL: Discord token missing`** — the secrets mount is missing/empty or
  `CONJURER_NETRC_FILE` points elsewhere. Check:
  `docker inspect -f '{{json .Mounts}}' conjurer-bot | jq` and
  `docker exec conjurer-bot ls -la /secrets/` (after a manual
  `docker run … sleep infinity` if it crash-loops too fast).
- **Missing state files** — not fatal anymore: missing dirs are created and
  missing JSON state is seeded from the repo templates baked into the image
  (existing files are never overwritten). Fix the mount at your leisure.

**About files "disappearing" from `/srv/conjurer/...`:** nothing in this stack
deletes host files — the entrypoint and the bot only ever *create* missing
files. With a bind mount, `/srv/conjurer/data` **is** the live state (not an
installation staging area): don't delete it after a successful install.
If files vanished, the usual suspects are `docker compose down -v` (only
affects *named* volumes, not binds), a re-provisioned VM, or copying the files
to a different path than the one in the compose `volumes:` line — verify with
`docker inspect -f '{{json .Mounts}}' conjurer-bot`.

---

## 2. Librarian (VM-librarian)

### 2a. Mount the DOI database

The librarian checks keyword hits from Crossref against a local database of
DOI chunk files (`0_chunk.txt … N_chunk.txt`). Put that database on the VM and
point the volume at it:

```bash
sudo mkdir -p /srv/librarian/doi /srv/librarian/secrets
# copy/replicate your chunk files into /srv/librarian/doi/
```

> The old Windows path `C:\Database\chunks\` is now `CONJURER_LIBRARIAN_DB_PATH`
> (defaults to `/doi/` in the container). `CONJURER_LIBRARIAN_MAXTHREADS` (41)
> and `CONJURER_LIBRARIAN_CHUNK` (`_chunk.txt`) are configurable too.

### 2b. Configure and launch

```bash
cp docker/env/librarian.env.example docker/env/librarian.env
# edit: CONJURER_MAIN_BOT=http://BOT_VM_IP:5000, CONJURER_CROSSREF_MAILTO, CONJURER_API_KEY

docker compose -f docker/compose.librarian.yaml up -d --build
docker logs -f conjurer-librarian
```

---

## 3. Musician (VM-musician)

The web service is container-ready (`docker/Dockerfile.musician` +
`compose.musician.yaml`). It containerises the **Flask file/playlist service
only** — the Liquidsoap radio (`radio_conjurer.liq`), `script.params` and any
Samba/NFS share tooling are separate and typically stay on the host or a
dedicated setup (you'll adapt those yourself).

### 3a. Two volumes: the library and the writable state

| Mount | Container path | Holds |
|-------|---------------|-------|
| `/srv/musician/music` | `/music` (`CONJURER_MUSIC_FOLDER`) | your mp3 library (indexed/served) |
| `/srv/musician/data` | `/data` (`CONJURER_MUSICIAN_BASE`) | playlists, logs, `radio_log.log`/`persistence.log` |

```bash
sudo mkdir -p /srv/musician/music /srv/musician/data
# point /srv/musician/music at (or copy in) your mp3s
```

### 3b. Preserve existing playlists/state (optional)

If you already run the musician, copy its working playlists into the data
volume so nothing is regenerated from scratch:

```bash
sudo cp all_playlist.playlist hit.playlist request.playlist \
        priority_queue.playlist playlist.json  /srv/musician/data/ 2>/dev/null || true
```

The container's entrypoint (`docker/entrypoint.musician.sh`) creates any
missing playlists and touches `radio_log.log` / `persistence.log` empty so the
track-forwarding thread waits instead of crashing when the radio runs
elsewhere. It never overwrites files you copied in.

### 3c. Configure and launch

```bash
cp docker/env/musician.env.example docker/env/musician.env
# edit: CONJURER_MAIN_BOT=http://BOT_VM_IP:5000 and CONJURER_API_KEY (match the bot)

docker compose -f docker/compose.musician.yaml up -d --build
docker logs -f conjurer-musician
```

### 3d. Radio coupling (if you keep Liquidsoap separate)

The musician only forwards "now playing" to the bot by tailing the radio's
`radio_log.log` / `persistence.log`. To wire them up, have Liquidsoap write
those two files into the same `/srv/musician/data` directory (or set
`CONJURER_RADIO_LOG` / `CONJURER_PERSISTENCE_LOG` to wherever it writes). The
`/stream` page template is served from the baked-in `/app/stream.html`
(override with `CONJURER_STREAM_TEMPLATE` if you customise it).

### 3e. Radio (Liquidsoap) container

`docker/Dockerfile.radio` builds the full Liquidsoap environment through
**opam**, with the OCaml/opam/Liquidsoap versions selectable at build time:

| Build arg | Default | Notes |
|-----------|---------|-------|
| `LIQUIDSOAP_VERSION` | `2.1.4` | what `radio_conjurer.liq` targets |
| `OCAML_VERSION` | `4.14.2` | 2.1.x needs OCaml **4.x** (never 5); prod ran 4.13.0 — pass it for exact parity |
| `OPAM_VERSION` | `2.1.5` | static binary from GitHub releases |
| `LIQ_OPAM_PACKAGES` | `mad lame cry taglib pulseaudio samplerate inotify ffmpeg` | exactly the features the script uses (mp3 in/out, icecast, tags/replaygain, pulse mic/out, watch-reload) |

The container runs **both Liquidsoap and Icecast** — the stream is served
from this one container (`output.icecast(host="localhost", …)` in the
script). Volume layout (same paths inside and outside):

| Host & container path | Holds |
|---|---|
| `/srv/betoniarka/data` | playlists, `script.params`, `persistence.log`, `radio_log.log`, the seeded `radio_conjurer.liq` |
| `/srv/betoniarka/music` | the mp3 library |
| `/srv/betoniarka/secrets` | `icecast_credentials.json` — **provision at install**, like the other services' secrets |

```bash
sudo mkdir -p /srv/betoniarka/data /srv/betoniarka/music /srv/betoniarka/secrets
# provision the icecast secret (same pattern as the bot's netrc):
echo '{ "password" : "SOURCE_PW", "admin_password" : "ADMIN_PW" }' \
  | sudo tee /srv/betoniarka/secrets/icecast_credentials.json
sudo chmod 600 /srv/betoniarka/secrets/icecast_credentials.json

docker compose -f docker/compose.radio.yaml up -d --build
docker logs -f conjurer-radio
```

At startup the entrypoint renders `/etc/icecast2/icecast.xml` from
`docker/icecast.xml.tpl`, filling the source/admin/relay passwords from the
secret (`admin_password`/`relay_password` are optional and default to
`password`), and starts icecast as its unprivileged user. If the secret is
missing, a `CHANGE_ME` placeholder is seeded with a loud warning — the stack
boots but the stream stays locked until you fix it.

The script + `script.params` are seeded into the data volume on first run
and never overwritten (live edits survive rebuilds). Missing playlists are
created empty and a silent emergency-fallback mp3 is generated if the
`single()` file is absent, so the script always boots.

Wire-up: listeners tune to `http://RADIO_VM_IP:8000/mp3-stream`; the bot
points at `CONJURER_RADIO_HARBOR=http://RADIO_VM_IP:54321` (harbor `/skip`
lives in the script itself) and `CONJURER_RADIO_SERVICE=http://RADIO_VM_IP:5005`
(the betoniarka API below).

### 3f. Betoniarka - the radio operator (and why the split)

**Permissions post-mortem.** The pre-split layout had three actors fighting
over the same files: the musician wrote radio playlists **as root**, the
radio expected them **as user `radio`**, and both met on a **root-owned
network share** where `chown` fails by design (root squash / uid mapping).
Every component worked; the combination could not.

**The fix is structural**: the process that *writes* the radio playlists now
lives in the same container as the process that *watches* them, running as
the same `radio` user on a **local** volume. No network share, no chown, no
uid mapping - the class of problem is gone, not patched.

`conjurer_betoniarka/betoniarka.py` runs inside the radio container
(started by the entrypoint as user `radio`, port **5005**) and owns:
- the library scan → `all_playlist.playlist` / `hit.playlist` (local paths,
  the same ones Liquidsoap resolves), on start + every 24h + `GET /rescan`
- the bot-facing radio API: `/add_to_priority`, `/create_priority_playlist`,
  `/request_radio_file`, `/clear_pr_pls` (+ `GET /ping` for health gating,
  `/stream` for the web page)
- tailing `radio_log.log`/`persistence.log` and forwarding play events to
  the bot's `/prepped_tracks` (with the shared API key)

The **musician** is now a pure Discord music player: `/mp3`, `/update_mp3`,
`/get_music` and the file-share endpoints. It no longer writes any radio
files and needs no shared partition with the radio VM. The music library
can still be replicated/mounted on both VMs (read-only on the radio side is
fine) - playlists reference the *radio VM's local* paths, generated locally.

Bot wiring after the split (env on the bot):
```
CONJURER_FILE_SERVICE=http://MUSICIAN_VM_IP:5000   # Discord music
CONJURER_RADIO_SERVICE=http://RADIO_VM_IP:5005     # betoniarka (radio cmds)
CONJURER_RADIO_HARBOR=http://RADIO_VM_IP:54321     # liquidsoap /skip
```
`CONJURER_RADIO_SERVICE` defaults to `CONJURER_FILE_SERVICE`, so an
un-split deployment keeps working unchanged. The bot health-gates
`radio_commands` on betoniarka's `/ping` (separate from the musician group,
which now covers only `music_commands` + `file_search_commands`).

**Privileges:** Liquidsoap refuses to run as root (`init: security exit`),
so the main process runs as the dedicated **`radio`** user (member of
`audio`/`pulse-access`) — no `settings.init.allow_root` override. The
entrypoint (root) seeds volumes, renders the icecast config, starts
icecast/pulse, chowns `/srv/betoniarka/data` to `radio` and drops
privileges via `setpriv`. This is also why the opam switch lives in
`/opt/opam` instead of `/root/.opam` (the binary and the liquidsoap stdlib
must be readable by `radio`).

**PulseAudio** (`PULSE_MODE` env):
- `internal` (default) — a system-wide pulse daemon runs inside the container
  with a **null sink** (`docker/pulse-system.pa`): no sound hardware needed,
  `output.pulseaudio()` plays into the void and the `input.pulseaudio()` mic
  path reads silence (which `blank.strip` already gates out). Right choice
  for a headless Proxmox VM. Started with `--disallow-module-loading`
  (startup modules from `system.pa` still load; only later client-requested
  loads are blocked). The `forcibly disabling SHM mode` notice is inherent
  to system mode and harmless.
- `host` — mount the host's pulse socket and set `PULSE_SERVER`, for a VM
  with real audio hardware (the Pi's Lexicon Lambda setup).
- `none` — you removed the pulse in/out from the script.

**Verdicts on the old Pi setup quirks** (asked during containerisation):
- `ffmpeg.pref` (Pin-Priority 1001 on the `libavcodec59/libavformat59/…`
  family): it **did have a purpose** — the opam-built ocaml-ffmpeg bindings
  for 2.1.x are compiled against Debian bookworm's FFmpeg 5.x sonames, and
  the pin forced those over conflicting Raspberry Pi OS repo builds (even as
  a downgrade). In this single-repo container the same versions come
  naturally, so the pin is redundant — **the file has been removed from the
  repo** (this note preserves the knowledge).
- adding root to `pulse-access`/`audio` groups: needed on the Pi for the
  system-wide pulse socket and ALSA device access. The image bakes the same
  memberships in (`usermod -aG audio,pulse-access root`) — harmless with the
  internal null sink, required for `PULSE_MODE=host`.

---

## 4. Networking & auth

- Open the ports between VMs on the Proxmox LAN: **bot 5000**, **librarian 5001**,
  **musician 5000** (+ radio harbor 54321 if used). A simple `ufw allow from
  <lan-subnet>` per port is enough; do not expose them to the internet.
- **Auth:** set the same `CONJURER_API_KEY` in all three `*.env` files. Then
  every internal call carries `X-Conjurer-Api-Key` and each service rejects
  requests without it (HTTP 401). Leave it empty everywhere to disable auth
  (fully backward compatible). ⚠️ Setting it on only one side breaks the link.
- The addresses point at each other by VM IP (or a DNS name). Set:
  - bot: `CONJURER_FILE_SERVICE`, `CONJURER_RADIO_HARBOR`, `CONJURER_LIBRARIAN_SERVICE`
  - librarian & musician: `CONJURER_MAIN_BOT`

---

## 5. Verify

```bash
# bot is up and reachable from another VM:
curl http://BOT_VM_IP:5000/conjurer            # -> "ALIVE"
# librarian answers:
curl http://LIBRARIAN_VM_IP:5001/ -I           # service reachable
docker ps            # all containers "Up"
docker logs conjurer-bot --tail 50
```

In Discord, exercise a command that round-trips through a service (e.g. a music
search that hits the musician, or a librarian query) to confirm the wiring and
the API key.

---

## 6. Updates

```bash
cd /opt/conjurer && git pull
docker compose -f docker/compose.bot.yaml up -d --build   # rebuild + restart
```

Data in `/srv/.../data` and `/doi` / `/music` volumes survives rebuilds, so
history and databases persist across updates.

---

## 7. Rollback / coexistence

- The native (Raspberry Pi / systemd) deployment is unaffected — none of the
  defaults changed; the container behaviour is opt-in via `CONJURER_DATA_DIR`
  and the other env vars. You can run both during migration.
- To roll back a VM: `docker compose -f docker/compose.<svc>.yaml down` and
  restart the previous deployment. The JSON state in `/srv/.../data` is plain
  files you can copy back to the Pi if needed.

---

## Notes on the images

- **Bot** uses the vendored `yt_dlp/` and `spotify_dl/` forks (they win over the
  pip packages because `/app` is first on `sys.path`), so your patches stay
  active without the old `sed` hacks from `install_main_bot.sh`.
- **Tectonic** (LaTeX `$latex` command) is installed best-effort; if the build
  step fails the bot still runs, just without LaTeX. Remove that layer from
  `Dockerfile.bot` if you don't need it.
- Voice needs `ffmpeg` + `libopus0` (both in the image). No microphone/pyaudio
  is required — voice is received over Discord and transcribed via AssemblyAI.