conjurer/docs/deployment/DOCKER_PROXMOX.md

Conjurer on Docker / Proxmox

Runbook for running Conjurer as Docker containers across Proxmox VMs:

Component	VM	Container	Port	Image
Main bot	VM-bot	conjurer-bot	5000 (Flask comm)	docker/Dockerfile.bot
Librarian	VM-librarian	conjurer-librarian	5001	docker/Dockerfile.librarian
Musician	VM-musician	conjurer-musician	5000 (+ radio)	docker/Dockerfile.musician

The three talk to each other over HTTP on the Proxmox LAN. Direction of calls:

   bot  --(/mp3,/get_music,/add_to_priority,...)-->  musician
   bot  --(/query)-------------------------------->  librarian
musician --(/prepped_tracks)-------------------->    bot
librarian --(/conjurer results)----------------->    bot

Everything is configured through CONJURER_* environment variables (see the docker/env/*.env.example files). Nothing is hardcoded to a host path anymore.

---

0. Prerequisites (per VM)

Create a small Linux VM in Proxmox (Debian 12 / Ubuntu 22.04+ is fine), then install Docker:

sudo apt-get update && sudo apt-get install -y ca-certificates curl git
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker "$USER"   # log out/in afterwards

Clone the repo on each VM (they build from it):

sudo mkdir -p /opt && cd /opt
git clone https://github.com/migatu/conjurer.git
cd conjurer

All docker compose commands below are run from the repo root (/opt/conjurer), because the compose files use context: .. relative to docker/.

---

1. Main bot (VM-bot)

1a. Prepare host directories

sudo mkdir -p /srv/conjurer/data /srv/conjurer/secrets

1b. Preserve existing command history ⭐

The bot's conversation memory and settings live in JSON files. Copy them from your current deployment (e.g. the Pi's /home/pi/Conjurer/) into the data volume so the history carries over:

# run on the Pi, or scp the files across, then place them here:
sudo cp pamiec.json            /srv/conjurer/data/   # AI conversation history
sudo cp pamiec_muzyki.json     /srv/conjurer/data/   # music-DJ memory
sudo cp settings.json          /srv/conjurer/data/   # word/cyclic reactions
sudo cp system_gpt_settings.json /srv/conjurer/data/
sudo cp accident_log.json      /srv/conjurer/data/   # if present

If you skip this, the container starts with the (empty) template files baked into the image and history begins fresh.

1c. Tokens

Drop your existing netrc (the one with discord, openai, spotipy, youtube entries) into the secrets dir:

sudo cp ~/.netrc /srv/conjurer/secrets/.netrc
sudo chmod 600 /srv/conjurer/secrets/.netrc

(Alternatively skip netrc and set DISCORD_TOKEN / OPENAI_API_KEY in the env file — those take precedence.)

1d. Configure and launch

cp docker/env/bot.env.example docker/env/bot.env
# edit docker/env/bot.env: set CONJURER_FILE_SERVICE / _LIBRARIAN_SERVICE to the
# other VMs' IPs, and CONJURER_API_KEY (same value on all three) if you want auth.

docker compose -f docker/compose.bot.yaml up -d --build
docker logs -f conjurer-bot

Look for Loading … module done for each cog, Loading conanjurer commands module done, and All systems: operational.

---

2. Librarian (VM-librarian)

2a. Mount the DOI database

The librarian checks keyword hits from Crossref against a local database of DOI chunk files (0_chunk.txt … N_chunk.txt). Put that database on the VM and point the volume at it:

sudo mkdir -p /srv/librarian/doi /srv/librarian/secrets
# copy/replicate your chunk files into /srv/librarian/doi/

The old Windows path C:\Database\chunks\ is now CONJURER_LIBRARIAN_DB_PATH (defaults to /doi/ in the container). CONJURER_LIBRARIAN_MAXTHREADS (41) and CONJURER_LIBRARIAN_CHUNK (_chunk.txt) are configurable too.

2b. Configure and launch

cp docker/env/librarian.env.example docker/env/librarian.env
# edit: CONJURER_MAIN_BOT=http://BOT_VM_IP:5000, CONJURER_CROSSREF_MAILTO, CONJURER_API_KEY

docker compose -f docker/compose.librarian.yaml up -d --build
docker logs -f conjurer-librarian

---

3. Musician (VM-musician)

The web service is container-ready (docker/Dockerfile.musician + compose.musician.yaml). It containerises the Flask file/playlist service only — the Liquidsoap radio (radio_conjurer.liq), script.params and any Samba/NFS share tooling are separate and typically stay on the host or a dedicated setup (you'll adapt those yourself).

3a. Two volumes: the library and the writable state

Mount	Container path	Holds
/srv/musician/music	/music (CONJURER_MUSIC_FOLDER)	your mp3 library (indexed/served)
/srv/musician/data	/data (CONJURER_MUSICIAN_BASE)	playlists, logs, radio_log.log/persistence.log

sudo mkdir -p /srv/musician/music /srv/musician/data
# point /srv/musician/music at (or copy in) your mp3s

3b. Preserve existing playlists/state (optional)

If you already run the musician, copy its working playlists into the data volume so nothing is regenerated from scratch:

sudo cp all_playlist.playlist hit.playlist request.playlist \
        priority_queue.playlist playlist.json  /srv/musician/data/ 2>/dev/null || true

The container's entrypoint (docker/entrypoint.musician.sh) creates any missing playlists and touches radio_log.log / persistence.log empty so the track-forwarding thread waits instead of crashing when the radio runs elsewhere. It never overwrites files you copied in.

3c. Configure and launch

cp docker/env/musician.env.example docker/env/musician.env
# edit: CONJURER_MAIN_BOT=http://BOT_VM_IP:5000 and CONJURER_API_KEY (match the bot)

docker compose -f docker/compose.musician.yaml up -d --build
docker logs -f conjurer-musician

3d. Radio coupling (if you keep Liquidsoap separate)

The musician only forwards "now playing" to the bot by tailing the radio's radio_log.log / persistence.log. To wire them up, have Liquidsoap write those two files into the same /srv/musician/data directory (or set CONJURER_RADIO_LOG / CONJURER_PERSISTENCE_LOG to wherever it writes). The /stream page template is served from the baked-in /app/stream.html (override with CONJURER_STREAM_TEMPLATE if you customise it).

---

4. Networking & auth

• Open the ports between VMs on the Proxmox LAN: bot 5000, librarian 5001, musician 5000 (+ radio harbor 54321 if used). A simple ufw allow from <lan-subnet> per port is enough; do not expose them to the internet.
• Auth: set the same CONJURER_API_KEY in all three *.env files. Then every internal call carries X-Conjurer-Api-Key and each service rejects requests without it (HTTP 401). Leave it empty everywhere to disable auth (fully backward compatible). ⚠️ Setting it on only one side breaks the link.
• The addresses point at each other by VM IP (or a DNS name). Set:
  • bot: CONJURER_FILE_SERVICE, CONJURER_RADIO_HARBOR, CONJURER_LIBRARIAN_SERVICE
  • librarian & musician: CONJURER_MAIN_BOT

---

5. Verify

# bot is up and reachable from another VM:
curl http://BOT_VM_IP:5000/conjurer            # -> "ALIVE"
# librarian answers:
curl http://LIBRARIAN_VM_IP:5001/ -I           # service reachable
docker ps            # all containers "Up"
docker logs conjurer-bot --tail 50

In Discord, exercise a command that round-trips through a service (e.g. a music search that hits the musician, or a librarian query) to confirm the wiring and the API key.

---

6. Updates

cd /opt/conjurer && git pull
docker compose -f docker/compose.bot.yaml up -d --build   # rebuild + restart

Data in /srv/.../data and /doi / /music volumes survives rebuilds, so history and databases persist across updates.

---

7. Rollback / coexistence

• The native (Raspberry Pi / systemd) deployment is unaffected — none of the defaults changed; the container behaviour is opt-in via CONJURER_DATA_DIR and the other env vars. You can run both during migration.
• To roll back a VM: docker compose -f docker/compose.<svc>.yaml down and restart the previous deployment. The JSON state in /srv/.../data is plain files you can copy back to the Pi if needed.

---

Notes on the images

• Bot uses the vendored yt_dlp/ and spotify_dl/ forks (they win over the pip packages because /app is first on sys.path), so your patches stay active without the old sed hacks from install_main_bot.sh.
• Tectonic (LaTeX $latex command) is installed best-effort; if the build step fails the bot still runs, just without LaTeX. Remove that layer from Dockerfile.bot if you don't need it.
• Voice needs ffmpeg + libopus0 (both in the image). No microphone/pyaudio is required — voice is received over Discord and transcribed via AssemblyAI.