mirror of
https://github.com/migatu/conjurer.git
synced 2026-07-14 21:38:38 +00:00
docker: Proxmox deployment for bot/librarian/musician + code fixes
Containerises the three services (each intended for its own Proxmox VM)
and adds the code changes needed to run cleanly on Linux/Docker.
Code fixes:
- constants.py: CONJURER_DATA_DIR roots all writable bot state under one
mounted volume (per-variable overrides still win; native Pi unaffected)
- conjurer_librarian/search_bot.py + scrape_bot.py: the hardcoded Windows
DOI database path (C:\Database\chunks\) is now CONJURER_LIBRARIAN_DB_PATH,
with CONJURER_LIBRARIAN_MAXTHREADS / _CHUNK also env-overridable
Docker:
- docker/Dockerfile.{bot,librarian,musician} + compose.{bot,librarian,musician}.yaml
- docker/env/*.env.example (force-added; real *.env stays gitignored)
- docker/entrypoint.bot.sh seeds default JSON state into /data only when
absent, so preserved history is never overwritten
- .dockerignore
- docs/deployment/DOCKER_PROXMOX.md: step-by-step runbook incl. preserving
the existing command/conversation history and cross-VM auth
The bot image uses the vendored yt_dlp/spotify_dl forks (they win on
sys.path over the pip packages), dropping the old sed patching.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Michal Tuszowski
@@ -0,0 +1,215 @@
|
||||
# Conjurer on Docker / Proxmox
|
||||
|
||||
Runbook for running Conjurer as Docker containers across Proxmox VMs:
|
||||
|
||||
| Component | VM | Container | Port | Image |
|
||||
|-----------|----|-----------|------|-------|
|
||||
| **Main bot** | VM-bot | `conjurer-bot` | 5000 (Flask comm) | `docker/Dockerfile.bot` |
|
||||
| **Librarian** | VM-librarian | `conjurer-librarian` | 5001 | `docker/Dockerfile.librarian` |
|
||||
| **Musician** | VM-musician | `conjurer-musician` | 5000 (+ radio) | `docker/Dockerfile.musician` |
|
||||
|
||||
The three talk to each other over HTTP on the Proxmox LAN. Direction of calls:
|
||||
|
||||
```
|
||||
bot --(/mp3,/get_music,/add_to_priority,...)--> musician
|
||||
bot --(/query)--------------------------------> librarian
|
||||
musician --(/prepped_tracks)--------------------> bot
|
||||
librarian --(/conjurer results)-----------------> bot
|
||||
```
|
||||
|
||||
Everything is configured through `CONJURER_*` environment variables (see the
|
||||
`docker/env/*.env.example` files). Nothing is hardcoded to a host path anymore.
|
||||
|
||||
---
|
||||
|
||||
## 0. Prerequisites (per VM)
|
||||
|
||||
Create a small Linux VM in Proxmox (Debian 12 / Ubuntu 22.04+ is fine), then
|
||||
install Docker:
|
||||
|
||||
```bash
|
||||
sudo apt-get update && sudo apt-get install -y ca-certificates curl git
|
||||
curl -fsSL https://get.docker.com | sh
|
||||
sudo usermod -aG docker "$USER" # log out/in afterwards
|
||||
```
|
||||
|
||||
Clone the repo on each VM (they build from it):
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /opt && cd /opt
|
||||
git clone https://github.com/migatu/conjurer.git
|
||||
cd conjurer
|
||||
```
|
||||
|
||||
> All `docker compose` commands below are run **from the repo root** (`/opt/conjurer`),
|
||||
> because the compose files use `context: ..` relative to `docker/`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Main bot (VM-bot)
|
||||
|
||||
### 1a. Prepare host directories
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /srv/conjurer/data /srv/conjurer/secrets
|
||||
```
|
||||
|
||||
### 1b. Preserve existing command history ⭐
|
||||
|
||||
The bot's conversation memory and settings live in JSON files. Copy them from
|
||||
your current deployment (e.g. the Pi's `/home/pi/Conjurer/`) into the data
|
||||
volume so the history carries over:
|
||||
|
||||
```bash
|
||||
# run on the Pi, or scp the files across, then place them here:
|
||||
sudo cp pamiec.json /srv/conjurer/data/ # AI conversation history
|
||||
sudo cp pamiec_muzyki.json /srv/conjurer/data/ # music-DJ memory
|
||||
sudo cp settings.json /srv/conjurer/data/ # word/cyclic reactions
|
||||
sudo cp system_gpt_settings.json /srv/conjurer/data/
|
||||
sudo cp accident_log.json /srv/conjurer/data/ # if present
|
||||
```
|
||||
|
||||
If you skip this, the container starts with the (empty) template files baked
|
||||
into the image and history begins fresh.
|
||||
|
||||
### 1c. Tokens
|
||||
|
||||
Drop your existing netrc (the one with `discord`, `openai`, `spotipy`,
|
||||
`youtube` entries) into the secrets dir:
|
||||
|
||||
```bash
|
||||
sudo cp ~/.netrc /srv/conjurer/secrets/.netrc
|
||||
sudo chmod 600 /srv/conjurer/secrets/.netrc
|
||||
```
|
||||
|
||||
(Alternatively skip netrc and set `DISCORD_TOKEN` / `OPENAI_API_KEY` in the env
|
||||
file — those take precedence.)
|
||||
|
||||
### 1d. Configure and launch
|
||||
|
||||
```bash
|
||||
cp docker/env/bot.env.example docker/env/bot.env
|
||||
# edit docker/env/bot.env: set CONJURER_FILE_SERVICE / _LIBRARIAN_SERVICE to the
|
||||
# other VMs' IPs, and CONJURER_API_KEY (same value on all three) if you want auth.
|
||||
|
||||
docker compose -f docker/compose.bot.yaml up -d --build
|
||||
docker logs -f conjurer-bot
|
||||
```
|
||||
|
||||
Look for `Loading … module done` for each cog, `Loading conanjurer commands
|
||||
module done`, and `All systems: operational`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Librarian (VM-librarian)
|
||||
|
||||
### 2a. Mount the DOI database
|
||||
|
||||
The librarian checks keyword hits from Crossref against a local database of
|
||||
DOI chunk files (`0_chunk.txt … N_chunk.txt`). Put that database on the VM and
|
||||
point the volume at it:
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /srv/librarian/doi /srv/librarian/secrets
|
||||
# copy/replicate your chunk files into /srv/librarian/doi/
|
||||
```
|
||||
|
||||
> The old Windows path `C:\Database\chunks\` is now `CONJURER_LIBRARIAN_DB_PATH`
|
||||
> (defaults to `/doi/` in the container). `CONJURER_LIBRARIAN_MAXTHREADS` (41)
|
||||
> and `CONJURER_LIBRARIAN_CHUNK` (`_chunk.txt`) are configurable too.
|
||||
|
||||
### 2b. Configure and launch
|
||||
|
||||
```bash
|
||||
cp docker/env/librarian.env.example docker/env/librarian.env
|
||||
# edit: CONJURER_MAIN_BOT=http://BOT_VM_IP:5000, CONJURER_CROSSREF_MAILTO, CONJURER_API_KEY
|
||||
|
||||
docker compose -f docker/compose.librarian.yaml up -d --build
|
||||
docker logs -f conjurer-librarian
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Musician (VM-musician) — you will adapt this
|
||||
|
||||
The web service is container-ready (`docker/Dockerfile.musician` +
|
||||
`compose.musician.yaml`). Note it containerises the **Flask file/playlist
|
||||
service only**; the Liquidsoap radio (`radio_conjurer.liq`) and any
|
||||
Samba/NFS share tooling are separate and typically stay on the host or a
|
||||
dedicated setup.
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /srv/musician/music /srv/musician/data
|
||||
cp docker/env/musician.env.example docker/env/musician.env
|
||||
# edit: CONJURER_MAIN_BOT=http://BOT_VM_IP:5000, CONJURER_API_KEY, mount your mp3s at /music
|
||||
|
||||
docker compose -f docker/compose.musician.yaml up -d --build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Networking & auth
|
||||
|
||||
- Open the ports between VMs on the Proxmox LAN: **bot 5000**, **librarian 5001**,
|
||||
**musician 5000** (+ radio harbor 54321 if used). A simple `ufw allow from
|
||||
<lan-subnet>` per port is enough; do not expose them to the internet.
|
||||
- **Auth:** set the same `CONJURER_API_KEY` in all three `*.env` files. Then
|
||||
every internal call carries `X-Conjurer-Api-Key` and each service rejects
|
||||
requests without it (HTTP 401). Leave it empty everywhere to disable auth
|
||||
(fully backward compatible). ⚠️ Setting it on only one side breaks the link.
|
||||
- The addresses point at each other by VM IP (or a DNS name). Set:
|
||||
- bot: `CONJURER_FILE_SERVICE`, `CONJURER_RADIO_HARBOR`, `CONJURER_LIBRARIAN_SERVICE`
|
||||
- librarian & musician: `CONJURER_MAIN_BOT`
|
||||
|
||||
---
|
||||
|
||||
## 5. Verify
|
||||
|
||||
```bash
|
||||
# bot is up and reachable from another VM:
|
||||
curl http://BOT_VM_IP:5000/conjurer # -> "ALIVE"
|
||||
# librarian answers:
|
||||
curl http://LIBRARIAN_VM_IP:5001/ -I # service reachable
|
||||
docker ps # all containers "Up"
|
||||
docker logs conjurer-bot --tail 50
|
||||
```
|
||||
|
||||
In Discord, exercise a command that round-trips through a service (e.g. a music
|
||||
search that hits the musician, or a librarian query) to confirm the wiring and
|
||||
the API key.
|
||||
|
||||
---
|
||||
|
||||
## 6. Updates
|
||||
|
||||
```bash
|
||||
cd /opt/conjurer && git pull
|
||||
docker compose -f docker/compose.bot.yaml up -d --build # rebuild + restart
|
||||
```
|
||||
|
||||
Data in `/srv/.../data` and `/doi` / `/music` volumes survives rebuilds, so
|
||||
history and databases persist across updates.
|
||||
|
||||
---
|
||||
|
||||
## 7. Rollback / coexistence
|
||||
|
||||
- The native (Raspberry Pi / systemd) deployment is unaffected — none of the
|
||||
defaults changed; the container behaviour is opt-in via `CONJURER_DATA_DIR`
|
||||
and the other env vars. You can run both during migration.
|
||||
- To roll back a VM: `docker compose -f docker/compose.<svc>.yaml down` and
|
||||
restart the previous deployment. The JSON state in `/srv/.../data` is plain
|
||||
files you can copy back to the Pi if needed.
|
||||
|
||||
---
|
||||
|
||||
## Notes on the images
|
||||
|
||||
- **Bot** uses the vendored `yt_dlp/` and `spotify_dl/` forks (they win over the
|
||||
pip packages because `/app` is first on `sys.path`), so your patches stay
|
||||
active without the old `sed` hacks from `install_main_bot.sh`.
|
||||
- **Tectonic** (LaTeX `$latex` command) is installed best-effort; if the build
|
||||
step fails the bot still runs, just without LaTeX. Remove that layer from
|
||||
`Dockerfile.bot` if you don't need it.
|
||||
- Voice needs `ffmpeg` + `libopus0` (both in the image). No microphone/pyaudio
|
||||
is required — voice is received over Discord and transcribed via AssemblyAI.
|
||||
Reference in New Issue
Block a user