conjurer/docs/deployment/DEPLOYMENT_GUIDE.md

Conjurer Deployment Guide

This document walks through deploying the Conjurer stack (Discord bot, music service, librarian service) on two Raspberry Pi 4s and one Windows host, first with plain Docker Compose, then with a future Kubernetes setup.

1. Current Hardware Layout

• Windows PC: Stores persistent data (JSON memories, configs, music catalogue). Shares folders over the network (SMB) for the Pis.
• Raspberry Pi A (“radio”): Runs the Liquidsoap/liq radio pipeline and the musician service (Flask + file watcher).
• Raspberry Pi B (“bot”): Runs the Discord bot, communication bridge, and librarian Flask service.

You can rebalance as follows:

Service	Recommended Host	Notes
Discord bot + comms	Raspberry Pi B	Needs outbound internet, moderate CPU
Librarian service	Raspberry Pi B	CPU-heavy during Crossref queries; keep close to bot
Musician service	Raspberry Pi A	Has direct disk access to music, same box as Liquidsoap
Data storage	Windows	Expose via SMB; mount inside containers

2. Prepare Shared Storage on Windows

1. Create directories, e.g. C:\Conjurer\config, C:\Conjurer\logs, C:\Conjurer\music, C:\Conjurer\playlists, C:\Conjurer\secrets.
2. Copy your existing JSON settings (settings.json, pamiec.json, pamiec_muzyki.json, system_gpt_settings.json, etc.) into config.
3. Create blank placeholder files if they do not exist yet.
4. Share the root folder (C:\Conjurer) over SMB with read/write access for the Pi user (create credentials if necessary).

3. Configure Environment Files

1. On your workstation, copy the example env files:

   cp docker/env/bot.env.example docker/env/bot.env
   cp docker/env/musician.env.example docker/env/musician.env
   cp docker/env/librarian.env.example docker/env/librarian.env
   

2. Edit each docker/env/*.env to replace HACKME! with real values:
   • DISCORD_TOKEN, OPENAI_API_KEY, CONJURER_API_KEY (use the same value for all services).
   • For musician/librarian, adjust mounts to the SMB paths you will mount on the Pis, e.g. /mnt/conjurer/music.
   • Set CONJURER_CROSSREF_MAILTO to a real email as required by Crossref.
3. If you rely on .netrc, copy it to C:\Conjurer\secrets\.netrc and set CONJURER_NETRC_FILE accordingly.

4. Install Docker on Raspberry Pis and Windows

Raspberry Pi

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
sudo reboot

# Install docker compose plugin
sudo apt-get install docker-compose-plugin

Windows

• Install Docker Desktop.
• Enable WSL2 backend and expose the shared Windows folders to the containers (Docker Desktop settings → Resources → File Sharing).

5. Deploy Musician Service (Pi A)

1. SSH into Raspberry Pi A.
2. Mount the Windows SMB share:

   sudo mkdir -p /mnt/conjurer
   sudo apt-get install cifs-utils
   sudo mount -t cifs //WINDOWS_HOST/Conjurer /mnt/conjurer -o user=YOURUSER
   

   Add an entry to /etc/fstab for persistence.
3. Copy the repo to the Pi or git clone it.
4. On Pi A, create override compose file (optional) pointing volumes to /mnt/conjurer.
5. Start only the musician service:

   docker compose --profile musician up --build -d musician
   

   Alternatively, duplicate docker-compose.yml, strip other services, and run docker compose up -d.

6. Deploy Bot + Librarian (Pi B)

1. Repeat SMB mount on Pi B (same mount path).
2. Copy repo / pull latest changes.
3. Create .env files with tokens (or copy from control machine).
4. Start bot and librarian:

   docker compose up -d bot librarian
   

7. Optional: Run Supporting Liquidsoap Radio

• Keep Liquidsoap on Pi A as-is, using the same music directories. Ensure the musician container has read access to those directories (bind mount).

8. Verifying

1. docker ps on each Pi to confirm containers running.
2. Inspect logs under the mounted logs directory (/mnt/conjurer/logs).
3. Join Discord server; issue commands to confirm functionality.
4. Hit health endpoints manually (e.g. curl http://PIB:5000/conjurer).

Rebalancing Suggestions

• If librarian CPU spikes become an issue, move it to Pi A or another host.
• If you add a dedicated NAS, mount the network share read-only for the musician container and read/write for other services.

9. Future Kubernetes Deployment (Outline)

Hardware Considerations

• Minimum three nodes for HA: use the existing two Pis plus one additional Pi 4 (8 GB preferred). Use Windows PC as storage provider via NFS/SMB CSI driver or as a data gateway.
• Consider Pi clusters with USB SSDs for better I/O.

Cluster Setup Steps

1. Install a lightweight Kubernetes distribution (e.g., k3s) on each Pi:

   curl -sfL https://get.k3s.io | sh -
   # On additional nodes
   curl -sfL https://get.k3s.io | K3S_URL=https://MASTER:6443 K3S_TOKEN=HACKME sh -
   

2. Install MetalLB for load balancer support on LAN.
3. Configure persistent volumes using:
   • nfs-subdir-external-provisioner pointing to Windows share (ensure Windows host supports NFS or run an NFS gateway on another machine).
   • Alternatively, attach individual USB drives to each Pi and use local-path-provisioner for node-local storage.
4. Create Kubernetes Secret objects for tokens (DISCORD_TOKEN, etc.).
5. Define Deployment manifests for each service (bot, musician, librarian) and associated Services.
6. Expose Discord bot ports via NodePort or Ingress.
7. Use StatefulSet if you need stable identity for the musician service (due to local storage).

Optimisation Tips

• Keep CPU-heavy librarian pods optionally on a beefier node; use nodeSelector/affinity to pin workloads.
• Consider splitting the persistent storage: music on Pi A (USB disk), logs and configs on Pi B, backups on Windows.
• For improved reliability, add at least one extra Pi for quorum and to host the communication bridge if the bot node fails.

Summary Checklist

1. Prepare Windows shares & tokens.
2. Configure docker/env/*.env using HACKME! templates as reference.
3. Install Docker on Pis, mount network shares.
4. Launch musician on Pi A, bot + librarian on Pi B.
5. Verify Discord functionality and API endpoints.
6. Plan Kubernetes migration when ready (k3s + MetalLB + storage provisioner).