conjurer: deploy bot on its own [deploy]-gated image channel

The production bot now tracks conjurer-bot-deploy instead of conjurer-bot,
so it updates only when the conjurer CI promotes a build (commit message
contains [deploy]). Adds the image-updater 'deploy-bot' alias and the
kustomization images entry for it; DEPLOY-BOT.md documents the channel and
the one-time bootstrap. Test bot + librarian keep tracking every build.

Pairs with conjurer#20 (the CI promotion step).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

astrololo/kustomization.yaml

@@ -10,10 +10,10 @@ resources:
   - ingress.yaml # wejście po https + przekierowanie z http
 images:
   - name: gitea.czernobog.pl/gitea/astrololo-data
-    newTag: 8ebce816
+    newTag: b36b3bee
   - name: gitea.czernobog.pl/gitea/astrololo-logic
-    newTag: 8ebce816
+    newTag: b36b3bee
   - name: gitea.czernobog.pl/gitea/astrololo-render
     newTag: latest
   - name: gitea.czernobog.pl/gitea/astrololo-presentation
-    newTag: 8ebce816
+    newTag: a9f2a038

conjurer/DEPLOY-BOT.md

@@ -0,0 +1,106 @@
+# Production ("deploy") Conjurer bot
+
+A second Conjurer bot that runs **alongside** the test `bot` in the `conjurer`
+namespace, sharing the same librarian / musician / radio and the shared
+`CONJURER_API_KEY`. It differs from the test bot only in:
+
+- its Discord token: the **`deploy-conjurer-netrc`** secret (already created),
+- distinct names/labels (`deploy-bot`) and NodePort **32443**,
+- **`/data` on NFS (RWX)** instead of a block PVC — so you can seed/upload files
+  while it runs and back it up concurrently.
+
+Files: `deploy-bot.yaml` (Deployment + Service), `deploy-bot-backup.yaml`
+(daily backup CronJob). Both are wired into `kustomization.yaml`.
+
+## Update channel — only on `[deploy]`
+
+Unlike the test bot (which tracks every build), the production bot updates **only
+when you promote a version**. It runs a **separate image**,
+`conjurer-bot-deploy`, which the conjurer CI tags **only when the commit message
+contains `[deploy]`** (it re-tags the already-built `conjurer-bot:<sha>` — same
+bytes). The image-updater's `deploy-bot` alias then bumps this bot's tag.
+
+So: normal commits update the test bot + librarian; a commit with `[deploy]` in
+its message is the one that also rolls the production bot.
+
+**Already bootstrapped:** the channel was seeded by the first `[deploy]` commit
+(the merge of conjurer#20), which promoted `conjurer-bot-deploy:fbd1ec9f` — the
+tag pinned in `kustomization.yaml`. From here the image-updater keeps it current
+on each future `[deploy]` commit. Note the librarian is shared and still tracks
+latest, so mind large bot⇄librarian version skews.
+
+If you ever need to seed a tag by hand:
+```bash
+docker pull gitea.czernobog.pl/gitea/conjurer-bot:<sha>
+docker tag  gitea.czernobog.pl/gitea/conjurer-bot:<sha> \
+            gitea.czernobog.pl/gitea/conjurer-bot-deploy:<sha>
+docker push gitea.czernobog.pl/gitea/conjurer-bot-deploy:<sha>
+```
+
+## One-time setup
+
+1. **Secret** — already exists as `deploy-conjurer-netrc` (a netrc carrying the
+   deploy Discord token, key `.netrc`). Nothing to do; the Deployment mounts it
+   at `/secrets/.netrc`.
+
+2. **NFS export** — create the data + backup dirs on the NFS server
+   (`192.168.1.34`, adjust to your real export):
+   ```
+   /mnt/Tank1/conjurer_swap/deploy-bot/data
+   /mnt/Tank1/conjurer_swap/deploy-bot/backups
+   ```
+
+## Seeding / uploading config & state into /data
+
+Because `/data` is a plain NFS export, you upload files by copying them onto the
+share — **no `kubectl cp`, no scaling the bot down**. From the PVE box that holds
+`/srv/data` (mount the same export there, or rsync over it):
+
+```bash
+# on a host that can see the NFS export:
+rsync -a /srv/data/ 192.168.1.34:/mnt/Tank1/conjurer_swap/deploy-bot/data/
+```
+
+The bot roots everything under `CONJURER_DATA_DIR=/data`, so these land where it
+expects them:
+
+```
+accident_log.json  Conjurer_graphics/  music/  pamiec.json
+pamiec_muzyki.json  settings.json  system_gpt_settings.json  transcripts/
+```
+
+Upload as much or as little as you like — the bot seeds any missing JSON on
+first run. Live-editing `pamiec.json` while the bot writes to it can race; prefer
+seeding before first start or during a brief `kubectl -n conjurer scale
+deploy/deploy-bot --replicas=0` window.
+
+## Backups (production only)
+
+`deploy-bot-backup` runs daily (04:17) and writes to the `backups` export:
+
+- `current/` — a full rsync mirror of `/data` (incl. music/graphics), always
+  the latest state,
+- `snapshots/deploy-state-<date>.tgz` — dated archives of the critical small
+  state (both memories, settings, accident log, transcripts), kept
+  `RETENTION_DAYS` (30) days.
+
+The test bot is deliberately **not** backed up (amnesia there is fine).
+
+Restore example:
+```bash
+tar xzf snapshots/deploy-state-2026-08-03-0417.tgz -C /mnt/.../deploy-bot/data/
+```
+
+## Routing librarian / musician callbacks
+
+**Librarian — handled automatically.** Each bot advertises its own
+`CONJURER_SELF_CALLBACK` (test `…:32442`, this bot `…:32443`) on every query and
+ping, and the librarian answers each result/pong back to the bot that asked. One
+librarian serves both bots — no `CONJURER_MAIN_BOT` repointing needed. (The
+librarian keeps `CONJURER_MAIN_BOT: http://bot:5000` only as a fallback for a bot
+that doesn't send a callback.)
+
+**Musician — still single-target.** The musician pushes "now playing" events to
+its one `CONJURER_MAIN_BOT` (currently the test bot, `192.168.1.73:32442`). Only
+one bot gets radio events; repoint the musician's `CONJURER_MAIN_BOT` to
+`192.168.1.73:32443` if the production bot should show them instead.

conjurer/bot.yaml

@@ -28,6 +28,9 @@ spec:
             - { name: CONJURER_FILE_SERVICE,     value: "http://192.168.1.89:5000" }
 
             - { name: CONJURER_RADIO_HARBOR,      value: "http://192.168.1.79:54321" }
+            # Where the librarian sends THIS bot's results/pongs back to (its own
+            # NodePort). Lets one librarian serve both bots - see deploy-bot.yaml.
+            - { name: CONJURER_SELF_CALLBACK,     value: "http://192.168.1.73:32442" }
           volumeMounts:
             - { name: data,  mountPath: /data }
             - { name: netrc, mountPath: /secrets, readOnly: true }

conjurer/conjurer-imageupdater.yaml

@@ -15,5 +15,8 @@ spec:
   applicationRefs:
     - namePattern: "conjurer"
       images:
-        - { alias: "librarian", imageName: "gitea.czernobog.pl/gitea/conjurer-librarian" }
-        - { alias: "bot",       imageName: "gitea.czernobog.pl/gitea/conjurer-bot" }
+        - { alias: "librarian",  imageName: "gitea.czernobog.pl/gitea/conjurer-librarian" }
+        - { alias: "bot",        imageName: "gitea.czernobog.pl/gitea/conjurer-bot" }
+        # Production bot tracks its own image, which only gets new tags on
+        # [deploy] commits (conjurer CI) - so it updates only on promoted builds.
+        - { alias: "deploy-bot", imageName: "gitea.czernobog.pl/gitea/conjurer-bot-deploy" }

conjurer/deploy-bot-backup.yaml

@@ -0,0 +1,65 @@
+# Daily backup of the PRODUCTION ("deploy") bot's /data.
+#
+# Only production is backed up - the test bot's amnesia is fine, this bot's is
+# not (pamiec.json / pamiec_muzyki.json are the conversation & music memory).
+# Both volumes are NFS (RWX), so this runs while the bot is live - no RWO clash.
+#
+# Two-part backup:
+#   * a full, space-efficient MIRROR (rsync --delete) of everything, incl. music
+#     and graphics - always the current state,
+#   * dated SNAPSHOTS of just the small critical state (the memories, settings,
+#     accident log, transcripts) so you can roll back to a point in time.
+# Snapshots older than RETENTION_DAYS are pruned.
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: deploy-bot-backup
+  namespace: conjurer
+spec:
+  schedule: "17 4 * * *"          # every day at 04:17
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      template:
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: backup
+              image: alpine:3.20
+              command: ["/bin/sh", "-c"]
+              args:
+                - |
+                  set -eu
+                  apk add --no-cache rsync >/dev/null
+                  STAMP="$(date +%F-%H%M)"
+                  mkdir -p /backup/current /backup/snapshots
+                  echo "[$STAMP] mirroring /data -> /backup/current"
+                  rsync -a --delete /data/ /backup/current/
+                  echo "[$STAMP] snapshotting critical state"
+                  tar czf "/backup/snapshots/deploy-state-$STAMP.tgz" -C /data \
+                      accident_log.json pamiec.json pamiec_muzyki.json \
+                      settings.json system_gpt_settings.json transcripts \
+                      2>/dev/null || echo "  (some files absent - skipped)"
+                  echo "[$STAMP] pruning snapshots older than ${RETENTION_DAYS}d"
+                  find /backup/snapshots -name 'deploy-state-*.tgz' -type f \
+                      -mtime +"${RETENTION_DAYS}" -delete
+                  echo "[$STAMP] backup done"
+              env:
+                - { name: RETENTION_DAYS, value: "30" }
+              volumeMounts:
+                - { name: data,   mountPath: /data,   readOnly: true }
+                - { name: backup, mountPath: /backup }
+          volumes:
+            # Same export the bot mounts (read-only here).
+            - name: data
+              nfs:
+                server: 192.168.1.34
+                path: /mnt/Tank1/conjurer_swap/deploy-bot/data
+            # Backup target - ADJUST to your export.
+            - name: backup
+              nfs:
+                server: 192.168.1.34
+                path: /mnt/Tank1/conjurer_swap/deploy-bot/backups

conjurer/deploy-bot.yaml

@@ -0,0 +1,80 @@
+# Production ("deploy") Conjurer bot.
+#
+# Same infrastructure as the test `bot` (shares librarian / musician / radio and
+# the CONJURER_API_KEY), with three deliberate differences:
+#   1. its Discord token comes from the `deploy-conjurer-netrc` secret,
+#   2. distinct names/labels so it coexists with the test bot in this namespace,
+#   3. /data is an NFS volume (RWX) instead of a block PVC - so config/state can
+#      be uploaded while the bot runs (just copy onto the share) and backed up
+#      concurrently (see deploy-bot-backup.yaml). The test bot keeps its RWO PVC.
+#
+# ┌─ IMPORTANT: librarian & musician call ONE bot back ─────────────────────────┐
+# │ The librarian (CONJURER_MAIN_BOT) and musician push search results and      │
+# │ "now playing" events to a SINGLE bot address - currently the test bot's     │
+# │ NodePort 192.168.1.73:32442. Only one bot can receive them. To make the     │
+# │ PRODUCTION bot the one that gets librarian results / radio events, repoint   │
+# │ those services' CONJURER_MAIN_BOT to this bot's NodePort (…:32443 below).    │
+# └─────────────────────────────────────────────────────────────────────────────┘
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: deploy-bot
+  namespace: conjurer
+spec:
+  replicas: 1                     # NIGDY więcej — jedna sesja gateway na token
+  strategy: { type: Recreate }    # NIE RollingUpdate — dwa pody = wojna o sesję Discord
+  selector: { matchLabels: { app: deploy-bot } }
+  template:
+    metadata: { labels: { app: deploy-bot } }
+    spec:
+      imagePullSecrets: [{ name: gitea-registry }]
+      containers:
+        - name: bot
+          # SEPARATE image from the test bot: conjurer-bot-deploy only gets a new
+          # tag when a commit message contains [deploy] (see the conjurer CI), so
+          # this bot updates only on versions you explicitly promote. Tag is
+          # managed by the image-updater (kustomization images:).
+          image: gitea.czernobog.pl/gitea/conjurer-bot-deploy:c8aae106
+          ports: [{ containerPort: 5000 }]
+          env:
+            - { name: CONJURER_DATA_DIR,      value: "/data" }
+            - { name: CONJURER_DISCORD_HOST,  value: "0.0.0.0" }
+            - { name: CONJURER_DISCORD_PORT,  value: "5000" }
+            - { name: CONJURER_API_KEY,  value: "d97008b3-7a5a-11f1-acd1-000b0e0f00ed" }
+            - { name: CONJURER_NETRC_FILE, value: "/secrets/.netrc" }
+            - { name: CONJURER_LIBRARIAN_SERVICE, value: "http://librarian:5001" }
+            - { name: CONJURER_MUSICIAN_SERVICE,  value: "http://192.168.1.89:5000" }
+            - { name: CONJURER_RADIO_SERVICE,     value: "http://192.168.1.79:5005" }
+            - { name: CONJURER_FILE_SERVICE,      value: "http://192.168.1.89:5000" }
+            - { name: CONJURER_RADIO_HARBOR,      value: "http://192.168.1.79:54321" }
+            # This bot's own callback (its NodePort). The librarian records it per
+            # query and answers results/pongs HERE - so it serves this bot AND the
+            # test bot from one instance, no CONJURER_MAIN_BOT repointing needed.
+            - { name: CONJURER_SELF_CALLBACK,     value: "http://192.168.1.73:32443" }
+          volumeMounts:
+            - { name: data,  mountPath: /data }
+            - { name: netrc, mountPath: /secrets, readOnly: true }
+          resources:
+            requests: { cpu: "200m", memory: "256Mi" }
+            limits:   { cpu: "2",    memory: "1Gi" }
+      volumes:
+        # /data on NFS (RWX): lets you seed/upload files while the bot runs (copy
+        # straight onto the export from the PVE box that holds /srv/data) and lets
+        # the backup CronJob read it concurrently. ADJUST server/path to your
+        # actual export - mirrors the librarian's 192.168.1.34 Tank1 layout.
+        - name: data
+          nfs:
+            server: 192.168.1.34
+            path: /mnt/Tank1/conjurer_swap/deploy-bot/data
+        - name: netrc
+          secret: { secretName: deploy-conjurer-netrc }
+---
+apiVersion: v1
+kind: Service
+metadata: { name: deploy-bot, namespace: conjurer }
+spec:
+  type: NodePort                  # so librarian/musician COULD reach it if repointed
+  selector: { app: deploy-bot }
+  # Distinct pinned nodePort (test bot owns 32442). Point librarian/musician
+  # CONJURER_MAIN_BOT at 192.168.1.73:32443 to route their callbacks here.
+  ports: [{ port: 5000, targetPort: 5000, nodePort: 32443 }]

conjurer/kustomization.yaml

@@ -4,8 +4,16 @@ resources:
   - namespace.yaml
   - librarian.yaml
   - bot.yaml
+  - deploy-bot.yaml
+  - deploy-bot-backup.yaml
 images:
   - name: gitea.czernobog.pl/gitea/conjurer-librarian
-    newTag: 5d321f2f
+    newTag: fbd1ec9f
   - name: gitea.czernobog.pl/gitea/conjurer-bot
-    newTag: 5d321f2f
+    newTag: fbd1ec9f
+  # Production bot channel - only bumped when a [deploy]-tagged build appears.
+  # Bootstrapped to fbd1ec9f: that's the image the first [deploy] build (merge
+  # of conjurer#20) promoted to conjurer-bot-deploy. From here the image-updater
+  # keeps it current across future [deploy] commits.
+  - name: gitea.czernobog.pl/gitea/conjurer-bot-deploy
+    newTag: fbd1ec9f

conjurer/librarian.yaml

@@ -10,6 +10,11 @@ spec:
     metadata: { labels: { app: librarian } }
     spec:
       imagePullSecrets: [{ name: gitea-registry }]
+      # On SIGTERM the librarian checkpoints the running search and exits within
+      # CONJURER_LIBRARIAN_GRACEFUL_TIMEOUT (default 45s). Give k8s enough grace
+      # to let that finish before it SIGKILLs - otherwise a long scan's checkpoint
+      # is cut short and the search restarts from scratch instead of resuming.
+      terminationGracePeriodSeconds: 60
       containers:
         - name: librarian
           image: gitea.czernobog.pl/gitea/conjurer-librarian:e243777e
@@ -33,8 +38,12 @@ spec:
             - { name: doi,   mountPath: /doi, readOnly: true }
             - { name: state, mountPath: /lib_temp_files }
           resources:
-            requests: { cpu: "100m", memory: "256Mi" }
-            limits:   { cpu: "1",    memory: "1Gi" }
+            requests: { cpu: "100m", memory: "512Mi" }
+            # Headroom bump: the work-queue OOM is fixed in code, but a deep
+            # search still pulls up to 15000 Crossref records into RAM and the
+            # accumulating result JSONs are loaded whole. 2Gi gives margin so a
+            # big search isn't OOM-killed. Tune down if the node is tight.
+            limits:   { cpu: "1",    memory: "2Gi" }
       volumes:
         - name: doi
           nfs: