Compare commits

..

1 Commits

Author SHA1 Message Date
gitea 5d13843a99 conjurer: give librarian 60s termination grace for checkpointing
The librarian now checkpoints an in-progress search on SIGTERM and exits
within CONJURER_LIBRARIAN_GRACEFUL_TIMEOUT (default 45s). Raise k8s
terminationGracePeriodSeconds to 60 so that graceful checkpoint isn't cut
short by SIGKILL - otherwise the long DB scan restarts instead of resuming.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-02 22:11:05 +02:00
7 changed files with 7 additions and 272 deletions
+3 -3
View File
@@ -10,10 +10,10 @@ resources:
- ingress.yaml # wejście po https + przekierowanie z http
images:
- name: gitea.czernobog.pl/gitea/astrololo-data
newTag: b36b3bee
newTag: 8ebce816
- name: gitea.czernobog.pl/gitea/astrololo-logic
newTag: b36b3bee
newTag: 8ebce816
- name: gitea.czernobog.pl/gitea/astrololo-render
newTag: latest
- name: gitea.czernobog.pl/gitea/astrololo-presentation
newTag: a9f2a038
newTag: 8ebce816
-106
View File
@@ -1,106 +0,0 @@
# Production ("deploy") Conjurer bot
A second Conjurer bot that runs **alongside** the test `bot` in the `conjurer`
namespace, sharing the same librarian / musician / radio and the shared
`CONJURER_API_KEY`. It differs from the test bot only in:
- its Discord token: the **`deploy-conjurer-netrc`** secret (already created),
- distinct names/labels (`deploy-bot`) and NodePort **32443**,
- **`/data` on NFS (RWX)** instead of a block PVC — so you can seed/upload files
while it runs and back it up concurrently.
Files: `deploy-bot.yaml` (Deployment + Service), `deploy-bot-backup.yaml`
(daily backup CronJob). Both are wired into `kustomization.yaml`.
## Update channel — only on `[deploy]`
Unlike the test bot (which tracks every build), the production bot updates **only
when you promote a version**. It runs a **separate image**,
`conjurer-bot-deploy`, which the conjurer CI tags **only when the commit message
contains `[deploy]`** (it re-tags the already-built `conjurer-bot:<sha>` — same
bytes). The image-updater's `deploy-bot` alias then bumps this bot's tag.
So: normal commits update the test bot + librarian; a commit with `[deploy]` in
its message is the one that also rolls the production bot.
**Already bootstrapped:** the channel was seeded by the first `[deploy]` commit
(the merge of conjurer#20), which promoted `conjurer-bot-deploy:fbd1ec9f` — the
tag pinned in `kustomization.yaml`. From here the image-updater keeps it current
on each future `[deploy]` commit. Note the librarian is shared and still tracks
latest, so mind large bot⇄librarian version skews.
If you ever need to seed a tag by hand:
```bash
docker pull gitea.czernobog.pl/gitea/conjurer-bot:<sha>
docker tag gitea.czernobog.pl/gitea/conjurer-bot:<sha> \
gitea.czernobog.pl/gitea/conjurer-bot-deploy:<sha>
docker push gitea.czernobog.pl/gitea/conjurer-bot-deploy:<sha>
```
## One-time setup
1. **Secret** — already exists as `deploy-conjurer-netrc` (a netrc carrying the
deploy Discord token, key `.netrc`). Nothing to do; the Deployment mounts it
at `/secrets/.netrc`.
2. **NFS export** — create the data + backup dirs on the NFS server
(`192.168.1.34`, adjust to your real export):
```
/mnt/Tank1/conjurer_swap/deploy-bot/data
/mnt/Tank1/conjurer_swap/deploy-bot/backups
```
## Seeding / uploading config & state into /data
Because `/data` is a plain NFS export, you upload files by copying them onto the
share — **no `kubectl cp`, no scaling the bot down**. From the PVE box that holds
`/srv/data` (mount the same export there, or rsync over it):
```bash
# on a host that can see the NFS export:
rsync -a /srv/data/ 192.168.1.34:/mnt/Tank1/conjurer_swap/deploy-bot/data/
```
The bot roots everything under `CONJURER_DATA_DIR=/data`, so these land where it
expects them:
```
accident_log.json Conjurer_graphics/ music/ pamiec.json
pamiec_muzyki.json settings.json system_gpt_settings.json transcripts/
```
Upload as much or as little as you like — the bot seeds any missing JSON on
first run. Live-editing `pamiec.json` while the bot writes to it can race; prefer
seeding before first start or during a brief `kubectl -n conjurer scale
deploy/deploy-bot --replicas=0` window.
## Backups (production only)
`deploy-bot-backup` runs daily (04:17) and writes to the `backups` export:
- `current/` — a full rsync mirror of `/data` (incl. music/graphics), always
the latest state,
- `snapshots/deploy-state-<date>.tgz` — dated archives of the critical small
state (both memories, settings, accident log, transcripts), kept
`RETENTION_DAYS` (30) days.
The test bot is deliberately **not** backed up (amnesia there is fine).
Restore example:
```bash
tar xzf snapshots/deploy-state-2026-08-03-0417.tgz -C /mnt/.../deploy-bot/data/
```
## Routing librarian / musician callbacks
**Librarian — handled automatically.** Each bot advertises its own
`CONJURER_SELF_CALLBACK` (test `…:32442`, this bot `…:32443`) on every query and
ping, and the librarian answers each result/pong back to the bot that asked. One
librarian serves both bots — no `CONJURER_MAIN_BOT` repointing needed. (The
librarian keeps `CONJURER_MAIN_BOT: http://bot:5000` only as a fallback for a bot
that doesn't send a callback.)
**Musician — still single-target.** The musician pushes "now playing" events to
its one `CONJURER_MAIN_BOT` (currently the test bot, `192.168.1.73:32442`). Only
one bot gets radio events; repoint the musician's `CONJURER_MAIN_BOT` to
`192.168.1.73:32443` if the production bot should show them instead.
-3
View File
@@ -28,9 +28,6 @@ spec:
- { name: CONJURER_FILE_SERVICE, value: "http://192.168.1.89:5000" }
- { name: CONJURER_RADIO_HARBOR, value: "http://192.168.1.79:54321" }
# Where the librarian sends THIS bot's results/pongs back to (its own
# NodePort). Lets one librarian serve both bots - see deploy-bot.yaml.
- { name: CONJURER_SELF_CALLBACK, value: "http://192.168.1.73:32442" }
volumeMounts:
- { name: data, mountPath: /data }
- { name: netrc, mountPath: /secrets, readOnly: true }
-3
View File
@@ -17,6 +17,3 @@ spec:
images:
- { alias: "librarian", imageName: "gitea.czernobog.pl/gitea/conjurer-librarian" }
- { alias: "bot", imageName: "gitea.czernobog.pl/gitea/conjurer-bot" }
# Production bot tracks its own image, which only gets new tags on
# [deploy] commits (conjurer CI) - so it updates only on promoted builds.
- { alias: "deploy-bot", imageName: "gitea.czernobog.pl/gitea/conjurer-bot-deploy" }
-65
View File
@@ -1,65 +0,0 @@
# Daily backup of the PRODUCTION ("deploy") bot's /data.
#
# Only production is backed up - the test bot's amnesia is fine, this bot's is
# not (pamiec.json / pamiec_muzyki.json are the conversation & music memory).
# Both volumes are NFS (RWX), so this runs while the bot is live - no RWO clash.
#
# Two-part backup:
# * a full, space-efficient MIRROR (rsync --delete) of everything, incl. music
# and graphics - always the current state,
# * dated SNAPSHOTS of just the small critical state (the memories, settings,
# accident log, transcripts) so you can roll back to a point in time.
# Snapshots older than RETENTION_DAYS are pruned.
apiVersion: batch/v1
kind: CronJob
metadata:
name: deploy-bot-backup
namespace: conjurer
spec:
schedule: "17 4 * * *" # every day at 04:17
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: backup
image: alpine:3.20
command: ["/bin/sh", "-c"]
args:
- |
set -eu
apk add --no-cache rsync >/dev/null
STAMP="$(date +%F-%H%M)"
mkdir -p /backup/current /backup/snapshots
echo "[$STAMP] mirroring /data -> /backup/current"
rsync -a --delete /data/ /backup/current/
echo "[$STAMP] snapshotting critical state"
tar czf "/backup/snapshots/deploy-state-$STAMP.tgz" -C /data \
accident_log.json pamiec.json pamiec_muzyki.json \
settings.json system_gpt_settings.json transcripts \
2>/dev/null || echo " (some files absent - skipped)"
echo "[$STAMP] pruning snapshots older than ${RETENTION_DAYS}d"
find /backup/snapshots -name 'deploy-state-*.tgz' -type f \
-mtime +"${RETENTION_DAYS}" -delete
echo "[$STAMP] backup done"
env:
- { name: RETENTION_DAYS, value: "30" }
volumeMounts:
- { name: data, mountPath: /data, readOnly: true }
- { name: backup, mountPath: /backup }
volumes:
# Same export the bot mounts (read-only here).
- name: data
nfs:
server: 192.168.1.34
path: /mnt/Tank1/conjurer_swap/deploy-bot/data
# Backup target - ADJUST to your export.
- name: backup
nfs:
server: 192.168.1.34
path: /mnt/Tank1/conjurer_swap/deploy-bot/backups
-80
View File
@@ -1,80 +0,0 @@
# Production ("deploy") Conjurer bot.
#
# Same infrastructure as the test `bot` (shares librarian / musician / radio and
# the CONJURER_API_KEY), with three deliberate differences:
# 1. its Discord token comes from the `deploy-conjurer-netrc` secret,
# 2. distinct names/labels so it coexists with the test bot in this namespace,
# 3. /data is an NFS volume (RWX) instead of a block PVC - so config/state can
# be uploaded while the bot runs (just copy onto the share) and backed up
# concurrently (see deploy-bot-backup.yaml). The test bot keeps its RWO PVC.
#
# ┌─ IMPORTANT: librarian & musician call ONE bot back ─────────────────────────┐
# │ The librarian (CONJURER_MAIN_BOT) and musician push search results and │
# │ "now playing" events to a SINGLE bot address - currently the test bot's │
# │ NodePort 192.168.1.73:32442. Only one bot can receive them. To make the │
# │ PRODUCTION bot the one that gets librarian results / radio events, repoint │
# │ those services' CONJURER_MAIN_BOT to this bot's NodePort (…:32443 below). │
# └─────────────────────────────────────────────────────────────────────────────┘
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-bot
namespace: conjurer
spec:
replicas: 1 # NIGDY więcej — jedna sesja gateway na token
strategy: { type: Recreate } # NIE RollingUpdate — dwa pody = wojna o sesję Discord
selector: { matchLabels: { app: deploy-bot } }
template:
metadata: { labels: { app: deploy-bot } }
spec:
imagePullSecrets: [{ name: gitea-registry }]
containers:
- name: bot
# SEPARATE image from the test bot: conjurer-bot-deploy only gets a new
# tag when a commit message contains [deploy] (see the conjurer CI), so
# this bot updates only on versions you explicitly promote. Tag is
# managed by the image-updater (kustomization images:).
image: gitea.czernobog.pl/gitea/conjurer-bot-deploy:c8aae106
ports: [{ containerPort: 5000 }]
env:
- { name: CONJURER_DATA_DIR, value: "/data" }
- { name: CONJURER_DISCORD_HOST, value: "0.0.0.0" }
- { name: CONJURER_DISCORD_PORT, value: "5000" }
- { name: CONJURER_API_KEY, value: "d97008b3-7a5a-11f1-acd1-000b0e0f00ed" }
- { name: CONJURER_NETRC_FILE, value: "/secrets/.netrc" }
- { name: CONJURER_LIBRARIAN_SERVICE, value: "http://librarian:5001" }
- { name: CONJURER_MUSICIAN_SERVICE, value: "http://192.168.1.89:5000" }
- { name: CONJURER_RADIO_SERVICE, value: "http://192.168.1.79:5005" }
- { name: CONJURER_FILE_SERVICE, value: "http://192.168.1.89:5000" }
- { name: CONJURER_RADIO_HARBOR, value: "http://192.168.1.79:54321" }
# This bot's own callback (its NodePort). The librarian records it per
# query and answers results/pongs HERE - so it serves this bot AND the
# test bot from one instance, no CONJURER_MAIN_BOT repointing needed.
- { name: CONJURER_SELF_CALLBACK, value: "http://192.168.1.73:32443" }
volumeMounts:
- { name: data, mountPath: /data }
- { name: netrc, mountPath: /secrets, readOnly: true }
resources:
requests: { cpu: "200m", memory: "256Mi" }
limits: { cpu: "2", memory: "1Gi" }
volumes:
# /data on NFS (RWX): lets you seed/upload files while the bot runs (copy
# straight onto the export from the PVE box that holds /srv/data) and lets
# the backup CronJob read it concurrently. ADJUST server/path to your
# actual export - mirrors the librarian's 192.168.1.34 Tank1 layout.
- name: data
nfs:
server: 192.168.1.34
path: /mnt/Tank1/conjurer_swap/deploy-bot/data
- name: netrc
secret: { secretName: deploy-conjurer-netrc }
---
apiVersion: v1
kind: Service
metadata: { name: deploy-bot, namespace: conjurer }
spec:
type: NodePort # so librarian/musician COULD reach it if repointed
selector: { app: deploy-bot }
# Distinct pinned nodePort (test bot owns 32442). Point librarian/musician
# CONJURER_MAIN_BOT at 192.168.1.73:32443 to route their callbacks here.
ports: [{ port: 5000, targetPort: 5000, nodePort: 32443 }]
+2 -10
View File
@@ -4,16 +4,8 @@ resources:
- namespace.yaml
- librarian.yaml
- bot.yaml
- deploy-bot.yaml
- deploy-bot-backup.yaml
images:
- name: gitea.czernobog.pl/gitea/conjurer-librarian
newTag: fbd1ec9f
newTag: 40605b95
- name: gitea.czernobog.pl/gitea/conjurer-bot
newTag: fbd1ec9f
# Production bot channel - only bumped when a [deploy]-tagged build appears.
# Bootstrapped to fbd1ec9f: that's the image the first [deploy] build (merge
# of conjurer#20) promoted to conjurer-bot-deploy. From here the image-updater
# keeps it current across future [deploy] commits.
- name: gitea.czernobog.pl/gitea/conjurer-bot-deploy
newTag: fbd1ec9f
newTag: 8e18071b