docker: Proxmox deployment for bot/librarian/musician + code fixes

Containerises the three services (each intended for its own Proxmox VM)
and adds the code changes needed to run cleanly on Linux/Docker.

Code fixes:
- constants.py: CONJURER_DATA_DIR roots all writable bot state under one
  mounted volume (per-variable overrides still win; native Pi unaffected)
- conjurer_librarian/search_bot.py + scrape_bot.py: the hardcoded Windows
  DOI database path (C:\Database\chunks\) is now CONJURER_LIBRARIAN_DB_PATH,
  with CONJURER_LIBRARIAN_MAXTHREADS / _CHUNK also env-overridable

Docker:
- docker/Dockerfile.{bot,librarian,musician} + compose.{bot,librarian,musician}.yaml
- docker/env/*.env.example (force-added; real *.env stays gitignored)
- docker/entrypoint.bot.sh seeds default JSON state into /data only when
  absent, so preserved history is never overwritten
- .dockerignore
- docs/deployment/DOCKER_PROXMOX.md: step-by-step runbook incl. preserving
  the existing command/conversation history and cross-VM auth

The bot image uses the vendored yt_dlp/spotify_dl forks (they win on
sys.path over the pip packages), dropping the old sed patching.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Michal Tuszowski
2026-07-02 08:42:31 +02:00
committed by Michał Tuszowski
parent 9c8384b10c
commit 597bc004fc
15 changed files with 528 additions and 12 deletions
+4 -3
View File
@@ -4,6 +4,7 @@ This module contains the code for the scrape bot.
import json
import logging
import os
import random
import re
import time
@@ -15,9 +16,9 @@ from urllib.request import urlopen
from requests import ConnectionError as RequestsConnectionError
from requests import ConnectTimeout, Timeout
SCR_DATABASE_PATH = r"C:\\Database\\chunks\\"
SCR_FILENAME = "40_chunk.txt"
SCR_ENCODING = "utf-8"
SCR_DATABASE_PATH = os.getenv("CONJURER_LIBRARIAN_DB_PATH", r"C:\\Database\\chunks\\")
SCR_FILENAME = os.getenv("CONJURER_LIBRARIAN_SCRAPE_CHUNK", "40_chunk.txt")
SCR_ENCODING = os.getenv("CONJURER_ENCODING", "utf-8")
WORK_Q = Queue()
random.seed()
+7 -9
View File
@@ -19,22 +19,20 @@ Global Variables:
"""
# TODO: Wpiemdolić to wszystko w klasę z loggerem przysłanym z góry
import os
from queue import Empty, Queue
from threading import Thread
import time
q = Queue()
#TODO: Count number of lines in files and print to approximate on which part of the file search is
# DATA FOR TEST ONLY
# MAXTHREADS = 5
# DATABASE_PATH = r'C:\\Database\\chunks_1\\'
# Deployment data is environment-overridable so the local DOI database can live
# on a mounted volume (Docker/Linux) instead of the hardcoded Windows path.
MAXTHREADS = int(os.getenv("CONJURER_LIBRARIAN_MAXTHREADS", "41"))
DATABASE_PATH = os.getenv("CONJURER_LIBRARIAN_DB_PATH", r"C:\\Database\\chunks\\")
# DEPLOYMENT DATA
MAXTHREADS = 41
DATABASE_PATH = r"C:\\Database\\chunks\\"
ENCODING = "utf-8"
CHUNK = "_chunk.txt"
ENCODING = os.getenv("CONJURER_ENCODING", "utf-8")
CHUNK = os.getenv("CONJURER_LIBRARIAN_CHUNK", "_chunk.txt")
_sentinel = object()
WORK_Q_SIZE = 35500000