fix: make bot startup resilient (self-healing state, gated cogs, loud failures)

Container/native startup died silently on any missing file/service. Now:

constants.py - self-healing runtime layout at import:
- create missing dirs (log dir, LOGSTORE, GRAPHICS_PATH, MUSIC_FOLDER)
- seed missing state files from the repo templates shipped next to
  constants.py (settings/system_gpt/pamiec/pamiec_muzyki/accident_log),
  falling back to safe empty JSON; existing files are NEVER overwritten

bot.py:
- log to stdout too, so 'docker logs' finally shows the crash reason
- missing Discord token = loud sys.exit with mount/env instructions
  (was: silent return -> container crash-loop with empty logs)
- every cog loads independently (one broken cog = skipped with traceback,
  bot continues)
- musician/librarian cogs are health-gated: enabled only when the service
  answers HTTP; a watchdog re-checks every 5 min and enables them the
  moment the service comes alive (no restart needed); tree re-synced
- on_ready reconnects no longer re-load extensions

requirements_conan.txt + Dockerfile.bot: aiomcrcon (Python <=3.11 only)
moved to best-effort extras so the 3.13 image builds clean and the
conanjurer cog stays dormant without it.

DOCKER_PROXMOX.md: startup model (core vs gated cogs) + crash-loop
troubleshooting incl. the 'disappearing files' checklist (nothing in the
stack deletes host files; bind mount = live state).

Verified: fresh-volume seeding creates dirs+templates, existing files
untouched, missing-token exits with FATAL message, health-gating logic
(stub-based runpy tests).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bot.py

@@ -4,9 +4,21 @@
 # pylint: disable=too-many-lines
 """
 Module of a python bot named Conjurer - used to work on BDSM discord servers.
+
+Startup is defensive by design:
+
+* every cog is loaded independently - one broken/missing dependency disables
+  that cog only, never the whole bot,
+* cogs that need a sibling service (musician / librarian) are only enabled
+  after a positive health check; a watchdog keeps re-checking and enables them
+  the moment the service comes alive,
+* fatal misconfiguration (missing Discord token) exits loudly on stderr with a
+  clear message instead of dying silently into a log file.
 """
 import asyncio
 import logging
+import os
+import sys
 
 # *=========================================== Standard Library Imports
 import random
@@ -15,12 +27,26 @@ from logging import handlers
 
 # *==============Imported libraries
 import discord
+import requests
 from discord.ext import commands
 
 from communication_subroutine import comm_subroutine
-from constants import ENCODING, LOGFILE, TOKEN
+from constants import (
+    ENCODING,
+    FILE_SERVICE_ADDRESS,
+    GET_MP3,
+    LIBRARIAN_SERVICE_ADDRESS,
+    LOGFILE,
+    TOKEN,
+)
+
 logger = logging.getLogger("discord")
 logger.setLevel(logging.DEBUG)
+formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+
+# File log (rotated). constants ensures the directory exists; this is belt and
+# braces for exotic overrides.
+os.makedirs(os.path.dirname(LOGFILE) or ".", exist_ok=True)
 handler = handlers.RotatingFileHandler(
     filename=LOGFILE,
     encoding=ENCODING,
@@ -28,10 +54,15 @@ handler = handlers.RotatingFileHandler(
     maxBytes=6 * 1024 * 1024,
     backupCount=6,
     )
-formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
 handler.setFormatter(formatter)
 logger.addHandler(handler)
 
+# Console log so `docker logs` / journalctl actually show what happened.
+console_handler = logging.StreamHandler()
+console_handler.setLevel(logging.INFO)
+console_handler.setFormatter(formatter)
+logger.addHandler(console_handler)
+
 # *=========================================== Initializations
 intents = discord.Intents.default()
 intents.message_content = True
@@ -48,37 +79,129 @@ intents.moderation = True
 random.seed()
 client = commands.Bot(intents=intents, command_prefix="$")
 
+# *=========================================== Extension groups
+# Core cogs depend on nothing but the bot itself (broken ones are skipped
+# individually). Service groups are gated on a health check of the service
+# they talk to and enabled later by the watchdog when the service appears.
+CORE_EXTENSIONS = [
+    "administration_commands",
+    "ai_commands",
+    "other_commands",
+    "latex_commands",
+    "voice_recognition_commands",
+    "conanjurer_commands",
+]
+
+SERVICE_EXTENSION_GROUPS = {
+    # musician (file service): music download/search, radio control, file shares
+    "musician": {
+        "health_url": f"{FILE_SERVICE_ADDRESS}{GET_MP3}",
+        "extensions": ["music_commands", "radio_commands", "file_search_commands"],
+    },
+    # librarian: DOI / Crossref search
+    "librarian": {
+        "health_url": f"{LIBRARIAN_SERVICE_ADDRESS}/",
+        "extensions": ["librarian_commands"],
+    },
+}
+
+SERVICE_RECHECK_SECONDS = 300
+
+
+def _service_alive(url: str) -> bool:
+    """True when the service answers HTTP at all (any status code counts)."""
+    try:
+        requests.get(url, timeout=3)
+        return True
+    except requests.exceptions.RequestException:
+        return False
+
+
+async def _load_extension_safe(name: str) -> bool:
+    """Load one extension; log and continue instead of killing startup."""
+    if name in client.extensions:
+        return True
+    try:
+        await client.load_extension(name)
+        logger.info("Extension loaded: %s", name)
+        return True
+    except Exception:  # pylint: disable=broad-exception-caught
+        logger.exception("Extension FAILED (disabled, bot continues): %s", name)
+        return False
+
+
+async def _load_service_groups() -> bool:
+    """Health-check each service group and load its cogs when alive.
+
+    Returns True when anything new was loaded (caller may want to re-sync).
+    """
+    loaded_any = False
+    for service, group in SERVICE_EXTENSION_GROUPS.items():
+        missing = [e for e in group["extensions"] if e not in client.extensions]
+        if not missing:
+            continue
+        alive = await asyncio.to_thread(_service_alive, group["health_url"])
+        if not alive:
+            logger.warning(
+                "Service '%s' unreachable (%s) - cogs stay disabled: %s",
+                service,
+                group["health_url"],
+                ", ".join(missing),
+            )
+            continue
+        logger.info("Service '%s' is alive - enabling: %s", service, ", ".join(missing))
+        for extension in missing:
+            if await _load_extension_safe(extension):
+                loaded_any = True
+    return loaded_any
+
+
+async def _sync_tree() -> None:
+    try:
+        await client.tree.sync()
+    except Exception:  # pylint: disable=broad-exception-caught
+        logger.exception("Slash-command tree sync failed (commands may lag)")
+
+
+async def _service_watchdog() -> None:
+    """Periodically retry offline services and enable their cogs when up."""
+    while not client.is_closed():
+        await asyncio.sleep(SERVICE_RECHECK_SECONDS)
+        try:
+            if await _load_service_groups():
+                await _sync_tree()
+        except Exception:  # pylint: disable=broad-exception-caught
+            logger.exception("Service watchdog tick failed")
+
+
+_STARTUP_DONE = False
+
 
 # *=========================================== Define Events
 @client.event
 async def on_ready():
     """Metoda wywoływana przy połączeniu do serwera."""
-    logger = logging.getLogger("discord")
+    global _STARTUP_DONE  # pylint: disable=global-statement
-    logger.debug("SAMPLE DEBUG LOG")
     logger.info("%s has connected to Discord!", client.user)
-    # TODO: load vs reload
+    if _STARTUP_DONE:
+        logger.info("Reconnected - extensions already loaded")
+        return
+    _STARTUP_DONE = True
     logger.info("Reactor: online")
 
-    await client.load_extension("administration_commands")
+    for extension in CORE_EXTENSIONS:
+        await _load_extension_safe(extension)
 
-    await client.load_extension("librarian_commands")
+    await _load_service_groups()
-    await client.load_extension("music_commands")
-    await client.load_extension("radio_commands")
-
-    await client.load_extension("ai_commands")
-
-    await client.load_extension("other_commands")
-    await client.load_extension("voice_recognition_commands")
-    await client.load_extension("file_search_commands")
-    await client.load_extension("latex_commands")
-    await client.load_extension("conanjurer_commands")
     logger.info("Sensors: online")
 
     logger.info(client.cogs)
-    await client.tree.sync()
+    await _sync_tree()
     for com in client.commands:
         logger.info("Command %s is awejleble", com.qualified_name)
 
+    asyncio.create_task(_service_watchdog())
+
     logger.info("Logged in as ----> %s", client.user)
     logger.info("ID:%s ", client.user.id)
     logger.info("All systems: operational")
@@ -106,15 +229,10 @@ async def _run_bot(token: str, shutdown_event: asyncio.Event) -> None:
 
 
 async def main() -> None:
-    if not TOKEN:
-        logger.error("Discord token missing - set DISCORD_TOKEN or configure netrc")
-        return
-
     logger.info("Starting discord bot")
     shutdown_event = asyncio.Event()
     comm_stop_event = threading.Event()
 
-
     comm_task = asyncio.create_task(_run_comm_subroutine(comm_stop_event))
     bot_task = asyncio.create_task(_run_bot(TOKEN, shutdown_event))
 
@@ -133,4 +251,17 @@ async def main() -> None:
 
 # *================================== Run
 if __name__ == "__main__":
+    if not TOKEN:
+        # Loud, unmissable and in `docker logs`: this is THE most common cause
+        # of a silent container crash-loop.
+        MSG = (
+            "FATAL: Discord token missing.\n"
+            "Provide it via the DISCORD_TOKEN environment variable or a netrc "
+            "file (machine 'discord') at the path in CONJURER_NETRC_FILE.\n"
+            "Docker: check that your secrets mount exists, e.g.\n"
+            "  -v /srv/conjurer/secrets/.netrc:/secrets/.netrc:ro\n"
+            "  CONJURER_NETRC_FILE=/secrets/.netrc"
+        )
+        logger.critical(MSG)
+        sys.exit(MSG)
     asyncio.run(main())

constants.py

@@ -202,6 +202,65 @@ PORT_ADDRESS = int(os.getenv("CONJURER_DISCORD_PORT", "5000"))
 API_SHARED_KEY = os.getenv("CONJURER_API_KEY", "")
 
 
+# *=========================================== Self-healing runtime layout
+# A fresh host/volume must never kill the bot at import time. Missing
+# directories are created and missing state files are seeded - first from the
+# templates shipped alongside this file (repo checkout / docker image), then
+# from a safe empty structure. Existing files are never touched, so preserved
+# history always wins.
+
+_TEMPLATE_DIR = os.path.dirname(os.path.abspath(__file__))
+
+
+def _ensure_dir(path: str) -> None:
+    if not path:
+        return
+    try:
+        os.makedirs(path, exist_ok=True)
+    except OSError as exc:
+        logger.warning("Cannot create directory %s: %s", path, exc)
+
+
+def _seed_file(path: str, template_name: str, empty_content: str) -> None:
+    """Create *path* from the repo template (or *empty_content*) if missing."""
+    if not path or os.path.exists(path):
+        return
+    _ensure_dir(os.path.dirname(path) or ".")
+    template = os.path.join(_TEMPLATE_DIR, template_name)
+    try:
+        if os.path.exists(template) and os.path.abspath(template) != os.path.abspath(path):
+            import shutil
+
+            shutil.copyfile(template, path)
+            logger.warning("Seeded missing %s from repo template", path)
+        else:
+            with open(path, "w", encoding=ENCODING) as handle:
+                handle.write(empty_content)
+            logger.warning("Created missing %s as empty state", path)
+    except OSError as exc:
+        logger.warning("Cannot seed %s: %s", path, exc)
+
+
+def _ensure_runtime_layout() -> None:
+    for directory in (
+        os.path.dirname(LOGFILE) or ".",
+        LOGSTORE,
+        GRAPHICS_PATH,
+        MUSIC_FOLDER,
+    ):
+        _ensure_dir(directory)
+
+    # (target path, template shipped next to this file, empty fallback)
+    _seed_file(SETTINGS_FILE, "settings.json", "{}")
+    _seed_file(SYSTEM_GPT_SETTINGS, "system_gpt_settings.json", "{}")
+    _seed_file(MEMORY_FIVE_SIARA, "pamiec.json", "[]")
+    _seed_file(MEMORY_FIVE_MUZYKA, "pamiec_muzyki.json", "[]")
+    _seed_file(ACCIDENT_LOG, "accident_log.json", "[]")
+
+
+_ensure_runtime_layout()
+
+
 # *=========================================== Defensive state loading
 def _load_json(path: str, fallback):
     """Load JSON from *path*, falling back gracefully on missing/corrupt files."""

docker/Dockerfile.bot

@@ -29,9 +29,14 @@ RUN curl -fsSL https://drop-sh.fullyjustified.net | sh \
     && mv tectonic /usr/local/bin/tectonic \
     || echo "tectonic not installed - the LaTeX feature will be disabled"
 
-COPY requirements_bot.txt ./
+COPY requirements_bot.txt requirements_conan.txt ./
 RUN pip install --no-cache-dir --upgrade pip \
     && pip install --no-cache-dir -r requirements_bot.txt
+# Best-effort Conan bridge extras: aiomcrcon supports Python <= 3.11 only, so
+# on this 3.13 image the install fails harmlessly and the conanjurer cog stays
+# dormant (its imports are guarded).
+RUN pip install --no-cache-dir -r requirements_conan.txt \
+    || echo "conan extras skipped - conanjurer cog will stay dormant"
 
 # Vendored forks take import precedence over the pip packages of the same name,
 # because /app (the script dir) is first on sys.path when running `python bot.py`.

docs/deployment/DOCKER_PROXMOX.md

@@ -96,8 +96,47 @@ docker compose -f docker/compose.bot.yaml up -d --build
 docker logs -f conjurer-bot
 ```
 
-Look for `Loading … module done` for each cog, `Loading conanjurer commands
+Look for `Extension loaded: …` for each cog and `All systems: operational`.
-module done`, and `All systems: operational`.
+
+### 1e. Startup model: core cogs vs service-gated cogs
+
+The bot **always** starts with the cogs that depend on nothing but itself
+(administration, AI, other, latex, voice, conanjurer). Cogs that need a
+sibling service are **health-gated**:
+
+| Group | Cogs | Enabled when |
+|-------|------|--------------|
+| musician | `music_commands`, `radio_commands`, `file_search_commands` | `GET {FILE_SERVICE}/mp3` answers |
+| librarian | `librarian_commands` | librarian answers HTTP at all |
+
+When a service is down its cogs stay disabled (commands simply don't exist)
+and the log says so. A watchdog re-checks every 5 minutes and enables the
+cogs the moment the service starts answering — no bot restart needed.
+A single broken cog (missing pip package, bad import) is skipped with a full
+traceback in the log; it never takes the whole bot down.
+
+### 1f. Troubleshooting a crash-looping container
+
+`docker logs conjurer-bot` now shows the real reason (the bot logs to stdout
+as well as the rotating file). The most common cases:
+
+- **`FATAL: Discord token missing`** — the secrets mount is missing/empty or
+  `CONJURER_NETRC_FILE` points elsewhere. Check:
+  `docker inspect -f '{{json .Mounts}}' conjurer-bot | jq` and
+  `docker exec conjurer-bot ls -la /secrets/` (after a manual
+  `docker run … sleep infinity` if it crash-loops too fast).
+- **Missing state files** — not fatal anymore: missing dirs are created and
+  missing JSON state is seeded from the repo templates baked into the image
+  (existing files are never overwritten). Fix the mount at your leisure.
+
+**About files "disappearing" from `/srv/conjurer/...`:** nothing in this stack
+deletes host files — the entrypoint and the bot only ever *create* missing
+files. With a bind mount, `/srv/conjurer/data` **is** the live state (not an
+installation staging area): don't delete it after a successful install.
+If files vanished, the usual suspects are `docker compose down -v` (only
+affects *named* volumes, not binds), a re-provisioned VM, or copying the files
+to a different path than the one in the compose `volumes:` line — verify with
+`docker inspect -f '{{json .Mounts}}' conjurer-bot`.
 
 ---
 

requirements_conan.txt

@@ -0,0 +1,6 @@
+# Optional extras for the Conan Exiles bridge (conanjurer cog).
+#
+# aiomcrcon currently installs only on Python <= 3.11, while the main image
+# runs 3.13. Install is therefore best-effort: when it fails the conanjurer
+# cog simply stays dormant (its imports are guarded) and the bot runs fine.
+aiomcrcon