Durable result delivery: OUTBOX + idempotent INBOX so results never die

An 8h search result must survive a transient bot outage, an api/address
misroute, or a restart of either side. Make the librarian->bot result
path durably at-least-once with idempotent rendering:

Shared: durable_queue.DiskQueue - a dependency-free, atomically-written,
one-file-per-key disk queue (unit-tested), shared by both images
(added to Dockerfile.librarian; the bot already COPYs *.py).

Librarian (sender): finished results go to a persistent OUTBOX before
sending; delivery retries with backoff; an entry is removed only on a
positive ACK; a resender thread keeps flushing the OUTBOX, so a result
survives a bot outage AND a librarian restart (OUTBOX is on the state
volume) - it simply keeps trying until acked.

Bot (receiver): /conjurer is now idempotent and durable - each result is
persisted to an INBOX before acking and only queued if its uuid was not
already delivered (dropped as a duplicate) or already pending. Once the
cog actually renders it, mark_delivered() records the uuid and clears the
inbox, so the librarian's resends become no-ops. On startup the bot
replays any accepted-but-unrendered result from the INBOX, so a bot crash
mid-flight doesn't lose it. Pongs stay ephemeral.

Together: the librarian keeps a result until the bot confirms it; the bot
keeps it until it is on screen; duplicates never double-render. Combined
with the deploy return-path fix, an expensive result no longer vanishes.

Tests: unit test_durable_queue; integration test_librarian_outbox
(retry/backoff, resend survives outage) and test_result_durable_delivery
(persist, dedup pending, dedup delivered, replay, pong not persisted).
Suite: 55 unit + 39 integration green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

communication_subroutine.py

@@ -13,6 +13,9 @@ import requests
 from flask import Flask, abort, jsonify, request
 from waitress import serve
 
+from constants import DELIVERED_DIR, DELIVERED_MAX, RESULT_INBOX_DIR
+from durable_queue import DiskQueue
+
 HOST_ADDRESS = os.getenv("CONJURER_DISCORD_HOST", "192.168.1.92")
 PORT_ADDRESS = int(os.getenv("CONJURER_DISCORD_PORT", "5000"))
 ICECAST_ADDRESS = os.getenv("CONJURER_ICECAST", "http://192.168.1.12:8000")
@@ -33,9 +36,43 @@ incoming_q = Queue()
 # records older than this. Kept well above the ping timeout so a slow-but-alive
 # round-trip is never swept out from under a waiter.
 PING_TTL_SECONDS = 30
+
+# Durable result delivery. Every incoming search result is persisted to _inbox
+# before we ack the librarian, and only removed once it has actually been
+# rendered to the user (its uuid recorded in _delivered). This makes /conjurer
+# idempotent (the librarian can safely resend until acked) and lets an accepted
+# result survive a bot restart mid-flight (replayed from _inbox on startup).
+_inbox = DiskQueue(RESULT_INBOX_DIR)
+_delivered = DiskQueue(DELIVERED_DIR)
 app = Flask(__name__)
 
 
+def mark_delivered(query_uuid) -> None:
+    """Record that a result was rendered to the user.
+
+    After this, the librarian's resends of that uuid are dropped as duplicates
+    and it is never replayed from the inbox again. Called by the cog once it has
+    actually posted the result to Discord."""
+    _delivered.put(query_uuid, {})
+    _delivered.prune(DELIVERED_MAX)
+    _inbox.remove(query_uuid)
+
+
+def replay_inbox() -> None:
+    """Re-queue INBOX results that were accepted but not yet rendered.
+
+    Recovers an expensive result that reached the bot (and was acked to the
+    librarian, so it won't be resent) but whose render was lost to a bot restart.
+    """
+    logger = logging.getLogger("discord")
+    for query_uuid, payload, _ts in _inbox.items():
+        if _delivered.contains(query_uuid):
+            _inbox.remove(query_uuid)
+            continue
+        logger.info("Replaying un-rendered result %s from INBOX", query_uuid)
+        incoming_q.put(payload)
+
+
 def _authorize_request() -> None:
     """Reject inbound calls lacking the shared key (no-op if key is unset)."""
     if API_KEY and request.headers.get("X-Conjurer-Api-Key") != API_KEY:
@@ -113,11 +150,30 @@ def answer_external_command():
     """
     _authorize_request()
     logger = logging.getLogger("discord")
-    logger.info(request)
     record = json.loads(request.data)
-    logger.info(record)
-    logger.info("DATA RECEIVED")
-    incoming_q.put(record)
+    logger.info("DATA RECEIVED: %s", record)
+    # Health-check pongs are ephemeral - never persisted or deduped.
+    if isinstance(record, dict) and "__pong__" in record:
+        incoming_q.put(record)
+        return jsonify("SUCCESS")
+    # Search results: idempotent, durable intake. Persist each uuid to the inbox
+    # before acking, and queue only a uuid we have NOT already delivered or
+    # accepted. This lets the librarian's resender retry safely (a duplicate is
+    # dropped, never double-rendered) and lets an accepted-but-unrendered result
+    # be replayed after a bot restart.
+    if isinstance(record, dict):
+        for query_uuid in list(record.keys()):
+            if _delivered.contains(query_uuid):
+                logger.info("Result %s already delivered - dropping duplicate", query_uuid)
+                continue
+            if _inbox.contains(query_uuid):
+                logger.info("Result %s already pending - dropping duplicate", query_uuid)
+                continue
+            single = {query_uuid: record[query_uuid]}
+            _inbox.put(query_uuid, single)
+            incoming_q.put(single)
+    else:
+        incoming_q.put(record)  # unexpected shape - preserve old behaviour
     return jsonify("SUCCESS")
 
 
@@ -405,6 +461,13 @@ def comm_subroutine(stop_event: Optional[threading.Event] = None):
     for worker in threads:
         worker.start()
 
+    # Recover any result that was accepted before a previous shutdown but never
+    # rendered - re-queue it now that scan_incoming is running.
+    try:
+        replay_inbox()
+    except Exception:  # pylint: disable=broad-exception-caught
+        logger.exception("INBOX replay on startup failed")
+
     try:
         while any(thread.is_alive() for thread in threads):
             if stop_event and stop_event.is_set():