Durable result delivery: OUTBOX + idempotent INBOX so results never die
build / build (push) Successful in 46s
CI / compile (push) Successful in 10s
CI / unit (push) Successful in 20s
CI / integration (push) Successful in 27s

An 8h search result must survive a transient bot outage, an api/address
misroute, or a restart of either side. Make the librarian->bot result
path durably at-least-once with idempotent rendering:

Shared: durable_queue.DiskQueue - a dependency-free, atomically-written,
one-file-per-key disk queue (unit-tested), shared by both images
(added to Dockerfile.librarian; the bot already COPYs *.py).

Librarian (sender): finished results go to a persistent OUTBOX before
sending; delivery retries with backoff; an entry is removed only on a
positive ACK; a resender thread keeps flushing the OUTBOX, so a result
survives a bot outage AND a librarian restart (OUTBOX is on the state
volume) - it simply keeps trying until acked.

Bot (receiver): /conjurer is now idempotent and durable - each result is
persisted to an INBOX before acking and only queued if its uuid was not
already delivered (dropped as a duplicate) or already pending. Once the
cog actually renders it, mark_delivered() records the uuid and clears the
inbox, so the librarian's resends become no-ops. On startup the bot
replays any accepted-but-unrendered result from the INBOX, so a bot crash
mid-flight doesn't lose it. Pongs stay ephemeral.

Together: the librarian keeps a result until the bot confirms it; the bot
keeps it until it is on screen; duplicates never double-render. Combined
with the deploy return-path fix, an expensive result no longer vanishes.

Tests: unit test_durable_queue; integration test_librarian_outbox
(retry/backoff, resend survives outage) and test_result_durable_delivery
(persist, dedup pending, dedup delivered, replay, pong not persisted).
Suite: 55 unit + 39 integration green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit was merged in pull request #12.
This commit is contained in:
2026-08-02 17:17:17 +02:00
committed by gitea
parent 04070ea7f1
commit 44b7298a15
9 changed files with 557 additions and 42 deletions
+15
View File
@@ -230,6 +230,21 @@ MEMORY_COMPACT_THRESHOLD = int(os.getenv("CONJURER_MEMORY_COMPACT_THRESHOLD", "4
MEMORY_KEEP_RECENT = int(os.getenv("CONJURER_MEMORY_KEEP_RECENT", "200"))
MEMORY_COMPACT_HOURS = float(os.getenv("CONJURER_MEMORY_COMPACT_HOURS", "6"))
# Durable result-delivery spool (librarian -> bot). The bot persists every
# incoming search result to RESULT_INBOX_DIR before acking and only forgets it
# once rendered (uuid recorded in DELIVERED_DIR), so an expensive (hours-long)
# result survives a bot restart mid-flight and duplicate resends are idempotent.
# Rooted under CONJURER_DATA_DIR when set (a mounted volume), else next to the
# log file. DELIVERED_MAX bounds the remembered-uuid set.
_STATE_ROOT = _DATA_DIR or (os.path.dirname(LOGFILE) or ".")
RESULT_INBOX_DIR = os.getenv(
"CONJURER_RESULT_INBOX", os.path.join(_STATE_ROOT, "result_inbox")
)
DELIVERED_DIR = os.getenv(
"CONJURER_DELIVERED_DIR", os.path.join(_STATE_ROOT, "delivered_uuids")
)
DELIVERED_MAX = int(os.getenv("CONJURER_DELIVERED_MAX", "10000"))
FILE_SERVICE_ADDRESS = os.getenv("CONJURER_FILE_SERVICE", "http://192.168.1.15:5000")
RADIO_HARBOR_ADDRESS = os.getenv("CONJURER_RADIO_HARBOR", "http://192.168.1.15:54321")
# Betoniarka (radio-operator service colocated with Liquidsoap). Falls back to