Durable result delivery: OUTBOX + idempotent INBOX so results never die
An 8h search result must survive a transient bot outage, an api/address misroute, or a restart of either side. Make the librarian->bot result path durably at-least-once with idempotent rendering: Shared: durable_queue.DiskQueue - a dependency-free, atomically-written, one-file-per-key disk queue (unit-tested), shared by both images (added to Dockerfile.librarian; the bot already COPYs *.py). Librarian (sender): finished results go to a persistent OUTBOX before sending; delivery retries with backoff; an entry is removed only on a positive ACK; a resender thread keeps flushing the OUTBOX, so a result survives a bot outage AND a librarian restart (OUTBOX is on the state volume) - it simply keeps trying until acked. Bot (receiver): /conjurer is now idempotent and durable - each result is persisted to an INBOX before acking and only queued if its uuid was not already delivered (dropped as a duplicate) or already pending. Once the cog actually renders it, mark_delivered() records the uuid and clears the inbox, so the librarian's resends become no-ops. On startup the bot replays any accepted-but-unrendered result from the INBOX, so a bot crash mid-flight doesn't lose it. Pongs stay ephemeral. Together: the librarian keeps a result until the bot confirms it; the bot keeps it until it is on screen; duplicates never double-render. Combined with the deploy return-path fix, an expensive result no longer vanishes. Tests: unit test_durable_queue; integration test_librarian_outbox (retry/backoff, resend survives outage) and test_result_durable_delivery (persist, dedup pending, dedup delivered, replay, pong not persisted). Suite: 55 unit + 39 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit was merged in pull request #12.
This commit is contained in:
@@ -230,6 +230,21 @@ MEMORY_COMPACT_THRESHOLD = int(os.getenv("CONJURER_MEMORY_COMPACT_THRESHOLD", "4
|
||||
MEMORY_KEEP_RECENT = int(os.getenv("CONJURER_MEMORY_KEEP_RECENT", "200"))
|
||||
MEMORY_COMPACT_HOURS = float(os.getenv("CONJURER_MEMORY_COMPACT_HOURS", "6"))
|
||||
|
||||
# Durable result-delivery spool (librarian -> bot). The bot persists every
|
||||
# incoming search result to RESULT_INBOX_DIR before acking and only forgets it
|
||||
# once rendered (uuid recorded in DELIVERED_DIR), so an expensive (hours-long)
|
||||
# result survives a bot restart mid-flight and duplicate resends are idempotent.
|
||||
# Rooted under CONJURER_DATA_DIR when set (a mounted volume), else next to the
|
||||
# log file. DELIVERED_MAX bounds the remembered-uuid set.
|
||||
_STATE_ROOT = _DATA_DIR or (os.path.dirname(LOGFILE) or ".")
|
||||
RESULT_INBOX_DIR = os.getenv(
|
||||
"CONJURER_RESULT_INBOX", os.path.join(_STATE_ROOT, "result_inbox")
|
||||
)
|
||||
DELIVERED_DIR = os.getenv(
|
||||
"CONJURER_DELIVERED_DIR", os.path.join(_STATE_ROOT, "delivered_uuids")
|
||||
)
|
||||
DELIVERED_MAX = int(os.getenv("CONJURER_DELIVERED_MAX", "10000"))
|
||||
|
||||
FILE_SERVICE_ADDRESS = os.getenv("CONJURER_FILE_SERVICE", "http://192.168.1.15:5000")
|
||||
RADIO_HARBOR_ADDRESS = os.getenv("CONJURER_RADIO_HARBOR", "http://192.168.1.15:54321")
|
||||
# Betoniarka (radio-operator service colocated with Liquidsoap). Falls back to
|
||||
|
||||
Reference in New Issue
Block a user