Durable result delivery: OUTBOX + idempotent INBOX so results never die
build / build (push) Successful in 46s
CI / compile (push) Successful in 10s
CI / unit (push) Successful in 20s
CI / integration (push) Successful in 27s

An 8h search result must survive a transient bot outage, an api/address
misroute, or a restart of either side. Make the librarian->bot result
path durably at-least-once with idempotent rendering:

Shared: durable_queue.DiskQueue - a dependency-free, atomically-written,
one-file-per-key disk queue (unit-tested), shared by both images
(added to Dockerfile.librarian; the bot already COPYs *.py).

Librarian (sender): finished results go to a persistent OUTBOX before
sending; delivery retries with backoff; an entry is removed only on a
positive ACK; a resender thread keeps flushing the OUTBOX, so a result
survives a bot outage AND a librarian restart (OUTBOX is on the state
volume) - it simply keeps trying until acked.

Bot (receiver): /conjurer is now idempotent and durable - each result is
persisted to an INBOX before acking and only queued if its uuid was not
already delivered (dropped as a duplicate) or already pending. Once the
cog actually renders it, mark_delivered() records the uuid and clears the
inbox, so the librarian's resends become no-ops. On startup the bot
replays any accepted-but-unrendered result from the INBOX, so a bot crash
mid-flight doesn't lose it. Pongs stay ephemeral.

Together: the librarian keeps a result until the bot confirms it; the bot
keeps it until it is on screen; duplicates never double-render. Combined
with the deploy return-path fix, an expensive result no longer vanishes.

Tests: unit test_durable_queue; integration test_librarian_outbox
(retry/backoff, resend survives outage) and test_result_durable_delivery
(persist, dedup pending, dedup delivered, replay, pong not persisted).
Suite: 55 unit + 39 integration green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit was merged in pull request #12.
This commit is contained in:
2026-08-02 17:17:17 +02:00
committed by gitea
parent 04070ea7f1
commit 44b7298a15
9 changed files with 557 additions and 42 deletions
+13 -1
View File
@@ -15,7 +15,13 @@ import requests
from discord.ext import commands, tasks
from ai_functions import handle_response
from communication_subroutine import IN_COMM_Q, OUT_COMM_Q, QueryControl, submit_ai_query
from communication_subroutine import (
IN_COMM_Q,
OUT_COMM_Q,
QueryControl,
mark_delivered,
submit_ai_query,
)
from constants import (
DIR_PATH_SADOX,
LIBRARIAN_SERVICE_ADDRESS,
@@ -169,6 +175,12 @@ class DataModule(commands.Cog):
await ctx.send(message)
message = ""
# The result is now on screen: mark it delivered so the
# librarian's resends become no-ops and it is dropped from the
# durable inbox (never replayed again). Done after the core
# render but before the optional AI review, which is a bonus.
mark_delivered(str(fresh_data.uuid))
# Optional AI pass: re-rank the (already Crossref-relevance-
# sorted) DOI list and review the sources. Enqueued to the AI
# worker so it runs on whatever backend $gadaj_teraz selected;