gitea
44b7298a15
An 8h search result must survive a transient bot outage, an api/address misroute, or a restart of either side. Make the librarian->bot result path durably at-least-once with idempotent rendering: Shared: durable_queue.DiskQueue - a dependency-free, atomically-written, one-file-per-key disk queue (unit-tested), shared by both images (added to Dockerfile.librarian; the bot already COPYs *.py). Librarian (sender): finished results go to a persistent OUTBOX before sending; delivery retries with backoff; an entry is removed only on a positive ACK; a resender thread keeps flushing the OUTBOX, so a result survives a bot outage AND a librarian restart (OUTBOX is on the state volume) - it simply keeps trying until acked. Bot (receiver): /conjurer is now idempotent and durable - each result is persisted to an INBOX before acking and only queued if its uuid was not already delivered (dropped as a duplicate) or already pending. Once the cog actually renders it, mark_delivered() records the uuid and clears the inbox, so the librarian's resends become no-ops. On startup the bot replays any accepted-but-unrendered result from the INBOX, so a bot crash mid-flight doesn't lose it. Pongs stay ephemeral. Together: the librarian keeps a result until the bot confirms it; the bot keeps it until it is on screen; duplicates never double-render. Combined with the deploy return-path fix, an expensive result no longer vanishes. Tests: unit test_durable_queue; integration test_librarian_outbox (retry/backoff, resend survives outage) and test_result_durable_delivery (persist, dedup pending, dedup delivered, replay, pong not persisted). Suite: 55 unit + 39 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
34 lines
1.2 KiB
Docker
34 lines
1.2 KiB
Docker
# Conjurer librarian (Crossref search + local DOI database lookup).
|
|
# Build from the repository root:
|
|
# docker build -f docker/Dockerfile.librarian -t conjurer-librarian .
|
|
FROM python:3.11-slim
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
build-essential \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
WORKDIR /app
|
|
|
|
COPY conjurer_librarian/requirements_librarian.txt ./
|
|
RUN pip install --no-cache-dir --upgrade pip \
|
|
&& pip install --no-cache-dir -r requirements_librarian.txt requests
|
|
|
|
COPY conjurer_librarian/ ./
|
|
# durable_queue lives at the repo root and is shared with the bot; the librarian
|
|
# imports it for the durable result OUTBOX.
|
|
COPY durable_queue.py ./
|
|
|
|
ENV PYTHONUNBUFFERED=1 \
|
|
CONJURER_LIBRARIAN_HOST=0.0.0.0 \
|
|
CONJURER_LIBRARIAN_PORT=5001 \
|
|
CONJURER_LIBRARIAN_DB_PATH=/doi/ \
|
|
CONJURER_LIBRARIAN_STATE_DIR=/lib_temp_files
|
|
|
|
# /doi = the local DOI chunk database (large, read-only).
|
|
# /lib_temp_files = runtime JSON state (cr_results/rr_results/not_in_db/
|
|
# s_results) - seeded on first run, persisted across restarts.
|
|
VOLUME ["/doi", "/lib_temp_files"]
|
|
EXPOSE 5001
|
|
|
|
CMD ["python", "conjurer_librarian.py"]
|