gitea
5d321f2f5b
CI / compile (pull_request) Successful in 10s
CI / unit (pull_request) Successful in 20s
CI / integration (pull_request) Successful in 20s
CI / compile (push) Successful in 10s
CI / unit (push) Successful in 21s
CI / integration (push) Successful in 20s
build / build (push) Successful in 57s
Two refinements to the librarian health/delivery story, matching how it actually behaves under load: 1. Busy-aware ping (case b - broken return path). A ping arriving while the worker is grinding a search no longer queues behind it (which made a healthy-but-busy librarian time out and look dead). The librarian tracks worker_busy and, when set, pongs back IMMEDIATELY without touching the queue. Being busy is fine - you can keep piling searches on. The ping still travels the librarian->bot return path, so it keeps catching the one thing it must: a disrupted/incompatible return path where queries vanish. Idle pings still go through the internal queue. 2. Per-query watchdog (case a - finished but result lost). The librarian now tracks every search uuid's lifecycle (queued -> processing -> gone) in active_queries, exposed via a new POST /query_status. After dispatching a search the bot records it in self.pending; watch_pending polls /query_status for each. While the librarian still knows the uuid the search is progressing - left alone. The moment a uuid VANISHES there while still pending on the bot, its result was computed but never delivered: after a grace window (to rule out an in-flight result) the bot posts a notice to the channel - but ONLY then. A normally delivered result is popped from self.pending by check_data_q and never flagged. Hardening: the worker's search body is now wrapped in try/except/finally so a crashing search can't kill the worker thread (which would freeze the queue), and worker_busy / active_queries are always cleared. The grace logic lives in a dependency-free librarian_watchdog.pending_verdict so it is unit-testable without discord/pdf libs. /ping and /query_status are plain (sync) views so they run without flask[async]. Tests: unit test_librarian_watchdog (verdict transitions); integration test_librarian_query_lifecycle (query_status known/unknown + auth, idle-ping-queues, busy-ping-pongs-directly). Suite: 28 integration + 48 unit green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
104 lines
3.4 KiB
Python
104 lines
3.4 KiB
Python
"""Integration: the librarian's health/liveness surface.
|
|
|
|
Two behaviours, both proven against the real Flask app:
|
|
|
|
* /ping is busy-aware - while a search is grinding it pongs back immediately
|
|
WITHOUT queueing (busy is healthy); when idle it routes the ping through the
|
|
internal queue for the worker to answer.
|
|
* /query_status reports whether a uuid is still known (queued/processing), which
|
|
is what the bot's per-query watchdog polls to catch a lost result.
|
|
|
|
Only the SYNC routes (/ping, /query_status) are exercised - the async /query
|
|
route needs flask[async], which the integration job doesn't install, so query
|
|
state is seeded directly on the module.
|
|
"""
|
|
import sys
|
|
import types
|
|
|
|
# conjurer_librarian does `from habanero import Crossref` at import time and
|
|
# habanero isn't installed in the integration job. Stub it before importing the
|
|
# service (we never build a real Librarian here, so Crossref is just a name).
|
|
if "habanero" not in sys.modules:
|
|
_habanero = types.ModuleType("habanero")
|
|
_habanero.Crossref = object
|
|
sys.modules["habanero"] = _habanero
|
|
|
|
import conjurer_librarian as lib # noqa: E402
|
|
|
|
|
|
def _client(key=None):
|
|
lib.API_KEY = key
|
|
return lib.app.test_client()
|
|
|
|
|
|
def _reset():
|
|
with lib._active_lock:
|
|
lib.active_queries.clear()
|
|
lib.worker_busy.clear()
|
|
while not lib.librarian_queue.empty():
|
|
lib.librarian_queue.get()
|
|
|
|
|
|
def test_query_status_known_vs_unknown():
|
|
_reset()
|
|
client = _client()
|
|
with lib._active_lock:
|
|
lib.active_queries["abc"] = "queued"
|
|
|
|
known = client.post("/query_status", json={"UUID": "abc"}).get_json()["data"]
|
|
assert known == {"uuid": "abc", "known": True, "state": "queued"}
|
|
|
|
unknown = client.post("/query_status", json={"UUID": "nope"}).get_json()["data"]
|
|
assert unknown == {"uuid": "nope", "known": False, "state": "unknown"}
|
|
|
|
|
|
def test_ping_idle_routes_through_internal_queue(monkeypatch):
|
|
_reset()
|
|
posted = []
|
|
monkeypatch.setattr(lib.requests, "post", lambda *a, **k: posted.append((a, k)))
|
|
client = _client()
|
|
|
|
resp = client.post("/ping", json={"UUID": "ping-idle"})
|
|
|
|
assert resp.status_code == 200
|
|
# Idle => it went onto the internal queue for the worker, NOT posted directly.
|
|
assert posted == []
|
|
assert lib.librarian_queue.get_nowait() == {"__ping__": "ping-idle"}
|
|
|
|
|
|
def test_ping_while_busy_pongs_directly_without_queue(monkeypatch):
|
|
_reset()
|
|
lib.worker_busy.set() # a search is grinding
|
|
posted = []
|
|
|
|
class _Resp:
|
|
status_code = 200
|
|
|
|
def fake_post(url, json=None, headers=None, timeout=None):
|
|
posted.append({"url": url, "json": json})
|
|
return _Resp()
|
|
|
|
monkeypatch.setattr(lib.requests, "post", fake_post)
|
|
client = _client()
|
|
|
|
resp = client.post("/ping", json={"UUID": "ping-busy"})
|
|
|
|
assert resp.status_code == 200
|
|
# Busy => direct pong back to the bot, and NOTHING queued (it would only wait
|
|
# behind the long search).
|
|
assert lib.librarian_queue.empty()
|
|
assert len(posted) == 1
|
|
assert posted[0]["json"] == {"__pong__": "ping-busy"}
|
|
assert posted[0]["url"].endswith(lib.SEND_RESULTS)
|
|
|
|
|
|
def test_query_status_enforces_api_key():
|
|
_reset()
|
|
client = _client(key="secret")
|
|
denied = client.post("/query_status", json={"UUID": "x"})
|
|
assert denied.status_code == 401
|
|
ok = client.post(
|
|
"/query_status", json={"UUID": "x"}, headers={"X-Conjurer-Api-Key": "secret"}
|
|
)
|
|
assert ok.status_code == 200
|