gitea
ce18b386d3
Two refinements to the librarian health/delivery story, matching how it actually behaves under load: 1. Busy-aware ping (case b - broken return path). A ping arriving while the worker is grinding a search no longer queues behind it (which made a healthy-but-busy librarian time out and look dead). The librarian tracks worker_busy and, when set, pongs back IMMEDIATELY without touching the queue. Being busy is fine - you can keep piling searches on. The ping still travels the librarian->bot return path, so it keeps catching the one thing it must: a disrupted/incompatible return path where queries vanish. Idle pings still go through the internal queue. 2. Per-query watchdog (case a - finished but result lost). The librarian now tracks every search uuid's lifecycle (queued -> processing -> gone) in active_queries, exposed via a new POST /query_status. After dispatching a search the bot records it in self.pending; watch_pending polls /query_status for each. While the librarian still knows the uuid the search is progressing - left alone. The moment a uuid VANISHES there while still pending on the bot, its result was computed but never delivered: after a grace window (to rule out an in-flight result) the bot posts a notice to the channel - but ONLY then. A normally delivered result is popped from self.pending by check_data_q and never flagged. Hardening: the worker's search body is now wrapped in try/except/finally so a crashing search can't kill the worker thread (which would freeze the queue), and worker_busy / active_queries are always cleared. The grace logic lives in a dependency-free librarian_watchdog.pending_verdict so it is unit-testable without discord/pdf libs. /ping and /query_status are plain (sync) views so they run without flask[async]. Tests: unit test_librarian_watchdog (verdict transitions); integration test_librarian_query_lifecycle (query_status known/unknown + auth, idle-ping-queues, busy-ping-pongs-directly). Suite: 28 integration + 48 unit green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
52 lines
2.2 KiB
Python
52 lines
2.2 KiB
Python
"""Pure decision logic for the librarian per-query watchdog.
|
|
|
|
Split out of ``librarian_commands`` (which pulls in discord / pdf libs, so it is
|
|
not importable in the pytest-only unit job) so the one subtle part - the grace
|
|
window that stops a just-delivered result from being falsely flagged as lost -
|
|
can be unit-tested in isolation.
|
|
|
|
The watchdog polls the librarian's /query_status for each dispatched query:
|
|
|
|
* the librarian reports the uuid ``known`` while it is queued or processing,
|
|
* once the search finishes it is dropped there, so the uuid goes ``unknown``.
|
|
|
|
A result that arrives normally is removed from the pending set by the result
|
|
handler, so the watchdog never even sees it. Only a uuid that goes ``unknown``
|
|
on the librarian *and is still pending on the bot* is a lost result - but we
|
|
require it to stay that way for a grace window first, because there is always a
|
|
brief moment where the librarian has finished (uuid gone) yet the result is
|
|
still in flight / not yet rendered.
|
|
"""
|
|
|
|
WAIT = "wait"
|
|
FLAG = "flag"
|
|
|
|
|
|
def pending_verdict(known, unknown_since, now, grace_seconds):
|
|
"""Decide what to do this tick for one pending query.
|
|
|
|
Args:
|
|
known: did the librarian report the uuid as still known this tick?
|
|
unknown_since: monotonic timestamp the uuid was first seen unknown, or
|
|
None if it was known last tick.
|
|
now: current monotonic time.
|
|
grace_seconds: how long a uuid must stay unknown-but-pending before it
|
|
is declared lost.
|
|
|
|
Returns:
|
|
(action, unknown_since) where action is WAIT or FLAG and the returned
|
|
``unknown_since`` is what the caller should store for the next tick.
|
|
"""
|
|
if known:
|
|
# Still queued/processing (or freshly back to known) - reset the clock.
|
|
return WAIT, None
|
|
if unknown_since is None:
|
|
# First tick we see it gone: start the grace clock, don't flag yet - the
|
|
# result may simply be in flight.
|
|
return WAIT, now
|
|
if now - unknown_since >= grace_seconds:
|
|
# Gone for the whole grace window and still pending: the result was lost.
|
|
return FLAG, unknown_since
|
|
# Gone, but not long enough yet - keep waiting.
|
|
return WAIT, unknown_since
|