gitea
5d321f2f5b
CI / compile (pull_request) Successful in 10s
CI / unit (pull_request) Successful in 20s
CI / integration (pull_request) Successful in 20s
CI / compile (push) Successful in 10s
CI / unit (push) Successful in 21s
CI / integration (push) Successful in 20s
build / build (push) Successful in 57s
Two refinements to the librarian health/delivery story, matching how it actually behaves under load: 1. Busy-aware ping (case b - broken return path). A ping arriving while the worker is grinding a search no longer queues behind it (which made a healthy-but-busy librarian time out and look dead). The librarian tracks worker_busy and, when set, pongs back IMMEDIATELY without touching the queue. Being busy is fine - you can keep piling searches on. The ping still travels the librarian->bot return path, so it keeps catching the one thing it must: a disrupted/incompatible return path where queries vanish. Idle pings still go through the internal queue. 2. Per-query watchdog (case a - finished but result lost). The librarian now tracks every search uuid's lifecycle (queued -> processing -> gone) in active_queries, exposed via a new POST /query_status. After dispatching a search the bot records it in self.pending; watch_pending polls /query_status for each. While the librarian still knows the uuid the search is progressing - left alone. The moment a uuid VANISHES there while still pending on the bot, its result was computed but never delivered: after a grace window (to rule out an in-flight result) the bot posts a notice to the channel - but ONLY then. A normally delivered result is popped from self.pending by check_data_q and never flagged. Hardening: the worker's search body is now wrapped in try/except/finally so a crashing search can't kill the worker thread (which would freeze the queue), and worker_busy / active_queries are always cleared. The grace logic lives in a dependency-free librarian_watchdog.pending_verdict so it is unit-testable without discord/pdf libs. /ping and /query_status are plain (sync) views so they run without flask[async]. Tests: unit test_librarian_watchdog (verdict transitions); integration test_librarian_query_lifecycle (query_status known/unknown + auth, idle-ping-queues, busy-ping-pongs-directly). Suite: 28 integration + 48 unit green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
52 lines
1.7 KiB
Python
52 lines
1.7 KiB
Python
"""Unit tests for the per-query watchdog verdict logic.
|
|
|
|
The grace window is the whole point: a search that has just finished is briefly
|
|
'unknown' on the librarian while its result is still in flight, and we must NOT
|
|
flag that as lost. Only a uuid that stays unknown-but-pending past the grace
|
|
window is a genuinely lost result.
|
|
"""
|
|
from librarian_watchdog import FLAG, WAIT, pending_verdict
|
|
|
|
GRACE = 45
|
|
|
|
|
|
def test_known_resets_clock_and_waits():
|
|
action, unknown_since = pending_verdict(
|
|
known=True, unknown_since=100.0, now=200.0, grace_seconds=GRACE
|
|
)
|
|
assert action == WAIT
|
|
assert unknown_since is None # clock reset while it's still known
|
|
|
|
|
|
def test_first_unknown_starts_grace_but_does_not_flag():
|
|
action, unknown_since = pending_verdict(
|
|
known=False, unknown_since=None, now=1000.0, grace_seconds=GRACE
|
|
)
|
|
assert action == WAIT
|
|
assert unknown_since == 1000.0 # clock started now
|
|
|
|
|
|
def test_unknown_within_grace_keeps_waiting():
|
|
action, unknown_since = pending_verdict(
|
|
known=False, unknown_since=1000.0, now=1000.0 + GRACE - 1, grace_seconds=GRACE
|
|
)
|
|
assert action == WAIT
|
|
assert unknown_since == 1000.0 # unchanged, still counting
|
|
|
|
|
|
def test_unknown_past_grace_flags_lost():
|
|
action, unknown_since = pending_verdict(
|
|
known=False, unknown_since=1000.0, now=1000.0 + GRACE, grace_seconds=GRACE
|
|
)
|
|
assert action == FLAG
|
|
assert unknown_since == 1000.0
|
|
|
|
|
|
def test_recovered_to_known_after_being_unknown_resets():
|
|
# It reappeared (e.g. requeued / status flapped): do not flag, reset.
|
|
action, unknown_since = pending_verdict(
|
|
known=True, unknown_since=1000.0, now=1000.0 + GRACE + 10, grace_seconds=GRACE
|
|
)
|
|
assert action == WAIT
|
|
assert unknown_since is None
|