Librarian: busy-aware ping + per-query lost-result watchdog

Two refinements to the librarian health/delivery story, matching how it
actually behaves under load:

1. Busy-aware ping (case b - broken return path). A ping arriving while
   the worker is grinding a search no longer queues behind it (which made
   a healthy-but-busy librarian time out and look dead). The librarian
   tracks worker_busy and, when set, pongs back IMMEDIATELY without
   touching the queue. Being busy is fine - you can keep piling searches
   on. The ping still travels the librarian->bot return path, so it keeps
   catching the one thing it must: a disrupted/incompatible return path
   where queries vanish. Idle pings still go through the internal queue.

2. Per-query watchdog (case a - finished but result lost). The librarian
   now tracks every search uuid's lifecycle (queued -> processing ->
   gone) in active_queries, exposed via a new POST /query_status. After
   dispatching a search the bot records it in self.pending; watch_pending
   polls /query_status for each. While the librarian still knows the uuid
   the search is progressing - left alone. The moment a uuid VANISHES
   there while still pending on the bot, its result was computed but never
   delivered: after a grace window (to rule out an in-flight result) the
   bot posts a notice to the channel - but ONLY then. A normally delivered
   result is popped from self.pending by check_data_q and never flagged.

Hardening: the worker's search body is now wrapped in try/except/finally
so a crashing search can't kill the worker thread (which would freeze the
queue), and worker_busy / active_queries are always cleared. The grace
logic lives in a dependency-free librarian_watchdog.pending_verdict so it
is unit-testable without discord/pdf libs. /ping and /query_status are
plain (sync) views so they run without flask[async].

Tests: unit test_librarian_watchdog (verdict transitions); integration
test_librarian_query_lifecycle (query_status known/unknown + auth,
idle-ping-queues, busy-ping-pongs-directly). Suite: 28 integration + 48
unit green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests/integration/test_librarian_query_lifecycle.py

@@ -0,0 +1,103 @@
+"""Integration: the librarian's health/liveness surface.
+
+Two behaviours, both proven against the real Flask app:
+
+* /ping is busy-aware - while a search is grinding it pongs back immediately
+  WITHOUT queueing (busy is healthy); when idle it routes the ping through the
+  internal queue for the worker to answer.
+* /query_status reports whether a uuid is still known (queued/processing), which
+  is what the bot's per-query watchdog polls to catch a lost result.
+
+Only the SYNC routes (/ping, /query_status) are exercised - the async /query
+route needs flask[async], which the integration job doesn't install, so query
+state is seeded directly on the module.
+"""
+import sys
+import types
+
+# conjurer_librarian does `from habanero import Crossref` at import time and
+# habanero isn't installed in the integration job. Stub it before importing the
+# service (we never build a real Librarian here, so Crossref is just a name).
+if "habanero" not in sys.modules:
+    _habanero = types.ModuleType("habanero")
+    _habanero.Crossref = object
+    sys.modules["habanero"] = _habanero
+
+import conjurer_librarian as lib  # noqa: E402
+
+
+def _client(key=None):
+    lib.API_KEY = key
+    return lib.app.test_client()
+
+
+def _reset():
+    with lib._active_lock:
+        lib.active_queries.clear()
+    lib.worker_busy.clear()
+    while not lib.librarian_queue.empty():
+        lib.librarian_queue.get()
+
+
+def test_query_status_known_vs_unknown():
+    _reset()
+    client = _client()
+    with lib._active_lock:
+        lib.active_queries["abc"] = "queued"
+
+    known = client.post("/query_status", json={"UUID": "abc"}).get_json()["data"]
+    assert known == {"uuid": "abc", "known": True, "state": "queued"}
+
+    unknown = client.post("/query_status", json={"UUID": "nope"}).get_json()["data"]
+    assert unknown == {"uuid": "nope", "known": False, "state": "unknown"}
+
+
+def test_ping_idle_routes_through_internal_queue(monkeypatch):
+    _reset()
+    posted = []
+    monkeypatch.setattr(lib.requests, "post", lambda *a, **k: posted.append((a, k)))
+    client = _client()
+
+    resp = client.post("/ping", json={"UUID": "ping-idle"})
+
+    assert resp.status_code == 200
+    # Idle => it went onto the internal queue for the worker, NOT posted directly.
+    assert posted == []
+    assert lib.librarian_queue.get_nowait() == {"__ping__": "ping-idle"}
+
+
+def test_ping_while_busy_pongs_directly_without_queue(monkeypatch):
+    _reset()
+    lib.worker_busy.set()  # a search is grinding
+    posted = []
+
+    class _Resp:
+        status_code = 200
+
+    def fake_post(url, json=None, headers=None, timeout=None):
+        posted.append({"url": url, "json": json})
+        return _Resp()
+
+    monkeypatch.setattr(lib.requests, "post", fake_post)
+    client = _client()
+
+    resp = client.post("/ping", json={"UUID": "ping-busy"})
+
+    assert resp.status_code == 200
+    # Busy => direct pong back to the bot, and NOTHING queued (it would only wait
+    # behind the long search).
+    assert lib.librarian_queue.empty()
+    assert len(posted) == 1
+    assert posted[0]["json"] == {"__pong__": "ping-busy"}
+    assert posted[0]["url"].endswith(lib.SEND_RESULTS)
+
+
+def test_query_status_enforces_api_key():
+    _reset()
+    client = _client(key="secret")
+    denied = client.post("/query_status", json={"UUID": "x"})
+    assert denied.status_code == 401
+    ok = client.post(
+        "/query_status", json={"UUID": "x"}, headers={"X-Conjurer-Api-Key": "secret"}
+    )
+    assert ok.status_code == 200