Librarian: busy-aware ping + per-query lost-result watchdog

Two refinements to the librarian health/delivery story, matching how it
actually behaves under load:

1. Busy-aware ping (case b - broken return path). A ping arriving while
   the worker is grinding a search no longer queues behind it (which made
   a healthy-but-busy librarian time out and look dead). The librarian
   tracks worker_busy and, when set, pongs back IMMEDIATELY without
   touching the queue. Being busy is fine - you can keep piling searches
   on. The ping still travels the librarian->bot return path, so it keeps
   catching the one thing it must: a disrupted/incompatible return path
   where queries vanish. Idle pings still go through the internal queue.

2. Per-query watchdog (case a - finished but result lost). The librarian
   now tracks every search uuid's lifecycle (queued -> processing ->
   gone) in active_queries, exposed via a new POST /query_status. After
   dispatching a search the bot records it in self.pending; watch_pending
   polls /query_status for each. While the librarian still knows the uuid
   the search is progressing - left alone. The moment a uuid VANISHES
   there while still pending on the bot, its result was computed but never
   delivered: after a grace window (to rule out an in-flight result) the
   bot posts a notice to the channel - but ONLY then. A normally delivered
   result is popped from self.pending by check_data_q and never flagged.

Hardening: the worker's search body is now wrapped in try/except/finally
so a crashing search can't kill the worker thread (which would freeze the
queue), and worker_busy / active_queries are always cleared. The grace
logic lives in a dependency-free librarian_watchdog.pending_verdict so it
is unit-testable without discord/pdf libs. /ping and /query_status are
plain (sync) views so they run without flask[async].

Tests: unit test_librarian_watchdog (verdict transitions); integration
test_librarian_query_lifecycle (query_status known/unknown + auth,
idle-ping-queues, busy-ping-pongs-directly). Suite: 28 integration + 48
unit green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

librarian_commands.py

@@ -3,6 +3,7 @@ import io
 import logging
 import os
 import random
+import time
 import uuid
 from queue import Empty
 
@@ -15,15 +16,40 @@ from discord.ext import commands, tasks
 
 from ai_functions import handle_response
 from communication_subroutine import IN_COMM_Q, OUT_COMM_Q, QueryControl, submit_ai_query
-from constants import DIR_PATH_SADOX, LIBRARIAN_SERVICE_ADDRESS, SEND_QUERY, service_headers
+from constants import (
+    DIR_PATH_SADOX,
+    LIBRARIAN_SERVICE_ADDRESS,
+    QUERY_STATUS,
+    SEND_QUERY,
+    service_headers,
+)
+from librarian_watchdog import FLAG, pending_verdict
 
 SERVICE_HEADERS = service_headers()
 
+# Per-query watchdog tuning.
+PENDING_WATCH_SECONDS = 30            # how often to ask the librarian about a uuid
+PENDING_GRACE_SECONDS = 45           # unknown-but-pending must persist this long
+PENDING_HARD_TTL = 60 * 60 * 24 * 3  # drop tracking after 3 days no matter what
+
 
 class DataModule(commands.Cog):
     def __init__(self, bot, logger_name):
         self.bot = bot
         self.logger = logging.getLogger(logger_name)
+        # uuid -> {"ctx", "query", "created", "unknown_since"} for every search
+        # dispatched but not yet answered. watch_pending polls the librarian for
+        # each; check_data_q removes an entry the moment its result is rendered.
+        self.pending = {}
+
+    def _track_pending(self, query_uuid, query, ctx):
+        """Start watching a dispatched search so a lost result can be caught."""
+        self.pending[str(query_uuid)] = {
+            "ctx": ctx,
+            "query": query,
+            "created": time.monotonic(),
+            "unknown_since": None,
+        }
 
     @commands.hybrid_command(
         nsfw=True,
@@ -100,6 +126,9 @@ class DataModule(commands.Cog):
             fresh_data = IN_COMM_Q.get(block=False)
             entries = []
             if fresh_data.stop:
+                # The result arrived and is about to be rendered - stop the
+                # watchdog from ever flagging this uuid as lost.
+                self.pending.pop(str(fresh_data.uuid), None)
                 searcher = fresh_data.author
                 query = fresh_data.content
                 # ai_lines is a clean, plain rendering of the SAME list in the
@@ -175,6 +204,77 @@ class DataModule(commands.Cog):
         except Empty:
             pass
 
+    @tasks.loop(seconds=PENDING_WATCH_SECONDS)
+    async def watch_pending(self):
+        """Per-query safety net for lost results (case a).
+
+        For each dispatched-but-unanswered search, ask the librarian whether it
+        still knows the uuid (queued or processing). While it does, the search is
+        progressing - leave it alone (a busy librarian is fine). The moment a
+        uuid VANISHES on the librarian while still pending here, its result was
+        computed but never reached us: after a short grace window (to rule out a
+        result that is merely in flight) we tell the channel - but ONLY then.
+
+        A normally-delivered result is popped from self.pending by check_data_q,
+        so it never reaches the flag path.
+        """
+        now = time.monotonic()
+        for query_uuid in list(self.pending.keys()):
+            info = self.pending.get(query_uuid)
+            if info is None:
+                continue
+            # Hard cap so a permanently-unreachable librarian can't leak entries.
+            if now - info["created"] > PENDING_HARD_TTL:
+                self.logger.warning("Dropping stale pending query %s (hard TTL)", query_uuid)
+                self.pending.pop(query_uuid, None)
+                continue
+            try:
+                response = await asyncio.to_thread(
+                    requests.post,
+                    f"{LIBRARIAN_SERVICE_ADDRESS}{QUERY_STATUS}",
+                    json={"UUID": query_uuid},
+                    headers=SERVICE_HEADERS,
+                    timeout=5,
+                )
+                known = (
+                    response.status_code == 200
+                    and response.json().get("data", {}).get("known", False)
+                )
+            except (
+                requests.exceptions.RequestException,
+                ValueError,
+                AttributeError,
+                KeyError,
+                TypeError,
+            ) as exc:
+                # Librarian unreachable / garbled or unexpected answer: we can't
+                # judge, so don't cry wolf, and don't let one bad poll kill the
+                # loop. Reset the clock and try again next tick.
+                self.logger.info("Pending check for %s inconclusive: %s", query_uuid, exc)
+                info["unknown_since"] = None
+                continue
+            action, info["unknown_since"] = pending_verdict(
+                known, info["unknown_since"], now, PENDING_GRACE_SECONDS
+            )
+            # Re-check membership: the await above yields, so check_data_q may
+            # have just delivered (and popped) this result.
+            if action == FLAG and query_uuid in self.pending:
+                await self._flag_lost(query_uuid, info)
+                self.pending.pop(query_uuid, None)
+
+    async def _flag_lost(self, query_uuid, info):
+        """Tell the querent their finished search never made it back."""
+        message = (
+            "*Winda na książki z hukiem wraca z podziemi PUSTA. Z głośnika trzeszczy:* "
+            f"Twoje zapytanie {query_uuid} (\"{info['query']}\") przemieliło się w "
+            "bibliotece do końca, ale wynik przepadł gdzieś w drodze do baru - nic nie "
+            "dotarło. Zawołaj szefa albo puść jeszcze raz."
+        )
+        try:
+            await info["ctx"].send(message)
+        except Exception:  # pylint: disable=broad-exception-caught
+            self.logger.exception("Failed to post lost-result notice for %s", query_uuid)
+
     @commands.hybrid_command(
         name="wyszukaj_linki_do_dokumentow",
         description="Szuka linkow doi w bazie crossref i podaje linki do scihuba",
@@ -233,6 +333,7 @@ class DataModule(commands.Cog):
             username = ctx.message.author.name
         query_object = QueryControl(username, query_uuid, query, ctx)
         OUT_COMM_Q.put(query_object)
+        self._track_pending(query_uuid, query, ctx)
         await ctx.send(
             f"No dobra poszło. Zapamiętaj proszę {query_uuid} - to identyfikator twojego zapytania. Jesteś {queue_size} w kolejce."
             + " Zapytania obsługuje algorytm zasilany czterema chomikami zapierdalającymi w kołowrotku - więc wyniki najwcześniej za kilka godzi - ale mogą być też dni."
@@ -291,6 +392,7 @@ class DataModule(commands.Cog):
             username = ctx.message.author.name
         query_object = QueryControl(username, query_uuid, query, ctx, ai_review=True)
         OUT_COMM_Q.put(query_object)
+        self._track_pending(query_uuid, query, ctx)
         await ctx.send(
             f"Poszło z recenzją AI. Identyfikator: {query_uuid}. Jesteś {queue_size} w kolejce."
             + " Najpierw dojadą surowe wyniki, a zaraz po nich przesortowanie i recenzja od AI."
@@ -387,6 +489,7 @@ class DataModule(commands.Cog):
             username = ctx.message.author.name
         query_object = QueryControl(username, query_uuid, query, ctx)
         OUT_COMM_Q.put(query_object)
+        self._track_pending(query_uuid, query, ctx)
         await ctx.send(
             f"Wypij wypi {query_uuid} - to identyfikator twojego zapytania. Jesteś {queue_size} w kolejce niestety kolejka obowiazuje zawsze."
             + " Zapytania obsługuje algorytm zasilany czterema chomikami zapierdalającymi w kołowrotku - więc wyniki najwcześniej za kilka godzin - ale mogą być też dni. Głębokie zabawy trwają dłużej. *Znowu mruga*"
@@ -397,5 +500,6 @@ async def setup(bot):
     logger = logging.getLogger("discord")
     dm = DataModule(bot, "discord")
     dm.check_data_q.start()
+    dm.watch_pending.start()
     await bot.add_cog(dm)
     logger.info("Loading data sharing commands module done")