Gate librarian cog on a full ping round-trip, not a bare GET

The librarian health check was a plain GET to '/', which only proved
Flask was listening - not that the service could actually take a query,
run it through its internal queue+worker, and answer back. So the cog
could load against a librarian whose worker was wedged or that couldn't
reach the bot on the return leg.

Replace it with a ping that travels the SAME path a real search does, on
both sides:
  bot: QueryControl -> OUT_COMM_Q -> scan_queue -> awaiting_q
  librarian: POST /ping -> librarian_queue -> worker pulls it off
             (no Crossref/DOI search) -> pongs back with the same uuid
  bot: /conjurer -> incoming_q -> scan_incoming matches uuid, wakes waiter
The cog enables only when that whole loop closes within 3s. This also
proves the librarian->bot return path, which a GET never did.

Safety: uuid is random per ping; the wait and POST are both bounded so
startup can't stall; a pong that finds no waiter is dropped (never
orphaned into IN_COMM_Q, which would make the cog post a bogus 'no
results' message); and a ping whose pong never returns is swept out of
awaiting_q after PING_TTL_SECONDS so nothing leaks. All awaiting_q writes
stay within scan_queue (append) and scan_incoming (remove) - no locks,
no cross-thread mutation.

Integration tests cover: OK round-trip, timeout when accepted-but-no-pong,
unreachable, non-200, orphan-pong-dropped, and that real results still
reach IN_COMM_Q. Suite: 24 integration + 41 unit green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

conjurer_librarian/conjurer_librarian.py

@@ -411,7 +411,30 @@ class BackgroundTaskSearch(threading.Thread):
         while True:
             database = None
             ndb_database = None
-            librarian = librarian_queue.get()
+            item = librarian_queue.get()
+            # Health-check ping: it has flowed through the internal queue and is
+            # now pulled off it - that is the whole point. Pong it straight back
+            # with the same uuid and DO NOT run a search.
+            if isinstance(item, dict) and "__ping__" in item:
+                ping_uuid = item["__ping__"]
+                self.app.logger.info(
+                    "PING %s pulled off internal queue - ponging back (no search)",
+                    ping_uuid,
+                )
+                try:
+                    await asyncio.to_thread(
+                        requests.post,
+                        f"{MAIN_BOT_ADDRESS}{SEND_RESULTS}",
+                        json={"__pong__": ping_uuid},
+                        headers=_service_headers(),
+                        timeout=5,
+                    )
+                except requests.exceptions.RequestException as exc:
+                    self.app.logger.warning(
+                        "PING pong send failed for %s: %s", ping_uuid, exc
+                    )
+                continue
+            librarian = item
             self.app.logger.info("STARTED")
             result = await librarian.answer_query(librarian.deep_search)
             result = {librarian.uuid: result}
@@ -530,6 +553,29 @@ async def query_database():
     return return_data
 
 
+@app.route("/ping", methods=["POST"])
+async def ping_roundtrip():
+    _authorize_request()
+    """
+    Health-check round-trip.
+
+    Puts a lightweight ping marker onto the SAME internal ``librarian_queue``
+    that real searches go through and returns 200 immediately. The background
+    worker pulls it off the queue and pongs it back to the bot with the same
+    uuid, WITHOUT running any Crossref/DOI search. A successful pong therefore
+    proves the whole pipeline (HTTP in -> internal queue -> worker -> HTTP out)
+    is flowing, not just that Flask is up.
+    """
+    record = json.loads(request.data)
+    ping_uuid = record["UUID"]
+    app.logger.info("PING received %s - queued for round-trip", ping_uuid)
+    librarian_queue.put({"__ping__": ping_uuid})
+    return (
+        jsonify(isError=False, message="ping-queued", statusCode=200, data=ping_uuid),
+        200,
+    )
+
+
 @app.route("/get_partial_result", methods=["POST"])
 async def get_partial():
     _authorize_request()