Librarian: 'still searching' heartbeat every 20 min

A deep scan runs for hours with nothing in the log between start and
finish, so it's impossible to tell a working search from a wedged one.
Every CONJURER_LIBRARIAN_HEARTBEAT_SECONDS (default 1200 = 20 min) a
running search now logs that it is still going, with its uuid, the search
phrase, hits so far, elapsed minutes, and a rough how-far-along.

The estimate is deliberately cheap: the producers ALREADY record a byte
offset per chunk file (the resume watermarks), and the total size is
stat()'d once per search when the chunk list is discovered. A reading is
then just a sum over ~40 ints - nothing extra happens per line, and no
cycles are spent estimating how many cycles are left.

search_for_doi takes an optional progress dict it fills with the live
positions dict + total_bytes; the librarian publishes the running search
(uuid/query/progress/live hits) while the scan runs and clears it in
finally. Nothing running => the heartbeat stays quiet.

Tests: percentage maths incl. unknown-total and >100% clamping, the
register/clear round-trip, and an end-to-end check that a real scan fills
progress so the offsets cover the chunk files on disk. Suite: 58 unit +
65 integration green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

conjurer_librarian/search_bot.py

@@ -247,7 +247,8 @@ def consumer(in_q, control_q, doi, live_results, result_list, control_dict, expe
 
 
 
-def search_for_doi(doi, live_results, _logger, stop_event=None, resume=None):
+def search_for_doi(doi, live_results, _logger, stop_event=None, resume=None,
+                   progress=None):
     """Search for DOI in live_results, resumably.
 
     Returns ``(result_list, positions, interrupted)``:
@@ -262,6 +263,11 @@ def search_for_doi(doi, live_results, _logger, stop_event=None, resume=None):
     ``resume`` is ``{"positions": {...}, "found": [doi, ...]}`` from a previous
     interrupted run: already-found DOIs are pre-marked and each producer seeks to
     its saved offset, so no already-scanned line is read twice.
+
+    ``progress``, if given, is a dict this fills with ``positions`` (the LIVE
+    dict, updated as producers read) and ``total_bytes`` (summed once, up front).
+    That makes a rough "how far along" reading free: sum the offsets, divide by
+    the total - no counting, no extra work in the read loop.
     """
     control_dict = {"sentinels":0}
     result_list = []
@@ -293,6 +299,19 @@ def search_for_doi(doi, live_results, _logger, stop_event=None, resume=None):
         )
         return result_list, positions, bool(stop_event and stop_event.is_set())
 
+    if progress is not None:
+        # One stat() per chunk file, ONCE - then progress is just sum(positions)
+        # / total_bytes, with nothing extra happening per line.
+        total_bytes = 0
+        for name in chunk_files:
+            try:
+                total_bytes += os.path.getsize(DATABASE_PATH + name)
+            except OSError:
+                pass
+        progress["positions"] = positions   # live dict, updated by the producers
+        progress["total_bytes"] = total_bytes
+        progress["chunk_files"] = expected
+
     for i in range (0, (len(doi)//1000)+2):
         t_cons = Thread(
             target=consumer,