librarian: tolerate bad bytes in chunks; log what the result-send does

Two field-reported robustness gaps on top of the hang fix.

1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised
UnicodeDecodeError from readline() - which is a ValueError, so the earlier
`except OSError` did NOT catch it. The finally-sentinel meant no hang, but the
producer died mid-file with a loud traceback and every DOI after the bad byte
went unsearched. Now chunks are opened with errors="replace" (bad bytes become
U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF,
and the producer's except is broadened from OSError to Exception so no per-file
error can ever crash the thread - it's logged and the sentinel still fires.

2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly
what goes out - target URL, uuid, DOI count and the DOI list - so the librarian
log plainly shows a result was sent and what was in it. And a failed POST is no
longer fatal: a RequestException used to propagate out of the worker loop and
kill the thread, stalling every future query until restart; it's now caught and
logged, and a non-200 from the bot is logged as a warning.

Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96
byte before a valid DOI and asserting that DOI is still found (file read to
completion, not aborted). All 5 search_bot unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

conjurer_librarian/search_bot.py

@@ -90,7 +90,12 @@ def producer(out_q, control_q, filename, _logger):
         _logger: Logger object for logging.
     """
     try:
-        with open(DATABASE_PATH + filename, "r", encoding=ENCODING) as operated_file:
+        # errors="replace" so a stray non-UTF-8 byte in a chunk (they happen in
+        # scraped DOI dumps) becomes U+FFFD instead of raising UnicodeDecodeError
+        # mid-file. Without it the readline() below would blow up, killing the
+        # producer partway and leaving every DOI after the bad byte unsearched.
+        # DOIs are ASCII, so a replaced byte can only affect junk, never a match.
+        with open(DATABASE_PATH + filename, "r", encoding=ENCODING, errors="replace") as operated_file:
             print(f"Worker {filename} ")
             line_no = 0
             while True:
@@ -113,11 +118,13 @@ def producer(out_q, control_q, filename, _logger):
                     control_q.put(check)
                     break
             print(f"Worker finished: {filename}")
-    except OSError as exc:
-        # A missing or unreadable chunk must not take the whole search down with
-        # it - log and move on. The sentinel below still fires (finally), so the
-        # consumers' count stays correct and nothing deadlocks.
-        _logger.warning("Chunk %s unreadable, skipping: %s", filename, exc)
+    except Exception as exc:  # pylint: disable=broad-except
+        # No per-file error (missing/unreadable chunk, a decode edge case that
+        # slips past errors="replace", anything unforeseen) may take the whole
+        # search down or crash the thread with a traceback. Log it and move on;
+        # the sentinel below still fires (finally), so the consumers' count stays
+        # correct and nothing deadlocks or silently loses a producer.
+        _logger.warning("Chunk %s failed, skipping rest of it: %s", filename, exc)
         print(f"Worker {filename} failed: {exc}")
     finally:
         # ALWAYS emit exactly one sentinel per producer, on every exit path (EOF,