librarian: tolerate bad bytes in chunks; log what the result-send does

Two field-reported robustness gaps on top of the hang fix.

1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised
UnicodeDecodeError from readline() - which is a ValueError, so the earlier
`except OSError` did NOT catch it. The finally-sentinel meant no hang, but the
producer died mid-file with a loud traceback and every DOI after the bad byte
went unsearched. Now chunks are opened with errors="replace" (bad bytes become
U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF,
and the producer's except is broadened from OSError to Exception so no per-file
error can ever crash the thread - it's logged and the sentinel still fires.

2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly
what goes out - target URL, uuid, DOI count and the DOI list - so the librarian
log plainly shows a result was sent and what was in it. And a failed POST is no
longer fatal: a RequestException used to propagate out of the worker loop and
kill the thread, stalling every future query until restart; it's now caught and
logged, and a non-200 from the bot is logged as a warning.

Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96
byte before a valid DOI and asserting that DOI is still found (file read to
completion, not aborted). All 5 search_bot unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests/unit/test_search_bot.py

@@ -77,6 +77,23 @@ def test_no_chunks_returns_immediately(tmp_path, monkeypatch):
     assert result == [{"DOI": "10.0/x", "exists": False, "data": "DATA"}]
 
 
+def test_survives_invalid_utf8_byte_and_still_finds_later_doi(tmp_path, monkeypatch):
+    # A chunk with a stray non-UTF-8 byte (0x96, the one from the field report)
+    # must not crash the producer or abort the file mid-read: DOIs AFTER the bad
+    # byte still have to be found.
+    monkeypatch.setattr(search_bot, "DATABASE_PATH", str(tmp_path) + "/")
+    target = "10.1234/after.the.bad.byte"
+    (tmp_path / "0_chunk.txt").write_bytes(
+        b"10.0000/before\n" + b"\x96 broken \x96 line \x96\n" + target.encode() + b"\n"
+    )
+
+    finished, result = _run_bounded([(target, "DATA")])
+
+    assert finished, "an invalid UTF-8 byte hung or crashed the search"
+    hit = [r for r in result if r["DOI"] == target and r["exists"]]
+    assert hit, "DOI after the bad byte was not found - the file was aborted mid-read"
+
+
 def test_discover_chunk_files_sorted_numerically(tmp_path, monkeypatch):
     monkeypatch.setattr(search_bot, "DATABASE_PATH", str(tmp_path) + "/")
     for n in (0, 2, 10, 1):