librarian: tolerate bad bytes in chunks; log what the result-send does

Two field-reported robustness gaps on top of the hang fix.

1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised
UnicodeDecodeError from readline() - which is a ValueError, so the earlier
`except OSError` did NOT catch it. The finally-sentinel meant no hang, but the
producer died mid-file with a loud traceback and every DOI after the bad byte
went unsearched. Now chunks are opened with errors="replace" (bad bytes become
U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF,
and the producer's except is broadened from OSError to Exception so no per-file
error can ever crash the thread - it's logged and the sentinel still fires.

2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly
what goes out - target URL, uuid, DOI count and the DOI list - so the librarian
log plainly shows a result was sent and what was in it. And a failed POST is no
longer fatal: a RequestException used to propagate out of the worker loop and
kill the thread, stalling every future query until restart; it's now caught and
logged, and a non-200 from the bot is logged as a warning.

Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96
byte before a valid DOI and asserting that DOI is still found (file read to
completion, not aborted). All 5 search_bot unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

conjurer_librarian/conjurer_librarian.py

@@ -451,18 +451,48 @@ class BackgroundTaskSearch(threading.Thread):
                 json.dump(database, s_file)
             self.app.logger.info("FINISHED")
 
-            self.app.logger.info(result)
-            coroutine = asyncio.to_thread(
-                requests.post,
-                f"{MAIN_BOT_ADDRESS}{SEND_RESULTS}",
-                json=result,
-                headers=_service_headers(),
-                timeout=360,
+            # Send the result back to the bot. Log EXACTLY what goes out (target,
+            # uuid, how many DOIs and which) so the librarian log makes it plain a
+            # result was sent and what was in it.
+            payload = result  # shape: {uuid: {DOI: {"Title": ..., "type": ...}}}
+            hits = payload.get(librarian.uuid, {}) if isinstance(payload, dict) else {}
+            target = f"{MAIN_BOT_ADDRESS}{SEND_RESULTS}"
+            self.app.logger.info(
+                "SENDING result for %s to %s: %d DOI(s): %s",
+                librarian.uuid,
+                target,
+                len(hits),
+                list(hits.keys()),
             )
-            self.app.logger.info("SENT")
-            result = await coroutine
-            self.app.logger.info(result.status_code)
-            self.app.logger.info("SEND CONFIRMED")
+            # A failed send must NOT kill this worker - otherwise a bot that is
+            # momentarily down stalls every future query until the librarian is
+            # restarted. Log and carry on to the next queued search.
+            try:
+                response = await asyncio.to_thread(
+                    requests.post,
+                    target,
+                    json=payload,
+                    headers=_service_headers(),
+                    timeout=360,
+                )
+                if response.status_code == 200:
+                    self.app.logger.info(
+                        "SENT result for %s -> HTTP 200 (bot accepted)", librarian.uuid
+                    )
+                else:
+                    self.app.logger.warning(
+                        "SENT result for %s but bot returned HTTP %s: %s",
+                        librarian.uuid,
+                        response.status_code,
+                        response.text[:500],
+                    )
+            except requests.exceptions.RequestException as exc:
+                self.app.logger.error(
+                    "FAILED to send result for %s to %s: %s",
+                    librarian.uuid,
+                    target,
+                    exc,
+                )
             await asyncio.sleep(1)