gitea 031c1f8aea librarian: tolerate bad bytes in chunks; log what the result-send does ...

Two field-reported robustness gaps on top of the hang fix.

1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised
UnicodeDecodeError from readline() - which is a ValueError, so the earlier
`except OSError` did NOT catch it. The finally-sentinel meant no hang, but the
producer died mid-file with a loud traceback and every DOI after the bad byte
went unsearched. Now chunks are opened with errors="replace" (bad bytes become
U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF,
and the producer's except is broadened from OSError to Exception so no per-file
error can ever crash the thread - it's logged and the sentinel still fires.

2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly
what goes out - target URL, uuid, DOI count and the DOI list - so the librarian
log plainly shows a result was sent and what was in it. And a failed POST is no
longer fatal: a RequestException used to propagate out of the worker loop and
kill the thread, stalling every future query until restart; it's now caught and
logged, and a non-200 from the bot is logged as a warning.

Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96
byte before a valid DOI and asserting that DOI is still found (file read to
completion, not aborted). All 5 search_bot unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-30 15:59:07 +02:00

..

integration

tests: retarget /clear_pr_pls auth tests after the musician/radio split

2026-07-30 11:41:39 +02:00

unit

librarian: tolerate bad bytes in chunks; log what the result-send does

2026-07-30 15:59:07 +02:00