Two field-reported robustness gaps on top of the hang fix.
1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised
UnicodeDecodeError from readline() - which is a ValueError, so the earlier
`except OSError` did NOT catch it. The finally-sentinel meant no hang, but the
producer died mid-file with a loud traceback and every DOI after the bad byte
went unsearched. Now chunks are opened with errors="replace" (bad bytes become
U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF,
and the producer's except is broadened from OSError to Exception so no per-file
error can ever crash the thread - it's logged and the sentinel still fires.
2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly
what goes out - target URL, uuid, DOI count and the DOI list - so the librarian
log plainly shows a result was sent and what was in it. And a failed POST is no
longer fatal: a RequestException used to propagate out of the worker loop and
kill the thread, stalling every future query until restart; it's now caught and
logged, and a non-200 from the bot is logged as a warning.
Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96
byte before a valid DOI and asserting that DOI is still found (file read to
completion, not aborted). All 5 search_bot unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(Re-applied cleanly on top of main: PR #18 was rebase-merged, so the branch
that became #19 conflicted on the already-landed commits. This carries ONLY
the librarian state fix - the sole content difference between that branch
and main - so nothing else is touched or lost.)
The worker threads open cr_results/rr_results/not_in_db/s_results.json in
place ('r+'), crashing with FileNotFoundError in a fresh container. New
conjurer_librarian/lib_paths.py resolves all four under a persistent dir
(CONJURER_LIBRARIAN_STATE_DIR, default /lib_temp_files) and seeds missing
ones with '{}' on import; shared by conjurer_librarian.py and scrape_bot.py
(no circular import). ndb_database/database initialised to {} before load so
a corrupt persisted file degrades to empty instead of NameError. search_bot
/search_bot2 open the DOI DB 'r' not 'r+' so /doi can stay read-only. Docker:
STATE_DIR env + VOLUME, compose mounts /srv/librarian/state:/lib_temp_files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Formatting and additional logging for bugfix
* Formatting and additional logging for bugfix
* Change of log location
* fixing_librarian_result
* Hopefully final fix that will be needed
* Temporary
* Glebokie gardlo
* Fix in scrape_bot
* Fix that was needed.
* Scraper settings
* Small fix in scraping
* FIxing bad naming convention
* Log message fix
* Test of deep search
* Fix in deep search
* Refactoring
* WHat and idiot coded that....
Ooops. That was me. A month ago.
* Refactoring continued
* Bugfix
* FIx
* Not needed after fix
aaa
aa
Bugfixing et masse
bbb
asas
aas
asas
aa
as
qq
as
as
as
as
as
aaa
asa
nn
aa
sda
88
asa
sasa
adsa
a
as
asdas
Final version before tests
Fix1
aa
aa
fx
dx
:)
aa
Hopefully last fix
aaa
aaa
bbb
bb
dd
aa
sdae
aa
bb
aa
kk
a
Fix
fx
aa
deploy
fx
asas
bb
as
fx
fx
aa
aa
mess
kk
A
fix
aa
aa
Viwe fix
a
test
a
aa
FX
asdasdasd
:)
asa
asdasda
aa
:)
asa
aa
:)
:)
sa
as
aa
aa
aa
aaa
as
11
aa
11
xxx
dd
ds
aa
a1
aa
Test
Suppress
Move
Installer
aaa
Serwis
`12`
aa
xd
asasa
xd
aa
asa
aaa
:)
Hejka
a
xd
aa
aaas
555
aaa
asa
aa
11
aaa
aaa
as
sa
async
asda
00
123
123
aa
as
ss
as
a
sa
aa
1
RES
T
Hejko
aa
11
AA
test
Test
gitea
Michal Tuszowski