librarian: mount + seed runtime JSON state under /lib_temp_files

(Re-applied cleanly on top of main: PR #18 was rebase-merged, so the branch
that became #19 conflicted on the already-landed commits. This carries ONLY
the librarian state fix - the sole content difference between that branch
and main - so nothing else is touched or lost.)

The worker threads open cr_results/rr_results/not_in_db/s_results.json in
place ('r+'), crashing with FileNotFoundError in a fresh container. New
conjurer_librarian/lib_paths.py resolves all four under a persistent dir
(CONJURER_LIBRARIAN_STATE_DIR, default /lib_temp_files) and seeds missing
ones with '{}' on import; shared by conjurer_librarian.py and scrape_bot.py
(no circular import). ndb_database/database initialised to {} before load so
a corrupt persisted file degrades to empty instead of NameError. search_bot
/search_bot2 open the DOI DB 'r' not 'r+' so /doi can stay read-only. Docker:
STATE_DIR env + VOLUME, compose mounts /srv/librarian/state:/lib_temp_files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Michal Tuszowski
2026-07-09 13:40:13 +02:00
parent 2a21fc9e8c
commit fd6427a282
8 changed files with 68 additions and 11 deletions
+3 -1
View File
@@ -16,6 +16,8 @@ from urllib.request import urlopen
from requests import ConnectionError as RequestsConnectionError
from requests import ConnectTimeout, Timeout
import lib_paths
SCR_DATABASE_PATH = os.getenv("CONJURER_LIBRARIAN_DB_PATH", r"C:\\Database\\chunks\\")
SCR_FILENAME = os.getenv("CONJURER_LIBRARIAN_SCRAPE_CHUNK", "40_chunk.txt")
SCR_ENCODING = os.getenv("CONJURER_ENCODING", "utf-8")
@@ -30,7 +32,7 @@ def load_ndb_to_q(logger):
"""
logger.info("Loader started")
while True:
with open("not_in_db.json", "r+", encoding="utf-8") as ndb_file:
with open(lib_paths.NOT_IN_DB, "r+", encoding="utf-8") as ndb_file:
try:
ndb_database = json.load(ndb_file)
for _ in range (1,10):