librarian: mount + seed runtime JSON state under /lib_temp_files

(Re-applied cleanly on top of main: PR #18 was rebase-merged, so the branch
that became #19 conflicted on the already-landed commits. This carries ONLY
the librarian state fix - the sole content difference between that branch
and main - so nothing else is touched or lost.)

The worker threads open cr_results/rr_results/not_in_db/s_results.json in
place ('r+'), crashing with FileNotFoundError in a fresh container. New
conjurer_librarian/lib_paths.py resolves all four under a persistent dir
(CONJURER_LIBRARIAN_STATE_DIR, default /lib_temp_files) and seeds missing
ones with '{}' on import; shared by conjurer_librarian.py and scrape_bot.py
(no circular import). ndb_database/database initialised to {} before load so
a corrupt persisted file degrades to empty instead of NameError. search_bot
/search_bot2 open the DOI DB 'r' not 'r+' so /doi can stay read-only. Docker:
STATE_DIR env + VOLUME, compose mounts /srv/librarian/state:/lib_temp_files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docker/Dockerfile.librarian

@@ -18,10 +18,13 @@ COPY conjurer_librarian/ ./
 ENV PYTHONUNBUFFERED=1 \
     CONJURER_LIBRARIAN_HOST=0.0.0.0 \
     CONJURER_LIBRARIAN_PORT=5001 \
-    CONJURER_LIBRARIAN_DB_PATH=/doi/
+    CONJURER_LIBRARIAN_DB_PATH=/doi/ \
+    CONJURER_LIBRARIAN_STATE_DIR=/lib_temp_files
 
-# The local DOI chunk database (large) is mounted here.
-VOLUME ["/doi"]
+# /doi          = the local DOI chunk database (large, read-only).
+# /lib_temp_files = runtime JSON state (cr_results/rr_results/not_in_db/
+#                 s_results) - seeded on first run, persisted across restarts.
+VOLUME ["/doi", "/lib_temp_files"]
 EXPOSE 5001
 
 CMD ["python", "conjurer_librarian.py"]

docker/compose.librarian.yaml

@@ -14,7 +14,11 @@ services:
     ports:
       - "5001:5001"
     volumes:
-      # Local DOI chunk database (0_chunk.txt ... N_chunk.txt).
+      # Local DOI chunk database (0_chunk.txt ... N_chunk.txt). Read-only:
+      # the search workers only read it (open mode "r").
       - /srv/librarian/doi:/doi:ro
+      # Runtime JSON state (cr_results/rr_results/not_in_db/s_results) -
+      # seeded on first run, persisted here across restarts.
+      - /srv/librarian/state:/lib_temp_files
       # Optional: netrc holding Crossref credentials (or use CONJURER_CROSSREF_MAILTO).
       - /srv/librarian/secrets/.netrc:/secrets/.netrc:ro

docker/env/librarian.env.example

@@ -17,5 +17,9 @@ CONJURER_LIBRARIAN_DB_PATH=/doi/
 CONJURER_LIBRARIAN_MAXTHREADS=41
 CONJURER_LIBRARIAN_CHUNK=_chunk.txt
 
+# Runtime JSON state dir (mounted, persistent): cr_results/rr_results/
+# not_in_db/s_results are seeded here on first run.
+CONJURER_LIBRARIAN_STATE_DIR=/lib_temp_files
+
 # Optional netrc (for Crossref credentials)
 CONJURER_NETRC_FILE=/secrets/.netrc