Librarian: graceful shutdown with resumable search state
CI / compile (pull_request) Successful in 10s
CI / unit (pull_request) Successful in 28s
CI / integration (pull_request) Successful in 27s
build / build (push) Successful in 42s
CI / compile (push) Successful in 10s
CI / unit (push) Successful in 26s
CI / integration (push) Successful in 25s
CI / compile (pull_request) Successful in 10s
CI / unit (pull_request) Successful in 28s
CI / integration (pull_request) Successful in 27s
build / build (push) Successful in 42s
CI / compile (push) Successful in 10s
CI / unit (push) Successful in 26s
CI / integration (push) Successful in 25s
A restart of the librarian used to throw away an in-flight search (and any
searches still queued). Now search state survives a restart:
* Resumable DB scan (search_bot): each producer records a tell()-cookie
watermark per chunk file as it goes (safe because search_for_doi drains
the work queue before returning), and can seek back to it. search_for_doi
now takes stop_event + resume and returns (result_list, positions,
interrupted).
* Persisted requests: /query writes the accepted request to a disk queue
before enqueuing; replay_requests re-enqueues unfinished ones on startup.
So even a search still waiting in the queue survives a restart.
* Checkpoints: when a graceful shutdown interrupts a scan, the librarian
writes {dois, found-so-far, per-file offsets}. On restart answer_query
loads it, skips the (already done) Crossref+refine, and continues the
scan from the saved offsets with the found DOIs pre-marked - no line is
read twice and none is missed. A finished or crashed search forgets its
request+checkpoint (no poison-pill replay).
* Graceful shutdown: SIGTERM/SIGINT set a shutdown event; the running scan
checkpoints and the worker stops. The main thread then exits within a
BOUNDED window (CONJURER_LIBRARIAN_GRACEFUL_TIMEOUT, default 45s) so the
pod can never become an un-killable zombie. Needs terminationGracePeriod
>= that in the deploy (separate PR).
Tests: search_bot resume correctness (seek past scanned, don't miss/re-scan;
stop_event -> interrupted) and librarian state mechanics (request replay,
forget, checkpoint round-trip, poison-pill drop). Suite: 58 unit + 49
integration green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit was merged in pull request #17.
This commit is contained in:
@@ -52,6 +52,14 @@ class DiskQueue:
|
||||
def contains(self, key) -> bool:
|
||||
return os.path.exists(self._path(key))
|
||||
|
||||
def get(self, key):
|
||||
"""Return the payload stored for ``key``, or None if absent/unreadable."""
|
||||
try:
|
||||
with open(self._path(key), encoding="utf-8") as handle:
|
||||
return json.load(handle)["payload"]
|
||||
except (OSError, ValueError, KeyError, TypeError):
|
||||
return None
|
||||
|
||||
def remove(self, key) -> None:
|
||||
try:
|
||||
os.remove(self._path(key))
|
||||
|
||||
Reference in New Issue
Block a user