5 Commits

Author	SHA1	Message	Date
gitea	ac16b77f56	Librarian: graceful shutdown with resumable search state ... A restart of the librarian used to throw away an in-flight search (and any searches still queued). Now search state survives a restart: * Resumable DB scan (search_bot): each producer records a tell()-cookie watermark per chunk file as it goes (safe because search_for_doi drains the work queue before returning), and can seek back to it. search_for_doi now takes stop_event + resume and returns (result_list, positions, interrupted). * Persisted requests: /query writes the accepted request to a disk queue before enqueuing; replay_requests re-enqueues unfinished ones on startup. So even a search still waiting in the queue survives a restart. * Checkpoints: when a graceful shutdown interrupts a scan, the librarian writes {dois, found-so-far, per-file offsets}. On restart answer_query loads it, skips the (already done) Crossref+refine, and continues the scan from the saved offsets with the found DOIs pre-marked - no line is read twice and none is missed. A finished or crashed search forgets its request+checkpoint (no poison-pill replay). * Graceful shutdown: SIGTERM/SIGINT set a shutdown event; the running scan checkpoints and the worker stops. The main thread then exits within a BOUNDED window (CONJURER_LIBRARIAN_GRACEFUL_TIMEOUT, default 45s) so the pod can never become an un-killable zombie. Needs terminationGracePeriod >= that in the deploy (separate PR). Tests: search_bot resume correctness (seek past scanned, don't miss/re-scan; stop_event -> interrupted) and librarian state mechanics (request replay, forget, checkpoint round-trip, poison-pill drop). Suite: 58 unit + 49 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-02 22:09:49 +02:00
gitea	40605b959f	Librarian: bound the DOI search work queue to stop OOM-killing the pod ... The pod restarted spontaneously mid-search (no liveness probe is set, so it was the kernel OOM-killer against the 1Gi limit). Cause: search_bot built its work queue with maxsize 35_500_000. The producers stream the WHOLE DOI database (tens of millions of lines across chunks) into it while a few consumers drain, so the queue could buffer gigabytes of lines - blowing the 1Gi container and taking the whole in-flight search with it. Bound the queue (default 100k lines, env CONJURER_LIBRARIAN_WORKQ_SIZE), so producers backpressure to consumers and RAM stays in the low MB. Because a bounded queue means a producer can now block on a FULL queue, make the producer's put timeout-poll the TERM sentinel, so a full queue whose consumers have already finished (all DOIs found) can never deadlock it. New test pins that: tiny queue + target on line 1 + thousands of trailing decoys still terminates and finds the target. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-02 19:34:56 +02:00
gitea	defc482a22	fix: batch A - crash bugs, a leak, and startup/edge fragility ... Seven confirmed defects from the code audit, each small and low-risk. * ai_functions.get_random_cyclic_message: random.randint(0, len(CYCLIC_WORDS)) is inclusive -> could return len -> IndexError. Now randrange(len) + guard on an empty CYCLIC_WORDS. * librarian_commands.get_image_sadox: random.randrange(0, len(res)-1) never picked the last comic and raised ValueError('empty range') on a single file. Now randrange(len) + an empty-dir guard. * ai_commands image generation: every DALL-E error branch replied but did not return, so control fell through to `if response:` with response unbound -> UnboundLocalError right after the friendly message. Each branch now returns; response is pre-initialised; and PermissionDeniedError no longer passes a (message, text) tuple as a single arg. * search_bot DOI match: `item["DOI"] in data` was a substring test, so a DOI that is a prefix of a longer one (10.1/1 vs 10.1/12) produced a false 'exists' hit. Now matches the line's first whitespace token exactly, via an O(1) dict index built once per consumer (also removes the O(queried-DOIs) per-line scan - a real win for large databases). * communication_subroutine.scan_incoming: matched records were never removed from awaiting_q, so it grew unbounded over uptime and a reused UUID could re-match a stale record. Matched records are now dropped after dispatch. * communication_subroutine.id3: (resp.headers.get("icy-name") or "").title() guards against a stream that omits headers (was AttributeError on None, 500-ing the /prepped_tracks "next" handler). * betoniarka.scan_tracks: waits for the radio logs to exist instead of dying with FileNotFoundError on a fresh deploy (which silently killed the now-playing forwarder until a restart). Verified: tests/unit/test_search_bot.py gains exact-match and trailing-metadata cases; full unit job 43 passed. Remaining observations (image-gen stale /home/pi fallback paths + dead FileNotFoundError-after-OSError branch; tailer still vulnerable to mid-run log rotation; DOI-first-token assumption) noted for follow-up - none are crashes on the normal path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-31 23:01:27 +02:00
gitea	031c1f8aea	librarian: tolerate bad bytes in chunks; log what the result-send does ... Two field-reported robustness gaps on top of the hang fix. 1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised UnicodeDecodeError from readline() - which is a ValueError, so the earlier `except OSError` did NOT catch it. The finally-sentinel meant no hang, but the producer died mid-file with a loud traceback and every DOI after the bad byte went unsearched. Now chunks are opened with errors="replace" (bad bytes become U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF, and the producer's except is broadened from OSError to Exception so no per-file error can ever crash the thread - it's logged and the sentinel still fires. 2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly what goes out - target URL, uuid, DOI count and the DOI list - so the librarian log plainly shows a result was sent and what was in it. And a failed POST is no longer fatal: a RequestException used to propagate out of the worker loop and kill the thread, stalling every future query until restart; it's now caught and logged, and a non-200 from the bot is logged as a warning. Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96 byte before a valid DOI and asserting that DOI is still found (file read to completion, not aborted). All 5 search_bot unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-30 15:59:07 +02:00
gitea	1a59c9f6c5	librarian: stop the DOI search from hanging on a chunk-count mismatch ... search_bot conflated MAXTHREADS into two jobs at once - how many chunk files to read (files 0..MAXTHREADS-1) AND how many producer sentinels to wait for - so the two had to match exactly. Set too low it silently skipped trailing chunks; set too high (or with any chunk missing/unreadable) a producer crashed before emitting its sentinel, the consumers' count never reached the threshold, and search_for_doi hung on join() forever. The idle-timeout failsafe that was meant to break a starved consumer was dead code: `if empty_counter > 5: ... elif empty_counter > 10: break` - >10 implies >5, so the elif never ran. Fix, three layers: * auto-discover the chunk files present (discover_chunk_files: <n>_chunk.txt in numeric order) instead of range(0, MAXTHREADS). All files are read regardless of count, and no producer is ever pointed at a missing file; * the sentinel threshold is now the number of producers actually started, so it can't drift from what's emitted; * producers emit their sentinel in a finally, so even a crash (missing/unreadable chunk) can't starve the count; and the idle backstop is reordered so it can actually fire (>EMPTY_LIMIT seconds) as a last resort. MAXTHREADS is deprecated and unused (kept only so old env files don't break); docs/env updated to say chunk files are auto-discovered. For the reported case (MAXTHREADS=40, files 0..43): before, files 40-43 were silently never searched, and any run that referenced a missing chunk hung forever. After, all 44 are searched and it always terminates. Verified in a pytest-only venv (tests/unit/test_search_bot.py): DOI in a trailing chunk is found; an unreadable chunk still terminates; empty dir returns at once; discovery is numeric-sorted. Full unit job 27 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-30 00:15:03 +02:00