15 Commits

Author	SHA1	Message	Date
gitea	13d2a04052	tests: fix the flaky heartbeat coverage assertion ... test_search_fills_progress_with_live_positions_and_total asserted 100% coverage after searching for a DOI that EXISTS. Once every queried DOI is found the consumer signals TERM and the producers stop mid-file, so the recorded offsets reach an arbitrary point - the assertion was racing the scan and failed roughly one full-suite run in two. Split into the two things that are actually deterministic: coverage is now measured with an ABSENT DOI (nothing can stop the scan early, so 100% is guaranteed), and the found-target case asserts what holds regardless of where the producers stopped - the total is known, progress is bounded and sane, and the hit is reported. Verified: 5 consecutive runs of the file and 3 consecutive full integration runs, all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-24 14:10:06 +02:00
gitea	ae1bd67772	Librarian: survive transient Crossref failures instead of losing the search ... Field report: one httpx ReadTimeout inside habanero surfaced as 'Search <uuid> crashed', and the worker's crash handler then FORGOT the request - so an expensive search vanished and the user was told it was eaten, because a public API blinked once. Two defences: * Every habanero call goes through _crossref_call, which retries with linear backoff (CONJURER_CROSSREF_ATTEMPTS, default 4; backoff CONJURER_CROSSREF_BACKOFF, 5s). habanero wraps httpx errors in a plain RuntimeError so we can't filter narrowly - retries are simply bounded and the last error is re-raised. They now also run via asyncio.to_thread, so a slow Crossref no longer blocks the worker's event loop. * A crashed search is no longer dropped on the first failure: the attempt count is persisted with the request and the search is requeued (keeping any checkpoint, so a crashed DB scan resumes rather than restarts) until CONJURER_SEARCH_MAX_ATTEMPTS (default 3). It stays 'queued' for the bot's watchdog while retrying, and only after the cap is it forgotten. Also: scrape_bot's 'Got blocked' is routine sci-hub behaviour (it backs off an hour and carries on) - log it as WARNING, not ERROR, so it stops looking like a fault when scanning for real problems. Tests: retry-then-succeed, bounded re-raise, no retry on success, the persisted attempt counter, and forget-on-give-up. Suite: 58 unit + 70 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-04 13:50:47 +02:00
gitea	9f22dbf94b	Librarian: 'still searching' heartbeat every 20 min ... A deep scan runs for hours with nothing in the log between start and finish, so it's impossible to tell a working search from a wedged one. Every CONJURER_LIBRARIAN_HEARTBEAT_SECONDS (default 1200 = 20 min) a running search now logs that it is still going, with its uuid, the search phrase, hits so far, elapsed minutes, and a rough how-far-along. The estimate is deliberately cheap: the producers ALREADY record a byte offset per chunk file (the resume watermarks), and the total size is stat()'d once per search when the chunk list is discovered. A reading is then just a sum over ~40 ints - nothing extra happens per line, and no cycles are spent estimating how many cycles are left. search_for_doi takes an optional progress dict it fills with the live positions dict + total_bytes; the librarian publishes the running search (uuid/query/progress/live hits) while the scan runs and clears it in finally. Nothing running => the heartbeat stays quiet. Tests: percentage maths incl. unknown-total and >100% clamping, the register/clear round-trip, and an end-to-end check that a real scan fills progress so the offsets cover the chunk files on disk. Suite: 58 unit + 65 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-04 12:09:46 +02:00
gitea	f4dea53502	Librarian: simple result cache for repeated queries ... A repeat of the same query (whitespace/case-normalised, scoped by deep-vs-shallow) returns the stored hits and skips the whole Crossref call and DB scan. Disk-backed (survives restart), TTL'd (CONJURER_LIBRARIAN_CACHE_TTL, default 7d; 0 disables) and size-bounded (CONJURER_LIBRARIAN_CACHE_MAX, default 500). Reuses DiskQueue, so it's a handful of lines. Nothing fancy - exact (normalised) match, not fuzzy. Checked before Crossref only on a fresh search (a resume from checkpoint still continues its scan), and stored after a completed search. Tests: hit/miss, normalisation, deep/shallow separation, expiry, disable, prune. Suite: 58 unit + 59 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-03 20:07:38 +02:00
gitea	26b6ab636e	Librarian: answer each query back to the bot that sent it ... So one librarian can serve several bots (test + deploy) instead of firing every result/pong at a single static CONJURER_MAIN_BOT. * The bot includes its own callback address (CONJURER_SELF_CALLBACK) in every /query and /ping. * The librarian stores that callback with the query (persisted with the request, so a replay after restart still answers the right bot) and, for results, in the OUTBOX entry ({target, payload}) so the resender delivers to the origin bot even across a librarian restart. * Pongs go back to the pinging bot too - otherwise a second bot's health check would be ponged to the first and always time out, so it could never enable its librarian cog. * Empty callback falls back to MAIN_BOT_ADDRESS, and a legacy OUTBOX entry (raw payload, pre-callback) is still delivered to the default bot, so the upgrade is seamless. Tests: per-origin result delivery + legacy-shape fallback (outbox), busy/idle pong routed to the callback bot vs default (lifecycle). Suite: 58 unit + 52 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-03 18:36:49 +02:00
gitea	ac16b77f56	Librarian: graceful shutdown with resumable search state ... A restart of the librarian used to throw away an in-flight search (and any searches still queued). Now search state survives a restart: * Resumable DB scan (search_bot): each producer records a tell()-cookie watermark per chunk file as it goes (safe because search_for_doi drains the work queue before returning), and can seek back to it. search_for_doi now takes stop_event + resume and returns (result_list, positions, interrupted). * Persisted requests: /query writes the accepted request to a disk queue before enqueuing; replay_requests re-enqueues unfinished ones on startup. So even a search still waiting in the queue survives a restart. * Checkpoints: when a graceful shutdown interrupts a scan, the librarian writes {dois, found-so-far, per-file offsets}. On restart answer_query loads it, skips the (already done) Crossref+refine, and continues the scan from the saved offsets with the found DOIs pre-marked - no line is read twice and none is missed. A finished or crashed search forgets its request+checkpoint (no poison-pill replay). * Graceful shutdown: SIGTERM/SIGINT set a shutdown event; the running scan checkpoints and the worker stops. The main thread then exits within a BOUNDED window (CONJURER_LIBRARIAN_GRACEFUL_TIMEOUT, default 45s) so the pod can never become an un-killable zombie. Needs terminationGracePeriod >= that in the deploy (separate PR). Tests: search_bot resume correctness (seek past scanned, don't miss/re-scan; stop_event -> interrupted) and librarian state mechanics (request replay, forget, checkpoint round-trip, poison-pill drop). Suite: 58 unit + 49 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-02 22:09:49 +02:00
gitea	c5643aa28f	Librarian: drop write-only result dumps + tame search logging ... Two hygiene fixes on top of the work-queue OOM bound: Result dumps: cr_results / rr_results / s_results.json were write-only (nothing reads them) yet accumulated EVERY search forever and json.load'd the whole growing file on each write - unbounded RAM and PVC growth, and for a deep search the raw cr_results dump is hundreds of MB. They are now off by default (CONJURER_LIBRARIAN_DEBUG_DUMPS) and, when enabled, are overwritten with just the latest search - never loaded or accumulated. not_in_db.json is untouched: it's a real queue the scraper drains. Search logging: search_bot logged via print(), including a per-line carriage-return progress line that flooded stdout / the log file with millions of entries - fine for a desktop app, unreadable and bloating in a container. All of it is now proper logging at DEBUG (with coarse per-500k-line progress), so a normal run is quiet. The librarian log level is configurable (CONJURER_LIBRARIAN_LOG_LEVEL, default INFO) and a stdout handler is added so stays useful now that the search no longer prints straight to stdout. Set DEBUG for full verbosity. Also: make test_result_delivery_contract hermetic (point the durable spool at a temp dir so it can't pollute or be poisoned by the real result_inbox/ between runs) and gitignore the runtime spool dirs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-02 20:19:49 +02:00
gitea	8e18071bb6	Librarian: stop warning about missing netrc when mailto is set via env ... Librarian.__init__ reads the Crossref contact from CONJURER_CROSSREF_MAILTO, then tries to override it from a 'crossref' netrc entry. When no netrc is mounted (the normal container setup - default /root/.netrc) the read raises FileNotFoundError and it logged 'Crossref credentials missing in netrc ...' on EVERY search, even though the env var was set and used. Pure noise. Only warn when there is genuinely no contact from either source (env unset AND netrc unreadable) - which is also the case that then raises. When the env var is set, a missing netrc is expected and logged at debug. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-02 17:01:49 +00:00
gitea	44b7298a15	Durable result delivery: OUTBOX + idempotent INBOX so results never die ... An 8h search result must survive a transient bot outage, an api/address misroute, or a restart of either side. Make the librarian->bot result path durably at-least-once with idempotent rendering: Shared: durable_queue.DiskQueue - a dependency-free, atomically-written, one-file-per-key disk queue (unit-tested), shared by both images (added to Dockerfile.librarian; the bot already COPYs *.py). Librarian (sender): finished results go to a persistent OUTBOX before sending; delivery retries with backoff; an entry is removed only on a positive ACK; a resender thread keeps flushing the OUTBOX, so a result survives a bot outage AND a librarian restart (OUTBOX is on the state volume) - it simply keeps trying until acked. Bot (receiver): /conjurer is now idempotent and durable - each result is persisted to an INBOX before acking and only queued if its uuid was not already delivered (dropped as a duplicate) or already pending. Once the cog actually renders it, mark_delivered() records the uuid and clears the inbox, so the librarian's resends become no-ops. On startup the bot replays any accepted-but-unrendered result from the INBOX, so a bot crash mid-flight doesn't lose it. Pongs stay ephemeral. Together: the librarian keeps a result until the bot confirms it; the bot keeps it until it is on screen; duplicates never double-render. Combined with the deploy return-path fix, an expensive result no longer vanishes. Tests: unit test_durable_queue; integration test_librarian_outbox (retry/backoff, resend survives outage) and test_result_durable_delivery (persist, dedup pending, dedup delivered, replay, pong not persisted). Suite: 55 unit + 39 integration green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-02 15:31:37 +00:00
gitea	04070ea7f1	tests: pin the librarian->bot result delivery contract ... Diagnostic coverage for the 'search vanished' report. Proves the bot side of result delivery is correct end to end (right shape reaches IN_COMM_Q; empty result still delivered; wrong api-key -> 401 vanish; uuid mismatch -> orphaned away from the querent), which isolates a SYSTEMATIC vanish to transport: the librarian being unable to reach the bot's /conjurer at all (CONJURER_MAIN_BOT). The bot Service is NodePort with no pinned nodePort while the librarian hardcodes :32442 - and being in-cluster it should use the Service DNS http://bot:5000 instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-02 16:11:08 +02:00
gitea	5d321f2f5b	Librarian: busy-aware ping + per-query lost-result watchdog ... Two refinements to the librarian health/delivery story, matching how it actually behaves under load: 1. Busy-aware ping (case b - broken return path). A ping arriving while the worker is grinding a search no longer queues behind it (which made a healthy-but-busy librarian time out and look dead). The librarian tracks worker_busy and, when set, pongs back IMMEDIATELY without touching the queue. Being busy is fine - you can keep piling searches on. The ping still travels the librarian->bot return path, so it keeps catching the one thing it must: a disrupted/incompatible return path where queries vanish. Idle pings still go through the internal queue. 2. Per-query watchdog (case a - finished but result lost). The librarian now tracks every search uuid's lifecycle (queued -> processing -> gone) in active_queries, exposed via a new POST /query_status. After dispatching a search the bot records it in self.pending; watch_pending polls /query_status for each. While the librarian still knows the uuid the search is progressing - left alone. The moment a uuid VANISHES there while still pending on the bot, its result was computed but never delivered: after a grace window (to rule out an in-flight result) the bot posts a notice to the channel - but ONLY then. A normally delivered result is popped from self.pending by check_data_q and never flagged. Hardening: the worker's search body is now wrapped in try/except/finally so a crashing search can't kill the worker thread (which would freeze the queue), and worker_busy / active_queries are always cleared. The grace logic lives in a dependency-free librarian_watchdog.pending_verdict so it is unit-testable without discord/pdf libs. /ping and /query_status are plain (sync) views so they run without flask[async]. Tests: unit test_librarian_watchdog (verdict transitions); integration test_librarian_query_lifecycle (query_status known/unknown + auth, idle-ping-queues, busy-ping-pongs-directly). Suite: 28 integration + 48 unit green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-01 18:58:03 +02:00
gitea	ed8b271b4e	Gate librarian cog on a full ping round-trip, not a bare GET ... The librarian health check was a plain GET to '/', which only proved Flask was listening - not that the service could actually take a query, run it through its internal queue+worker, and answer back. So the cog could load against a librarian whose worker was wedged or that couldn't reach the bot on the return leg. Replace it with a ping that travels the SAME path a real search does, on both sides: bot: QueryControl -> OUT_COMM_Q -> scan_queue -> awaiting_q librarian: POST /ping -> librarian_queue -> worker pulls it off (no Crossref/DOI search) -> pongs back with the same uuid bot: /conjurer -> incoming_q -> scan_incoming matches uuid, wakes waiter The cog enables only when that whole loop closes within 3s. This also proves the librarian->bot return path, which a GET never did. Safety: uuid is random per ping; the wait and POST are both bounded so startup can't stall; a pong that finds no waiter is dropped (never orphaned into IN_COMM_Q, which would make the cog post a bogus 'no results' message); and a ping whose pong never returns is swept out of awaiting_q after PING_TTL_SECONDS so nothing leaks. All awaiting_q writes stay within scan_queue (append) and scan_incoming (remove) - no locks, no cross-thread mutation. Integration tests cover: OK round-trip, timeout when accepted-but-no-pong, unreachable, non-200, orphan-pong-dropped, and that real results still reach IN_COMM_Q. Suite: 24 integration + 41 unit green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-01 15:00:04 +02:00
gitea	491f957315	tests: retarget /clear_pr_pls auth tests after the musician/radio split ... test_musician_auth.py still probed /clear_pr_pls on the musician, but that endpoint moved to betoniarka during the split - the musician now 404s it, so all three tests failed 404 != 401/200. This was pre-existing debt, unrelated to the AI/share/bridge work; it just kept the integration job red. Split the coverage to match the current architecture: * test_musician_auth.py exercises the same auth contract (no key -> 401, key -> 200, key unset -> open) against /get_share_list, an authenticated endpoint the musician still serves, with a valid body so the permitted case is a clean 200 rather than a 400; * new test_betoniarka_auth.py covers /clear_pr_pls where it now lives, pointing PRIORITY_PLAYLIST_PATH at a tmp file so the authorised case can truncate it, and checks /ping stays open; * conftest.py adds conjurer_betoniarka to sys.path so the service imports. Verified in a clean venv (pytest flask waitress requests), matching the CI integration job: 13 passed, up from 3 failed / 6 passed. Unit suite unaffected (23 passed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-30 11:41:39 +02:00
gitea	b13a8afa01	bot: queued AI query interface + librarian AI review of results ... Two connected features. 1) AI query interface (via the comm layer). communication_subroutine gains an AI_QUERY_Q, a submit_ai_query() in-process entry point, and an authed POST /ai_query endpoint ({prompt, channel_id, request_type?, username?}). The prompt is queued and answered asynchronously by a new tasks.loop worker in the always-loaded AI cog (Events), which calls handle_response - so it runs on whichever backend $gadaj_teraz currently selects (GPT or Claude) - and posts the answer to the requested channel, chunked to Discord's limit. request_type "NONE" (default) is a clean one-shot: no persona system prompt, no memory write. The worker starts before the OpenAI guard in cog_load, so it also runs on a Claude-only box; cog_unload cancels it. 2) Librarian AI review. New command $wyszukaj_z_recenzja mirrors $wyszukaj_linki_do_dokumentow but sets ai_review=True on the QueryControl, which rides the round-trip and is matched back by UUID. When the hits return, check_data_q sends the raw list as before, then - if flagged - hands the same list (already in Crossref-relevance order) plus the search phrase to the AI queue for a weighted re-rank and per-source review, delivered to the same channel. QueryControl gains an ai_review flag (default False, so the orphan path and all existing callers are unaffected). Confirmed separately (and noted in the docs): the DOI list the AI receives is pre-sorted by Crossref relevance - the librarian pipeline only filters (drops title-less items) and splits (in-db / not-in-db), never re-sorts, and relies on insertion-ordered dicts (Py 3.7+). Verified: /ai_query auth (401/200/400/open), submit_ai_query and the queued dict shape, and the QueryControl flag - via a Flask test client and tests/integration/test_ai_query_endpoint.py (5 tests, all pass; integration suite 11 passed, the 3 failures are the pre-existing /clear_pr_pls musician tests fixed on a separate branch). Full first-party compile clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-30 09:27:11 +00:00
Michal Tuszowski	a6c20a0054	ci: replace broken default workflows with compile/unit/integration CI ... The two scaffold workflows (Python application / Python package) failed on every PR: they installed deps from a non-existent requirements.txt, ran flake8/pytest over the vendored yt_dlp fork (new syntax under the 3.8/3.9 matrix), and collected ad-hoc root scripts — notably test_ai.py, which is an invalid pasted object dump (not Python). - Remove python-app.yml / python-package.yml and the junk root scripts (test.py, test_ai.py, test_time.py) - Add .github/workflows/ci.yml with three PR-check jobs: * compile — py_compile every first-party .py (no deps) * unit — pytest on pure logic (conanjurer_functions, constants) * integration — boot the Flask services and assert the X-Conjurer-Api-Key auth contract (communication_subroutine + conjurer_musician) - Add tests/ suite, pytest.ini (testpaths=tests) and conftest.py (sys.path) Fixes surfaced by the compile gate / needed for the integration job: - conjurer_librarian/search_bot.py + search_bot2.py: f-string reused the same quote ({item["exists"]}) -> SyntaxError on Python < 3.12 - conjurer_musician/media_search_functions.py: made import-safe (env-overridable paths, lazy DB load / mkdir) so the service can be imported and tested off the Pi Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 12:24:30 +02:00