6 Commits

Author	SHA1	Message	Date
gitea	031c1f8aea	librarian: tolerate bad bytes in chunks; log what the result-send does ... Two field-reported robustness gaps on top of the hang fix. 1) A stray non-UTF-8 byte in a chunk (0x96 in the report) raised UnicodeDecodeError from readline() - which is a ValueError, so the earlier `except OSError` did NOT catch it. The finally-sentinel meant no hang, but the producer died mid-file with a loud traceback and every DOI after the bad byte went unsearched. Now chunks are opened with errors="replace" (bad bytes become U+FFFD; DOIs are ASCII so a match is never affected) so the read runs to EOF, and the producer's except is broadened from OSError to Exception so no per-file error can ever crash the thread - it's logged and the sentinel still fires. 2) The result-send back to the bot (BackgroundTaskSearch._run) now logs exactly what goes out - target URL, uuid, DOI count and the DOI list - so the librarian log plainly shows a result was sent and what was in it. And a failed POST is no longer fatal: a RequestException used to propagate out of the worker loop and kill the thread, stalling every future query until restart; it's now caught and logged, and a non-200 from the bot is logged as a warning. Verified: tests/unit/test_search_bot.py gains a case writing a chunk with a 0x96 byte before a valid DOI and asserting that DOI is still found (file read to completion, not aborted). All 5 search_bot unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-30 15:59:07 +02:00
gitea	491f957315	tests: retarget /clear_pr_pls auth tests after the musician/radio split ... test_musician_auth.py still probed /clear_pr_pls on the musician, but that endpoint moved to betoniarka during the split - the musician now 404s it, so all three tests failed 404 != 401/200. This was pre-existing debt, unrelated to the AI/share/bridge work; it just kept the integration job red. Split the coverage to match the current architecture: * test_musician_auth.py exercises the same auth contract (no key -> 401, key -> 200, key unset -> open) against /get_share_list, an authenticated endpoint the musician still serves, with a valid body so the permitted case is a clean 200 rather than a 400; * new test_betoniarka_auth.py covers /clear_pr_pls where it now lives, pointing PRIORITY_PLAYLIST_PATH at a tmp file so the authorised case can truncate it, and checks /ping stays open; * conftest.py adds conjurer_betoniarka to sys.path so the service imports. Verified in a clean venv (pytest flask waitress requests), matching the CI integration job: 13 passed, up from 3 failed / 6 passed. Unit suite unaffected (23 passed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-30 11:41:39 +02:00
gitea	b13a8afa01	bot: queued AI query interface + librarian AI review of results ... Two connected features. 1) AI query interface (via the comm layer). communication_subroutine gains an AI_QUERY_Q, a submit_ai_query() in-process entry point, and an authed POST /ai_query endpoint ({prompt, channel_id, request_type?, username?}). The prompt is queued and answered asynchronously by a new tasks.loop worker in the always-loaded AI cog (Events), which calls handle_response - so it runs on whichever backend $gadaj_teraz currently selects (GPT or Claude) - and posts the answer to the requested channel, chunked to Discord's limit. request_type "NONE" (default) is a clean one-shot: no persona system prompt, no memory write. The worker starts before the OpenAI guard in cog_load, so it also runs on a Claude-only box; cog_unload cancels it. 2) Librarian AI review. New command $wyszukaj_z_recenzja mirrors $wyszukaj_linki_do_dokumentow but sets ai_review=True on the QueryControl, which rides the round-trip and is matched back by UUID. When the hits return, check_data_q sends the raw list as before, then - if flagged - hands the same list (already in Crossref-relevance order) plus the search phrase to the AI queue for a weighted re-rank and per-source review, delivered to the same channel. QueryControl gains an ai_review flag (default False, so the orphan path and all existing callers are unaffected). Confirmed separately (and noted in the docs): the DOI list the AI receives is pre-sorted by Crossref relevance - the librarian pipeline only filters (drops title-less items) and splits (in-db / not-in-db), never re-sorts, and relies on insertion-ordered dicts (Py 3.7+). Verified: /ai_query auth (401/200/400/open), submit_ai_query and the queued dict shape, and the QueryControl flag - via a Flask test client and tests/integration/test_ai_query_endpoint.py (5 tests, all pass; integration suite 11 passed, the 3 failures are the pre-existing /clear_pr_pls musician tests fixed on a separate branch). Full first-party compile clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-30 09:27:11 +00:00
gitea	1a59c9f6c5	librarian: stop the DOI search from hanging on a chunk-count mismatch ... search_bot conflated MAXTHREADS into two jobs at once - how many chunk files to read (files 0..MAXTHREADS-1) AND how many producer sentinels to wait for - so the two had to match exactly. Set too low it silently skipped trailing chunks; set too high (or with any chunk missing/unreadable) a producer crashed before emitting its sentinel, the consumers' count never reached the threshold, and search_for_doi hung on join() forever. The idle-timeout failsafe that was meant to break a starved consumer was dead code: `if empty_counter > 5: ... elif empty_counter > 10: break` - >10 implies >5, so the elif never ran. Fix, three layers: * auto-discover the chunk files present (discover_chunk_files: <n>_chunk.txt in numeric order) instead of range(0, MAXTHREADS). All files are read regardless of count, and no producer is ever pointed at a missing file; * the sentinel threshold is now the number of producers actually started, so it can't drift from what's emitted; * producers emit their sentinel in a finally, so even a crash (missing/unreadable chunk) can't starve the count; and the idle backstop is reordered so it can actually fire (>EMPTY_LIMIT seconds) as a last resort. MAXTHREADS is deprecated and unused (kept only so old env files don't break); docs/env updated to say chunk files are auto-discovered. For the reported case (MAXTHREADS=40, files 0..43): before, files 40-43 were silently never searched, and any run that referenced a missing chunk hung forever. After, all 44 are searched and it always terminates. Verified in a pytest-only venv (tests/unit/test_search_bot.py): DOI in a trailing chunk is found; an unreadable chunk still terminates; empty dir returns at once; discovery is numeric-sorted. Full unit job 27 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-30 00:15:03 +02:00
Michal Tuszowski	3d9d47aa90	ai: single-switch GPT/Claude backend for the chat cog ... Wire the bot's AI chat pipeline (ai_functions.handle_response) to talk to either OpenAI or the Anthropic Messages API, chosen by one active-config switch. Behaviour on the default "gpt" config is unchanged. constants.py: * guarded `import anthropic` + CLAUDECLIENT (mirrors OPENAICLIENT), netrc machine 'anthropic' / ANTHROPIC_API_KEY; * CLAUDE_LATEST_MODEL / CLAUDE_CHEAP_MODEL (opus-4-8 / haiku-4-5); * AI_CONFIGS + DEFAULT_AI_CONFIG loaded from an optional 3rd element of system_gpt_settings.json (backward compatible - a 2-element file falls back to built-in defaults, active "gpt"). Single switch: CONJURER_AI_CONFIG env > settings "active" > "gpt". ai_functions.py: * provider_generate() dispatches to OpenAI (unchanged openai_call) or the new _anthropic_call() (splits system out, alternating messages, max_tokens, temperature omitted - Opus 4.8 rejects sampling params); * AIError normalises both SDKs' exceptions into one category set so handle_response keeps its single set of in-character error replies; * select_model() reads the active config; legacy "gpt-4o" default auto-maps to the active provider's model so the switch actually changes the backend; * set_active_ai_config()/list_ai_configs() with best-effort persistence back into system_gpt_settings.json index 2. ai_commands.py: * $gadaj_teraz <config> hybrid command (Vykidailo-gated) switches backend at runtime; * graceful guards when OPENAICLIENT is None: personal assistants (OpenAI Assistants API) and DALL-E image gen degrade instead of crashing, so a Claude-only deployment boots. system_gpt_settings.json: add the configs block (gpt/claude/_template) as the collection point for future backends. requirements_bot.txt: add anthropic. bot.env.example: ANTHROPIC_API_KEY + CONJURER_AI_CONFIG. Unit tests cover the message splitter, model selection, config listing, and error mapping. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-09 21:34:59 +02:00
Michal Tuszowski	a6c20a0054	ci: replace broken default workflows with compile/unit/integration CI ... The two scaffold workflows (Python application / Python package) failed on every PR: they installed deps from a non-existent requirements.txt, ran flake8/pytest over the vendored yt_dlp fork (new syntax under the 3.8/3.9 matrix), and collected ad-hoc root scripts — notably test_ai.py, which is an invalid pasted object dump (not Python). - Remove python-app.yml / python-package.yml and the junk root scripts (test.py, test_ai.py, test_time.py) - Add .github/workflows/ci.yml with three PR-check jobs: * compile — py_compile every first-party .py (no deps) * unit — pytest on pure logic (conanjurer_functions, constants) * integration — boot the Flask services and assert the X-Conjurer-Api-Key auth contract (communication_subroutine + conjurer_musician) - Add tests/ suite, pytest.ini (testpaths=tests) and conftest.py (sys.path) Fixes surfaced by the compile gate / needed for the integration job: - conjurer_librarian/search_bot.py + search_bot2.py: f-string reused the same quote ({item["exists"]}) -> SyntaxError on Python < 3.12 - conjurer_musician/media_search_functions.py: made import-safe (env-overridable paths, lazy DB load / mkdir) so the service can be imported and tested off the Pi Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 12:24:30 +02:00