Field report: one httpx ReadTimeout inside habanero surfaced as
'Search <uuid> crashed', and the worker's crash handler then FORGOT the
request - so an expensive search vanished and the user was told it was
eaten, because a public API blinked once.
Two defences:
* Every habanero call goes through _crossref_call, which retries with
linear backoff (CONJURER_CROSSREF_ATTEMPTS, default 4; backoff
CONJURER_CROSSREF_BACKOFF, 5s). habanero wraps httpx errors in a plain
RuntimeError so we can't filter narrowly - retries are simply bounded
and the last error is re-raised. They now also run via asyncio.to_thread,
so a slow Crossref no longer blocks the worker's event loop.
* A crashed search is no longer dropped on the first failure: the attempt
count is persisted with the request and the search is requeued (keeping
any checkpoint, so a crashed DB scan resumes rather than restarts) until
CONJURER_SEARCH_MAX_ATTEMPTS (default 3). It stays 'queued' for the
bot's watchdog while retrying, and only after the cap is it forgotten.
Also: scrape_bot's 'Got blocked' is routine sci-hub behaviour (it backs off
an hour and carries on) - log it as WARNING, not ERROR, so it stops looking
like a fault when scanning for real problems.
Tests: retry-then-succeed, bounded re-raise, no retry on success, the
persisted attempt counter, and forget-on-give-up. Suite: 58 unit + 70
integration green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
