Librarian: survive transient Crossref failures instead of losing the search

Field report: one httpx ReadTimeout inside habanero surfaced as
'Search <uuid> crashed', and the worker's crash handler then FORGOT the
request - so an expensive search vanished and the user was told it was
eaten, because a public API blinked once.

Two defences:
* Every habanero call goes through _crossref_call, which retries with
  linear backoff (CONJURER_CROSSREF_ATTEMPTS, default 4; backoff
  CONJURER_CROSSREF_BACKOFF, 5s). habanero wraps httpx errors in a plain
  RuntimeError so we can't filter narrowly - retries are simply bounded
  and the last error is re-raised. They now also run via asyncio.to_thread,
  so a slow Crossref no longer blocks the worker's event loop.
* A crashed search is no longer dropped on the first failure: the attempt
  count is persisted with the request and the search is requeued (keeping
  any checkpoint, so a crashed DB scan resumes rather than restarts) until
  CONJURER_SEARCH_MAX_ATTEMPTS (default 3). It stays 'queued' for the
  bot's watchdog while retrying, and only after the cap is it forgotten.

Also: scrape_bot's 'Got blocked' is routine sci-hub behaviour (it backs off
an hour and carries on) - log it as WARNING, not ERROR, so it stops looking
like a fault when scanning for real problems.

Tests: retry-then-succeed, bounded re-raise, no retry on success, the
persisted attempt counter, and forget-on-give-up. Suite: 58 unit + 70
integration green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests/integration/test_librarian_crossref_retry.py

@@ -0,0 +1,102 @@
+"""Integration: transient Crossref failures must not destroy a search.
+
+Field report: a single httpx ReadTimeout inside habanero surfaced as
+"Search <uuid> crashed", and the worker then FORGOT the search - so an
+expensive query vanished and the user got told it was eaten, all because a
+public API blinked. These pin the two defences: retry each Crossref call, and
+retry the whole search a bounded number of times before giving up.
+"""
+import logging
+import sys
+import types
+
+import pytest
+
+if "habanero" not in sys.modules:
+    _habanero = types.ModuleType("habanero")
+    _habanero.Crossref = object
+    sys.modules["habanero"] = _habanero
+
+import conjurer_librarian as lib  # noqa: E402
+from durable_queue import DiskQueue  # noqa: E402
+
+_LOG = logging.getLogger("test-crossref-retry")
+_LOG.addHandler(logging.NullHandler())
+
+
+@pytest.fixture(autouse=True)
+def _no_backoff(monkeypatch):
+    monkeypatch.setattr(lib.time, "sleep", lambda _s: None)
+
+
+@pytest.fixture
+def state(tmp_path, monkeypatch):
+    monkeypatch.setattr(lib, "_requests", DiskQueue(str(tmp_path / "req")))
+    monkeypatch.setattr(lib, "_checkpoints", DiskQueue(str(tmp_path / "cp")))
+    return None
+
+
+def test_crossref_call_retries_then_succeeds():
+    calls = {"n": 0}
+
+    def flaky(**_kwargs):
+        calls["n"] += 1
+        if calls["n"] < 3:
+            raise RuntimeError("The read operation timed out")
+        return {"message": {"total-results": 1, "items": []}}
+
+    result = lib._crossref_call(_LOG, "works", flaky, query="q")
+    assert result["message"]["total-results"] == 1
+    assert calls["n"] == 3  # two failures survived
+
+
+def test_crossref_call_reraises_after_exhausting_attempts(monkeypatch):
+    monkeypatch.setattr(lib, "CROSSREF_ATTEMPTS", 2)
+    calls = {"n": 0}
+
+    def always_fails(**_kwargs):
+        calls["n"] += 1
+        raise RuntimeError("The read operation timed out")
+
+    with pytest.raises(RuntimeError):
+        lib._crossref_call(_LOG, "works", always_fails, query="q")
+    assert calls["n"] == 2  # bounded, not infinite
+
+
+def test_crossref_call_does_not_retry_a_success():
+    calls = {"n": 0}
+
+    def ok(**_kwargs):
+        calls["n"] += 1
+        return "fine"
+
+    assert lib._crossref_call(_LOG, "works", ok, query="q") == "fine"
+    assert calls["n"] == 1
+
+
+def test_attempt_counter_persists_and_bounds_retries(state):
+    # Mirrors what the worker does on a crash: bump the persisted attempt count
+    # and keep the request until SEARCH_MAX_ATTEMPTS is reached.
+    uuid = "u-crash"
+    lib._requests.put(uuid, {"query": "q", "deep_search": False, "callback": ""})
+
+    for expected in (1, 2):
+        stored = lib._requests.get(uuid) or {}
+        attempts = int(stored.get("attempts", 0)) + 1
+        assert attempts == expected
+        stored["attempts"] = attempts
+        lib._requests.put(uuid, stored)
+
+    assert lib._requests.get(uuid)["attempts"] == 2
+    # A third crash reaches the default cap (3) -> the search is forgotten.
+    assert 3 >= lib.SEARCH_MAX_ATTEMPTS
+    lib._forget_search(uuid)
+    assert not lib._requests.contains(uuid)
+
+
+def test_forget_search_clears_request_and_checkpoint(state):
+    lib._requests.put("u-x", {"query": "q", "deep_search": False})
+    lib._checkpoints.put("u-x", {"dois": {}, "found": [], "positions": {}})
+    lib._forget_search("u-x")
+    assert not lib._requests.contains("u-x")
+    assert not lib._checkpoints.contains("u-x")