RefLint

Inspiration

I'm a researcher at KAUST. Before every submission I hand-check the bibliography, and since LLM-drafted related-work sections became normal I check twice. The failure modes are no longer theoretical: the retracted Surgisphere hydroxychloroquine paper in The Lancet has been cited more than 1,200 times after retraction, and reviewers now routinely find references that simply don't exist. Checking 40 references by hand takes hours; it is exactly the repetitive, judgment-heavy work an agent should carry.

What it does

RefLint is a linter for your bibliography. Point it at a manuscript and the agent parses every claim-citation pair, resolves each reference against the open scholarly record (Crossref + OpenAlex + Semantic Scholar), and answers three questions the eye can't: does this paper exist at all (R001 PHANTOM), has it been retracted (R002 RETRACTED), and does it actually say what your sentence claims it says (R003 UNSUPPORTED — judged with mandatory quoted evidence from the abstract). It renders a terminal report and a markdown report, and exits non-zero on errors, so it drops into CI as a pre-submission gate. On the bundled demo paper it catches a retracted Lancet paper cited as evidence, a plausible-looking reference that exists nowhere, and a real Cell paper cited for a claim its abstract never makes — in one two-minute run.

How we built it

Two Strands agents. The orchestrator (strands.Agent on the Strands Gemini model provider, gemini-flash-lite-latest) drives five tools — load_manuscript, verify_references, search_scholar, judge_claims, submit_report — under a system prompt that forbids any verdict that doesn't trace to tool output. The claim judge is a second Strands agent (gemini-3.5-flash, deliberately a separate free-tier quota bucket) behind the judge_claims tool: it sees only claim + abstract, judges all pairs in one structured-output call, and must quote its evidence line. Verification is deterministic Python: DOI resolution with fuzzy-title fallback across two databases, OpenAlex is_retracted flags (the open Retraction Watch data), and a Semantic Scholar abstract fallback for publisher-restricted records. Every external call ships timeout + retry + disk cache + stale-cache fallback; raw Gemini 429/503s are remapped to Strands' ModelThrottledException so the SDK's exponential backoff absorbs quota spikes. The final report is schema-validated (pydantic) before rendering — the agent literally cannot submit a malformed audit.

Challenges

Free-tier quota was the hard wall: a ~20-request window per model meant an agent that "thinks with many small calls" died mid-audit. The fix was architectural — batch all claim judgments into one structured-output call, split orchestrator and judge across model quota buckets, and remap Gemini's raw 429/503 (whose status text Strands doesn't match) into throttle exceptions the SDK retries. The scholarly record fought back too: OpenAlex doesn't index arXiv DOIs the way papers cite them, and Elsevier abstracts are missing from OpenAlex entirely — solved with a candidate-DOI cascade (Crossref match, OpenAlex second opinion) and an abstract fallback chain. And accusing a reference of being hallucinated needs care: the agent double-checks every suspected phantom with a sharper search before raising R001.

Accomplishments

The full golden path — parse, resolve six references across three APIs, retraction check, batch claim judging, validated report — runs end-to-end live and catches all three planted failures with receipts. A custom Strands Model provider (ReplayModel) replays a recorded real run through the real agent loop and real tools, so judges can reproduce the exact demo with zero keys and zero network: DEMO_MODE=1 uv run reflint audit examples/demo_paper/paper.md. Solo build inside the submission window.

What we learned

An auditing agent is only as trustworthy as the worst tool response it's willing to repeat, so the engineering went into honesty plumbing: similarity scores and stale-cache flags surface in tool output, the judge must quote or say "no relevant line", and prompts push the model to double-check before accusing. Strands' model-agnostic Model interface turned out to be the killer feature — the same agent code runs Gemini live and a recorded replay offline.

What's next

BibTeX/LaTeX ingestion (the parser is already isolated behind one tool), a --fix mode that proposes replacement citations from the verified record, and a GitHub Action so R001/R002/R003 block a paper repo's merge the way a failing test does. Deploying the orchestrator on Amazon Bedrock AgentCore is a natural next step — the Strands agent is provider-agnostic, so it's a model-provider swap, not a rewrite.

Built with

strands-agents (SDK) · strands-agents-tools · Python 3.12 · uv · Gemini (gemini-flash-lite-latest + gemini-3.5-flash via the Strands Gemini model provider) · Crossref REST API · OpenAlex API · Semantic Scholar Graph API · pydantic · rich · httpx

Disclosures

Built during the submission period. An AI coding agent (Claude) wrote the implementation under my direction; design, thresholds, the planted-failure demo manuscript, and verification of every run are mine. The demo paper is synthetic with seeded failures (marked DEMO:) so the demo is reproducible; DEMO_MODE replays recorded real API/model responses and is clearly labeled. Full log in docs/ai-usage.md.

Built With

• crossref
• gemini
• openalex
• python
• rich
• semantic-scholar
• strands-agents
• uv