Scientific Claim Verifier
An open-source engine that verifies each cited claim in scientific text against its actual source and returns a grounded verdict with replayable provenance.
Why I built it
AI now generates hypotheses, molecules, literature reviews, and entire papers far faster than humans can check them. As that volume explodes, the bottleneck shifts from producing scientific content to verifying it. We need a way to confirm that scientific claims actually hold against their sources — and to make that confirmation reproducible and traceable.
This engine is my attempt at that missing assurance layer: independent, deterministic, traceable claim-to-source verification for an AI-saturated literature. It is the first open-source piece of the verification infrastructure I am building.
How it works
Free-form scientific text is decomposed into citation-anchored claims by Claude. Each cited source is resolved through an API cascade (bibliography DOI → CrossRef → OpenAlex → PubMed) and its full text fetched (open-access URL → PMC → Europe PMC → Unpaywall PDF). The text is split into IMRAD-aware chunks, and the most relevant passages are selected by lexical BM25 under a token budget. A router then picks one of five verifier modes by retrieval depth, and entailment is judged by prompting Claude — no dedicated NLI model, no embeddings.
The design choice that matters most: everything that does not strictly need a model stays LLM-free and deterministic — chunk selection, numeric consistency checks, and aggregation — so the same input yields the same verdict. Every step emits a provenance record with input/output SHA-256 hashes, model id, and token counts, producing a fully replayable audit chain.
- Extract citation-anchored claims (Claude).
- Resolve citations via a multi-source cascade (DOI, CrossRef, OpenAlex, PubMed).
- Enrich metadata (PubMed, Europe PMC open-access discovery).
- Fetch full text (OA URL, PMC, Europe PMC, Unpaywall PDF).
- Chunk and BM25-select passages (deterministic IMRAD sectioning, token-budgeted).
- Verify: a router selects one of five modes by retrieval depth.
- Audit-trail fallback: surface the searched passages when no quote is found.
- Deterministic numeric checks in pure Python (OR/CI consistency, p-value/CI null-crossing).
- Emit
report.json+provenance.jsonl.
Results
On the SciFact dev set, the verifier reaches an F1 of 0.92 on oracle inputs (3-class), against 0.62 for a naive direct-LLM baseline committed in the repo: roughly a 30-point gap on this verifier-only setup. A regression guard fails the build if SciFact F1 drops more than 1% below the committed baseline.
Benchmark numbers are author-reported, self-run, and committed to the repository. The SciFact figure is verifier-only (oracle inputs), not the full extract-and-resolve pipeline.
Stack
Apache 2.0 licensed. Runs as a CLI, a FastAPI HTTP service, or an MCP server for Claude Desktop and the Agent SDK.