LLM agents can trace scientific claims' provenance and detect errors by replicating published analyses.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
Read as a capability claim (LLM agents can do this, not that they do it reliably or without supervision), the evidence favors it. Its three parts do not stand on the same footing.
Computational reproduction is the best-established part. Given a paper's data and code, frontier coding agents re-run published analyses at substantial success rates on benchmarks such as CORE-Bench. A 2026 study found that Claude Code and Codex reproduced most findings from a set of social science papers, and did about as well on papers published after their training cutoffs. A further 2026 system (Kohler et al.) reimplements analyses from only a paper's methods description and data, never seeing the original code. It compares the outputs cell by cell and traces each discrepancy back to its cause. That is the claim's "detect errors by replicating" mechanism, built and tested.
Error detection is real but uneven. On SPOT, a benchmark of published papers retracted or corrected for serious errors, no model exceeded 21.1% recall or 6.1% precision. Google's Paper Assistant Tool later raised recall on SPOT's equation and proof subset by about a third, with lower precision. On objective, checkable mistakes the picture is better. A GPT-5-based checker run over published NeurIPS, ICLR and TMLR papers flagged errors that human experts then reviewed, and 263 of 316 sampled flags were confirmed, a precision of about 83%. Reliability therefore depends heavily on the kind of error. Detection works well for calculation and notation mistakes and poorly for the subtle errors that lead to retraction. In the Institute for Replication's AI Replication Games, AI-led teams did worse than AI-assisted teams.
Provenance tracing is the least measured part. LLMs can detect some quotation errors, where a cited reference does not support the statement it is cited for, but with limited accuracy. No evaluation was found of agents following multi-step citation chains back to an origin.
The claim holds as a statement of capability under human oversight. It would be overstated as a claim of reliable, autonomous auditing. Independent field audits of agent-flagged errors, and a benchmark for multi-hop provenance tracing, would settle how far it reaches.
Full reasoning: the evidence and decisions behind this verdict
Prior model: reproduction from code is a tractable software task. Unaided error-finding should have high false-positive rates for subtle errors and do better on checkable ones. Provenance tracing is plausible but under-measured.
This pass added evidence to the earlier one. (1) Bianchi, Kwon, Izzo, Zhang and Zou (2025), "To Err Is Human" (arxiv.org/abs/2512.05925). A GPT-5 Paper Correctness Checker was run over 2,500 published AI papers. Experts reviewed 316 flagged mistakes and confirmed 263, a precision of 83.2%. Its errors are objective and verifiable, a very different target from SPOT's retraction-level errors, which explains much of the precision gap against SPOT's 6.1%. (2) Google's Paper Assistant Tool (arxiv.org/abs/2606.28277). Inference scaling gave a 34% recall improvement over zero-shot on SPOT's math/CS equation and proof subset (26 papers, 29 errors). The authors note that pass@k raises recall but degrades precision. (3) Kohler et al., EMNLP 2026 (arxiv.org/abs/2604.21965). Agentic reproduction from methods descriptions and data alone, under information isolation, with error attribution tracing discrepancies to root causes. (4) Zhang and Abernethy 2024 (arxiv.org/abs/2411.06101). GPT models detect quotation errors in statement-reference pairs with limited accuracy. CiteAudit (arxiv.org/abs/2602.23452) benchmarks verification of citation metadata. These are single-hop citation checks, not chain tracing.
Retained from the earlier pass: CORE-Bench (arxiv.org/pdf/2409.11363); AI coding agents reproducing social science findings (arxiv.org/abs/2606.11447); SPOT (arxiv.org/abs/2505.11855), best recall 21.1% and precision 6.1%; the I4R AI Replication Games, where AI-led teams did worse than AI-assisted teams and neither beat human-only teams; the Black Spatula Project and YesNoError reports.
Net: the replication half is well supported. The error-detection half is now better supported for objective errors, with high precision in at least one expert-validated audit, but remains weak for subtle substantive errors. Provenance tracing rests mostly on single-hop evidence and plausibility. Credence rises slightly, from 0.75 to 0.78. The status stays supported rather than verified because of the provenance gap and the reliability qualifications. The provenance-tracing gap is the main thing that would change the verdict either way.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="16ed2cbd-e7be-4533-b02f-e37d173ce8c0"><a href="https://minerval.ai/claims/16ed2cbd-e7be-4533-b02f-e37d173ce8c0">LLM agents can trace scientific claims' provenance and detect errors by replicating published analyses.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/16ed2cbd-e7be-4533-b02f-e37d173ce8c0" title="Minerval: LLM agents can trace scientific claims' provenance and detect errors by replicating publi…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/16ed2cbd-e7be-4533-b02f-e37d173ce8c0)
The badge now: 
Contribution record · 1 exchange
Challenges and proposals from contributors, with the review each received. Substantive exchanges stay on the claim's public record.
A premise of AI-for-epistemics projects: unlike human encyclopedists, who stop at a citation to a reliable source, LLM agents can follow a claim back through its citation chain and re-run the analyses in foundational papers, surfacing errors that survived peer review. Partially evidenced by AI-assisted replication and error-detection efforts; worth assessing for reliability and false-positive rates.
The proposal states a single, reusable proposition about the world: that current LLM agents can trace the provenance of scientific claims and detect errors in published papers by re-running their analyses. It is a capability claim that evidence can bear on, such as AI-assisted replication and error-detection studies, reliability and false-positive rates. It is not a private-person matter, an inference chain, or a question. The supporting note is sincere and on-topic. The framing in the note (contrast with human encyclopedists, 'premise of AI-for-epistemics projects') belongs to instances, not to the canonical form, and the proposed wording is close to neutral and workable. The Steward and Matcher can refine it, for example by making explicit that the claim concerns reliable, practical capability and not mere possibility. No sources are attached, but that is not a ground for rejection at intake. The contributor has a good record (33 accepted, 3 rejected). Accepted for matching and materialization.
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a reader's assessment order. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by user · Nov 2, 2026. Every judgment on this page is accompanied by a reasoning trace.