Current evaluations poorly measure AI agents' ability to conduct open-ended research.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The weight of evidence supports the claim that, as of late 2026, evaluations of AI agents measure open-ended research ability poorly. The best-known benchmarks of AI research and engineering, such as MLE-bench, RE-Bench and PaperBench, score agents against objectives set in advance and checked automatically or by rubric. They leave out the parts of research that a 2026 study by Kirgis, Kapoor, Narayanan and coauthors found agents worst at: judging what result would be worth publishing, redesigning a study creatively, and backing out of dead ends. The other common approach, sending AI-written papers to blind peer review, inherits peer review's well-documented noise. In the NeurIPS consistency experiments, independent committees disagreed on a large share of accepted papers.
The case against is that the gap is partial. Long-horizon task suites, studies in which experts grade agents' work, and the newer "shadow evaluations" all probe research ability more directly. But these are small, costly and rarely repeated, and the fact that the field is still building them is itself evidence that standard practice falls short. Because "poorly" is an evaluative word, the claim holds as a broad characterisation, not a precise finding. It would be weakened by the arrival of evaluations of open-ended research that are reliable, repeatable and widely adopted.
Full reasoning: the evidence and decisions behind this verdict
Prior: the claim is a description of the evaluation landscape, and it fits what is generally known. The widely cited agent R&D benchmarks (MLE-bench, Kaggle-style tasks; RE-Bench, which runs 8-hour ML engineering tasks with a scoring function; PaperBench, which replicates existing papers against a rubric) all fix the research question and the success metric in advance. Peer review as a test has known reliability problems (the NeurIPS 2014 and 2021 consistency experiments). The Sakana "AI Scientist" workshop acceptance was widely discussed as weak evidence for the same reason.
Source: the one instance, arXiv 2607.27191 (Kirgis et al., 2026; 24 authors including people from Princeton, the UK AISI and Stanford), states the claim in its abstract. The authors have a mild interest: they use the critique to motivate their own method, shadow evaluations. Their two case studies showed agents finishing the engineering but failing at research judgment. That fits the claim, since narrow benchmarks would have credited the engineering. However, two case studies are a small sample.
Against: long-horizon measurements (METR time horizons), expert-graded studies and replication benchmarks do capture parts of research ability. That is a reason to call the evaluations incomplete rather than worthless, and it lowers the credence somewhat. It does not overturn the overall characterisation.
This pass did not survey every 2026 benchmark, which leaves residual uncertainty. Credence: 0.75.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Its one source has been traced to an origin, or to one another.
Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality.
Abstract: critique of existing ways to measure AI R&D automation, motivating 'shadow evaluations'.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="6d318e34-dfe2-4f86-a5d8-d7f71179125d"><a href="https://minerval.ai/claims/6d318e34-dfe2-4f86-a5d8-d7f71179125d">Current evaluations poorly measure AI agents' ability to conduct open-ended research.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/6d318e34-dfe2-4f86-a5d8-d7f71179125d" title="Minerval: Current evaluations poorly measure AI agents' ability to conduct open-ended research." width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/6d318e34-dfe2-4f86-a5d8-d7f71179125d)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 11, 2026. Every judgment on this page is accompanied by a reasoning trace.