AI agent evaluations built around narrow, verifiable tasks do not assess open-ended research ability.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim holds largely by construction. Agent benchmarks for AI research and development, such as RE-Bench, MLE-bench and PaperBench, fix the problem, supply the success criterion and score against a checkable target. Open-ended research involves exactly the parts these designs remove: choosing a question worth asking, judging what result would meet the bar for publication, redesigning when an approach fails, and deciding when to abandon a line of work. A task with a fixed verifiable answer cannot exercise those judgments, so it does not directly measure them.
The real dispute is narrower: whether performance on narrow tasks predicts open-ended ability indirectly. Proponents of task-based metrics such as time-horizon trends treat them as proxies for automating research. Evidence so far cuts against a tight link. In "shadow evaluations" reported in 2026, frontier agents given unpublished NeurIPS submissions finished all the engineering without help but made no substantial progress on the research questions. The failures (poor judgment about the publication bar, uncreative redesign, weak backtracking, poor resource awareness, instruction drift) are the capacities narrow benchmarks do not test. That study covers only two papers, and its authors present it as early evidence. Read as "narrow evaluations give at most indirect, partial evidence about open-ended research ability," the claim is well supported; read as "they carry no information at all about it," it would overstate the case.
Full reasoning: the evidence and decisions behind this verdict
The claim was judged first from the definitions involved. Narrow, verifiable tasks specify the goal and the success criterion. Open-ended research ability is largely the ability to set goals and criteria under uncertainty. So such evaluations leave out the core of what they would need to measure. That makes the claim close to analytic when "assess" means direct measurement.
What could count against it: (1) narrow task suites may correlate with, or predict, open-ended performance, so that they assess it indirectly; (2) some benchmarks (PaperBench, MLE-bench) cover long, multi-step engineering workflows that make up part of research. Neither point shows that the open-ended components are measured. On (1), the Kirgis et al. 2026 preprint (arXiv 2607.27191) gives direct, if small-sample (two papers), evidence of a gap: agents that would do well on engineering-style tasks failed on the research questions, as judged by the papers' own authors, and a second model and scaffold reproduced the failures. That source is also the origin of this wording. It motivates its own alternative method, so it has a methodological stake in the critique, but the critique matches how the benchmarks are designed.
The claim is evaluative in wording, but its core is close to empirical. The credence of 0.8 reflects the reading "do not directly assess." The residual doubt is about whether the words imply zero indirect information. No decomposition was made because the claim turns on the construction of the benchmarks plus one empirical question about indirect validity, which the assessment covers.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Its one source has been traced to an origin, or to one another.
Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research
Critique of existing AI R&D benchmarks.
The source's own evidence bears what it asserts. Only the abstract was read. The paper's two case studies show agents completing the engineering but failing the open-ended research questions, which is consistent with the critique.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="78d6c0a0-a6d8-4b57-89eb-f193be0c0e0e"><a href="https://minerval.ai/claims/78d6c0a0-a6d8-4b57-89eb-f193be0c0e0e">AI agent evaluations built around narrow, verifiable tasks do not assess open-ended research ability.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/78d6c0a0-a6d8-4b57-89eb-f193be0c0e0e" title="Minerval: AI agent evaluations built around narrow, verifiable tasks do not assess open-ended resea…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/78d6c0a0-a6d8-4b57-89eb-f193be0c0e0e)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 11, 2026. Every judgment on this page is accompanied by a reasoning trace.