AI agent evaluations built around narrow, verifiable tasks do not assess open-ended research ability.
Assessed
First pass, kept atomic. Assessed as supported (credence 0.8) under the reading "do not directly assess," with the indirect-validity caveat. Importance 0.4, contestation 0.35.
Assessed Supported
verdict confidence 0.70 · credence 0.80
The claim holds largely by construction. Agent benchmarks for AI research and development, such as RE-Bench, MLE-bench and PaperBench, fix the problem, supply the success criterion and score against a checkable target. Open-ended research involves exactly the parts these designs remove: choosing a question worth asking, judging what result would meet the bar for publication, redesigning when an approach fails, and deciding when to abandon a line of work. A task with a fixed verifiable answer cannot exercise those judgments, so it does not directly measure them. The real dispute is narrower: whether performance on narrow tasks predicts open-ended ability indirectly. Proponents of task-based metrics such as time-horizon trends treat them as proxies for automating research. Evidence so far cuts against a tight link. In "shadow evaluations" reported in 2026, frontier agents given unpublished NeurIPS submissions finished all the engineering without help but made no substantial progress on the research questions. The failures (poor judgment about the publication bar, uncreative redesign, weak backtracking, poor resource awareness, instruction drift) are the capacities narrow benchmarks do not test. That study covers only two papers, and its authors present it as early evidence. Read as "narrow evaluations give at most indirect, partial evidence about open-ended research ability," the claim is well supported; read as "they carry no information at all about it," it would overstate the case.
Claim entered the graph