Current evaluations poorly measure AI agents' ability to conduct open-ended research.
Structured and assessed
First pass. Added one argument for and one against, with premises stated in prose and no subclaims, since the claim is evaluative and not important enough for deeper structure. Placed the unplaced passage from Kirgis et al. on the argument for. Assessed as supported, credence 0.75. Importance set to 0.4, contestation to 0.35.
Assessed Supported
verdict confidence 0.70 · credence 0.75
The weight of evidence supports the claim that, as of late 2026, evaluations of AI agents measure open-ended research ability poorly. The best-known benchmarks of AI research and engineering, such as MLE-bench, RE-Bench and PaperBench, score agents against objectives set in advance and checked automatically or by rubric. They leave out the parts of research that a 2026 study by Kirgis, Kapoor, Narayanan and coauthors found agents worst at: judging what result would be worth publishing, redesigning a study creatively, and backing out of dead ends. The other common approach, sending AI-written papers to blind peer review, inherits peer review's well-documented noise. In the NeurIPS consistency experiments, independent committees disagreed on a large share of accepted papers. The case against is that the gap is partial. Long-horizon task suites, studies in which experts grade agents' work, and the newer "shadow evaluations" all probe research ability more directly. But these are small, costly and rarely repeated, and the fact that the field is still building them is itself evidence that standard practice falls short. Because "poorly" is an evaluative word, the claim holds as a broad characterisation, not a precise finding. It would be weakened by the arrival of evaluations of open-ended research that are reliable, repeatable and widely adopted.
Claim entered the graph