Evidence on whether AI agents can carry out open-ended AI research is thin.
Structured and assessed
Light first pass. Added one argument (the limits of existing evaluation methods) and placed the source passage on it. No subclaims were added because the claim is minor and evaluative. Assessed as supported, credence 0.8. Importance set to 0.35, contestation to 0.25.
Assessed Supported
verdict confidence 0.70 · credence 0.80
The claim holds that the evidence on whether AI agents can do open-ended AI research is limited. "Thin" is a judgment, but the record bears it out. Most published evaluations of AI research agents test narrow, scorable tasks: Kaggle-style competitions (MLE-bench), short engineering tasks against a clock (RE-Bench) and replications of known papers (PaperBench). None of these asks an agent to pick a question, design a study and judge whether the result is worth publishing. The other main line of evidence is AI-written papers submitted to peer review, such as the AI Scientist papers at workshop venues and the Agents4Science conference. Reviewer verdicts are noisy, and they show little about how much of the work the agent itself did. The 2026 "shadow evaluation" study adds a third kind of evidence. Frontier agents were set the central question of an unpublished paper, and the paper's own authors graded what they produced. The agents did all the engineering but made little progress on the research questions. That result covers only two cases. Even with it, direct evidence on open-ended research capability comes from a handful of studies with small samples, so it remains thin. Several large, independent evaluations of the same kind would weaken the claim.
Claim entered the graph