Frontier AI agents exhibit recurring failures in research judgment, design adaptation, backtracking, resource awareness, and instruction following during open-ended research.
Assessed
First pass. Kept atomic with no decomposition: one natural line of support, the shadow-evaluation study. Assessed as supported, credence 0.75, from the preprint abstract. Importance set to 0.45 and contestation to 0.4. Recorded a reading of the single instance.
Assessed Supported
verdict confidence 0.70 · credence 0.75
The claim comes from a 2026 preprint by Kirgis, Kapoor, Narayanan and coauthors. In that study, frontier agents spent six days on the open research questions of two unpublished NeurIPS 2026 submissions, and the papers' original authors graded the results. The agents did all the engineering without help but made little real progress on the research questions. The authors describe five recurring failures: misjudging what counts as publishable, responding uncreatively to flaws in the research design, backtracking poorly from dead ends, poor awareness of resources, and drifting from instructions. A second model and scaffold showed the same failures. The evidence is direct and comes from qualified graders, and the authors released their logs. The pattern also matches wider reports that autonomous research agents carry out tasks better than they judge them. Two limits keep the claim at supported rather than verified. First, the base is small: two case studies, a few models, and qualitative coding of failures. Second, "frontier agents" is a moving target, so the result describes systems as of mid-2026 and may not last as models improve. What would settle it is larger shadow evaluations across more papers and newer models, with failure frequencies reported.
Claim entered the graph