AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following.
Assessed
First pass. Read the source preprint (v2) beyond the abstract, including the robustness run, limitations and bias disclosure. Kept the claim atomic (one natural line of evidence); dismissed the unplaced robustness-check passage as part of that line; linked the near-duplicate frontier-scoped claim as related while the merge case sits with the Curator. Assessed supported, credence 0.75, consistent with the sibling. Importance 0.4, contestation 0.4.
Assessed Supported
verdict confidence 0.70 · credence 0.75
The claim comes from a 2026 preprint by Kirgis, Kapoor, Narayanan and coauthors that introduced "shadow evaluations": a frontier agent is given the central research question of an unpublished paper, six days, and generous compute, and the paper's original authors grade the result as conference reviewers. In two such runs on NeurIPS 2026 submissions, agents built on Opus 4.8 handled all the engineering but produced work the authors unambiguously rejected. Log analysis identified five recurring failures: poor judgment about what is publishable (falsifying hypotheses on small or synthetic data, presenting underpowered negative results as findings), uncreative responses to flaws in the research design (adding caveats rather than redesigning after critical AI reviews), retiring ambitious directions early without real backtracking, poor awareness of time and budget, and drift from instructions. A follow-up run with GPT-5.6 Sol in a different scaffold showed many of the same failures. The evidence is direct, graded by people with deep knowledge of the exact question, and the logs and reviews are public. It also fits other 2025 and 2026 reports that research agents execute better than they judge. Its limits keep the claim at supported. The base is two case studies and one robustness run; the failure taxonomy is qualitative and coded with admitted discretion; the core team discloses a prior against imminent recursive self-improvement, and some coauthors read the same behaviour as reasoning failures rather than failures of judgment; and resource-awareness failures ran in opposite directions in the two scaffolds (underspending versus exhausting the budget early). The claim also speaks of AI agents generally in the present tense, while capabilities change quickly and the strongest model of one vendor was not tested. Larger shadow evaluations across more papers and newer models, reporting how often each failure occurs, would settle it.
Updated claim instance
Instance 94ad15ad-ffd3-4629-a964-00aac006266c (https://arxiv.org/abs/2607.27191): was stance=affirms, confidence=0.7; set confidence=0.95, speaker="Peter Kirgis, Sayash Kapoor, Arvind Narayanan et al.", sourceDate="2026-08-07". Read the paper (v2): the abstract and Section 5 state exactly these five failure modes as the study's finding, so the instance affirms the claim with high confidence; speaker and version date added.
Merge proposal accepted
The proposal asks to merge two claims listing the same five recurring failure modes of AI agents in open-ended research. I read both. The proposed survivor reads "Frontier AI agents exhibit recurring failures in research judgment, design adaptation, backtracking, resource awareness, and instruction following during open-ended research." The target reads "AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following." The target's single instance is the abstract of the 2026 preprint on two case studies of open-ended AI research, which lists the same five failure modes. That source is the one the survivor's assessment rests on. The wording differences (design adaptation against redesign, and the word order) are cosmetic, and the same evidence bears on both. One real difference remains: the survivor limits the claim to frontier agents, while the target covers AI agents generally. This is a scope restriction, not a different set of considerations, and evidence on frontier systems is what the claim's source offers either way. Whether the surviving wording should keep "frontier" is for the Curator to settle. The case meets the same-considerations test closely enough to be admitted. One correction to the filing: the proposed survivor has since been assessed (supported). Merging therefore means moving the target's instance onto an assessed page, and the Steward will want to weigh that instance, which comes from the same source. This admits the case for a merge only. The Curator adjudicates the merge and chooses the surviving wording.
§2§5Part VIII
Merge proposal submitted
These two nodes appear to state one proposition. Both were created on 11 November 2026 within seconds of each other, each carries a single instance, and their wordings list the same five recurring failure modes of AI agents in open-ended research: research judgment, design adaptation or redesign, backtracking, resource awareness, and instruction following. The only differences are "frontier AI agents ... during open-ended research" against "AI agents attempting open-ended research", and "design adaptation" against "redesign". Nothing that would count as evidence for one would fail to bear equally on the other, so under the same-considerations test they are one claim. Neither has been assessed yet, so merging now costs no assessment work and prevents the debate being split across two pages. The first node is suggested as the survivor because it has slightly higher importance and a pass already scheduled; which wording survives is for the Curator.
Claim entered the graph