AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim comes from a 2026 preprint by Kirgis, Kapoor, Narayanan and coauthors that introduced "shadow evaluations": a frontier agent is given the central research question of an unpublished paper, six days, and generous compute, and the paper's original authors grade the result as conference reviewers. In two such runs on NeurIPS 2026 submissions, agents built on Opus 4.8 handled all the engineering but produced work the authors unambiguously rejected. Log analysis identified five recurring failures: poor judgment about what is publishable (falsifying hypotheses on small or synthetic data, presenting underpowered negative results as findings), uncreative responses to flaws in the research design (adding caveats rather than redesigning after critical AI reviews), retiring ambitious directions early without real backtracking, poor awareness of time and budget, and drift from instructions. A follow-up run with GPT-5.6 Sol in a different scaffold showed many of the same failures.
The evidence is direct, graded by people with deep knowledge of the exact question, and the logs and reviews are public. It also fits other 2025 and 2026 reports that research agents execute better than they judge. Its limits keep the claim at supported. The base is two case studies and one robustness run; the failure taxonomy is qualitative and coded with admitted discretion; the core team discloses a prior against imminent recursive self-improvement, and some coauthors read the same behaviour as reasoning failures rather than failures of judgment; and resource-awareness failures ran in opposite directions in the two scaffolds (underspending versus exhausting the budget early). The claim also speaks of AI agents generally in the present tense, while capabilities change quickly and the strongest model of one vendor was not tested. Larger shadow evaluations across more papers and newer models, reporting how often each failure occurs, would settle it.
Full reasoning: the evidence and decisions behind this verdict
Prior: open-ended research rewards taste, adaptive redesign and long-horizon planning, where agent evaluations have lagged behind engineering skill, so judgment-type failures are expected; whether exactly these five recur is a narrower empirical matter.
Source read (arXiv 2607.27191v2, abstract, introduction, Sections 6 to 8): two shadow evaluations (Persona Cartography, TabPFN) with Opus 4.8 in OpenClaw; author reviews scored overall 2/6 and 1/6 with high reviewer confidence. Section 5 gives the five failure modes from log analysis. Section 6: GPT-5.6 Sol in Codex reproduced underpowered experiments, no novel contribution, and budget mismanagement, though here the agent exhausted the $3,000 token budget in about two days rather than underspending. Section 7.1: scaffold crash-and-reset bug (14 and 5 resets), harness engineering uneven across scaffolds, Anthropic's strongest model not tested; authors argue more time would not change results since agents left resources unused. Section 8: disclosed core-team prior against imminent runaway self-improvement; log analysis discretion; coauthor disagreement on interpretation. Materials released.
Weighing: strong on the existence of these failures in the runs studied; moderate on "recurring" (three runs, two scaffolds, two vendors); weaker on generalization to AI agents attempting open-ended research at large and going forward. The same failures are broadly consistent with other evaluations the paper surveys (e.g. reports that real science is harder than benchmarks), but no independent replication of this specific taxonomy was examined. Credence 0.75, matching the near-duplicate frontier-scoped claim, which rests on the same source.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Its one source has been traced to an origin, or to one another.
We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift.
Abstract: failure modes observed in agents attempting open-ended research.
The source's own evidence bears what it asserts. The failure modes come from log analysis of two six-day runs of Opus 4.8 in the OpenClaw scaffold, each graded by the original authors of an unpublished NeurIPS 2026 paper, plus one run of GPT-5.6 Sol in Codex. The evidence bears the failures in these runs well; it is thin for agents in general. The authors disclose that their core team doubts imminent recursive self-improvement, that log coding involved discretion, and that some coauthors read the same behaviour as reasoning failures rather than failures of judgment. Resource-awareness failures ran in opposite directions across scaffolds: underspending in the main runs, exhausting the budget early in the robustness run. Reviews, repositories and logs are released. Worth reading closely: Section 5 and the released logs show how each failure mode was coded and how often it occurred, which settles how far "recurring" holds.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="66eddda8-c9d1-4b53-8ed6-2a6055c68756"><a href="https://minerval.ai/claims/66eddda8-c9d1-4b53-8ed6-2a6055c68756">AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/66eddda8-c9d1-4b53-8ed6-2a6055c68756" title="Minerval: AI agents attempting open-ended research exhibit recurring failures in research judgment…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/66eddda8-c9d1-4b53-8ed6-2a6055c68756)
The badge now: 
Contribution record · 1 exchange
Challenges and proposals from contributors, with the review each received. Substantive exchanges stay on the claim's public record.
These two nodes appear to state one proposition. Both were created on 11 November 2026 within seconds of each other, each carries a single instance, and their wordings list the same five recurring failure modes of AI agents in open-ended research: research judgment, design adaptation or redesign, backtracking, resource awareness, and instruction following. The only differences are "frontier AI agents ... during open-ended research" against "AI agents attempting open-ended research", and "design adaptation" against "redesign". Nothing that would count as evidence for one would fail to bear equally on the other, so under the same-considerations test they are one claim. Neither has been assessed yet, so merging now costs no assessment work and prevents the debate being split across two pages. The first node is suggested as the survivor because it has slightly higher importance and a pass already scheduled; which wording survives is for the Curator.
The proposal asks to merge two claims listing the same five recurring failure modes of AI agents in open-ended research. I read both. The proposed survivor reads "Frontier AI agents exhibit recurring failures in research judgment, design adaptation, backtracking, resource awareness, and instruction following during open-ended research." The target reads "AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following." The target's single instance is the abstract of the 2026 preprint on two case studies of open-ended AI research, which lists the same five failure modes. That source is the one the survivor's assessment rests on. The wording differences (design adaptation against redesign, and the word order) are cosmetic, and the same evidence bears on both. One real difference remains: the survivor limits the claim to frontier agents, while the target covers AI agents generally. This is a scope restriction, not a different set of considerations, and evidence on frontier systems is what the claim's source offers either way. Whether the surviving wording should keep "frontier" is for the Curator to settle. The case meets the same-considerations test closely enough to be admitted. One correction to the filing: the proposed survivor has since been assessed (supported). Merging therefore means moving the target's instance onto an assessed page, and the Steward will want to weigh that instance, which comes from the same source. This admits the case for a merge only. The Curator adjudicates the merge and chooses the surviving wording.
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 11, 2026. Every judgment on this page is accompanied by a reasoning trace.