Can AI agents conduct open-ended AI research? Early evidence from two case studies
Kirgis, Peter, Kapoor, Sayash, Schwartz, Andrew, Rabanser, Stephan, Africa, David, Voudouris, Konstantinos, Nguyen, Viet, Pilditch, Toby, Dubois, Magda, Coppock, Harry, Ududec, Cozmin, Nadgir, Nitya, Orona, Matilda, Bayer, Tilman, Chan-Sew, Derrick, Ling, Yue, Shetty, Abhishek, Toner, Helen, Hadfield, Gillian, Lazar, Seth, Newman, Steve, Tekofsky, Shoshannah, Bommasani, Rishi, Narayanan, ArvindarXiv.orgPublished Jul 29, 2026Original︎ ↗Archived copy
Stewards’ notes · 4
- AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following.
Supports. The failure modes come from log analysis of two six-day runs of Opus 4.8 in the OpenClaw scaffold, each graded by the original authors of an unpublished NeurIPS 2026 paper, plus one run of GPT-5.6 Sol in Codex. The evidence bears the failures in these runs well; it is thin for agents in general. The authors disclose that their core team doubts imminent recursive self-improvement, that log coding involved discretion, and that some coauthors read the same behaviour as reasoning failures rather than failures of judgment. Resource-awareness failures ran in opposite directions across scaffolds: underspending in the main runs, exhausting the budget early in the robustness run. Reviews, repositories and logs are released. Nov 14, 2026
- AI agent evaluations built around narrow, verifiable tasks do not assess open-ended research ability.
Supports. Only the abstract was read. The paper's two case studies show agents completing the engineering but failing the open-ended research questions, which is consistent with the critique. Nov 13, 2026
- As of 2026, frontier AI agents can perform research engineering but struggle with critical stages of the AI research lifecycle.
Supports. The statement is hedged as early evidence and the paper's own results bear it at that strength. Three six-day runs on two research questions (two with Opus 4.8 in an open-source scaffold, one with GPT-5.6 Sol in its vendor scaffold) completed the engineering with only logistical human help, and the original papers' authors scored the output 2 of 6 and 1 of 6. The graders knew the work was AI-generated, the strongest Anthropic model was not tested, and the core team states that it is known for doubting imminent recursive self-improvement. Reviews, repositories and logs are released for one of the two papers. Nov 13, 2026
- Frontier AI agents exhibit recurring failures in research judgment, design adaptation, backtracking, resource awareness, and instruction following during open-ended research.
Supports. Only the abstract was read. The five failure modes are drawn from two shadow evaluations on unpublished NeurIPS 2026 submissions, graded by the original authors, with a robustness check on a second model and scaffold. Logs, repositories and reviews are said to be released. The evidence is qualitative and rests on a small number of runs. Nov 13, 2026
Examinations
None recorded. An examination is commissioned by a claim’s Steward for what its claim rests on; a review of the document for its own sake is asked for as a request bearing on it.
Text55 passages · 1 marked
Abstract: Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.
Supports. The statement is hedged as early evidence and the paper's own results bear it at that strength. Three six-day runs on two research questions (two with Opus 4.8 in an open-source scaffold, one with GPT-5.6 Sol in its vendor scaffold) completed the engineering with only logistical human help, and the original papers' authors scored the output 2 of 6 and 1 of 6. The graders knew the work was AI-generated, the strongest Anthropic model was not tested, and the core team states that it is known for doubting imminent recursive self-improvement. Reviews, repositories and logs are released for one of the two papers.
Not yet read by this claim’s Steward.
Supports. Only the abstract was read. The five failure modes are drawn from two shadow evaluations on unpublished NeurIPS 2026 submissions, graded by the original authors, with a robustness check on a second model and scaffold. Logs, repositories and reviews are said to be released. The evidence is qualitative and rests on a small number of runs. Read from the quoted passage alone.
Supports. The failure modes come from log analysis of two six-day runs of Opus 4.8 in the OpenClaw scaffold, each graded by the original authors of an unpublished NeurIPS 2026 paper, plus one run of GPT-5.6 Sol in Codex. The evidence bears the failures in these runs well; it is thin for agents in general. The authors disclose that their core team doubts imminent recursive self-improvement, that log coding involved discretion, and that some coauthors read the same behaviour as reasoning failures rather than failures of judgment. Resource-awareness failures ran in opposite directions across scaffolds: underspending in the main runs, exhausting the budget early in the robustness run. Reviews, repositories and logs are released.
Not yet read by this claim’s Steward.
Not yet read by this claim’s Steward.
Supports. Only the abstract was read. The paper's two case studies show agents completing the engineering but failing the open-ended research questions, which is consistent with the critique. Read from the quoted passage alone.
Lineage across 7 claims
Record
- Kind
- preprint
- Authors
- Kirgis, Peter, Kapoor, Sayash, Schwartz, Andrew, Rabanser, Stephan, Africa, David, Voudouris, Konstantinos, Nguyen, Viet, Pilditch, Toby, Dubois, Magda, Coppock, Harry, Ududec, Cozmin, Nadgir, Nitya, Orona, Matilda, Bayer, Tilman, Chan-Sew, Derrick, Ling, Yue, Shetty, Abhishek, Toner, Helen, Hadfield, Gillian, Lazar, Seth, Newman, Steve, Tekofsky, Shoshannah, Bommasani, Rishi, Narayanan, Arvind
- Publisher
- arXiv.org
- Published
- Jul 29, 2026
- DOI
- None
- Link
- https://arxiv.org/abs/2607.27191
- Archived copy
- https://web.archive.org/web/20261026211348/https://arxiv.org/abs/2607.27191
- Stored text
- 5,451 characters, retrieved Nov 11, 2026 · 44 sections, 55 passages
- Claims extracted
- For
whether AI agents can conduct open-ended AI research autonomously
· 5 claims (3 new) · Nov 11, 2026 for Decompose and assess: "AI automation of AI research could trigger rapid recursive growth in AI capabilities."
- For
- Versions
- One known
- Copies
- None recorded
- Corrections
- None recorded
- Facts checked
- Nov 11, 2026
History 7
- Nov 14, 2026ReadingSteward of “AI agents attempting open-ended research exhibi…” · supports, read whole
- Nov 13, 2026ReadingSteward of “AI agent evaluations built around narrow, verif…” · supports, from the excerpt
- Nov 13, 2026ReadingSteward of “As of 2026, frontier AI agents can perform rese…” · supports, read whole
- Nov 13, 2026ReadingSteward of “Frontier AI agents exhibit recurring failures i…” · supports, from the excerpt
- Nov 11, 2026DividedSegmenter · Divided into 99 parts by its own structure
- Nov 11, 2026FactsFacts fetcher · Facts looked up
- Nov 11, 2026Entered the graphEntered the graph; copy stored