Step 6: tool results

read_page

{
  "url": "https://arxiv.org/html/2607.27191v2",
  "max_chars": 8000
}

{"ok":true,"url":"https://arxiv.org/html/2607.27191v2","text":"Can AI agents conduct open-ended AI research?\nSkip to main content\nReport GitHub Issue\n×\nTitle:\nContent selection saved. Describe the issue below:\nDescription:\nSubmit without GitHub\nSubmit in GitHub\nWhy HTML?\nReport Issue\nBack to Abstract\nDownload PDF\nContents\nReading mode\nColour theme\nAbstract\n1 Introduction\n2 Shadow evaluations: A new method for measuring progress towards automating AI research\n3 The task and the research setup\n4 Results\n4.1 The agents did not produce research at the caliber of a top ML\nconference\n4.2 The agents produced a small number of findings that interested the\nauthors of the original papers\n4.3 The agents committed to unpromising approaches too quickly\n4.4 The agents could not make good use of feedback from AI reviews\n4.5 The agents were capable of all of the engineering steps required\nto conduct the research\n4.6 The agents struggled to effectively present their work\n4.7 We found no significant reward hacking\n4.8 Higher reasoning effort improved the quality of results\n4.9 We tried to use frontier models to improve the harness, with\nlimited success\n5 Log analysis reveals five failure modes\n5.1 Lack of judgment about the bar for high-quality research\n5.2 Lack of creative problem solving\n5.3 Lack of effective backtracking\n5.4 Lack of context awareness\n5.5 Instruction drift\n6 A robustness experiment with Codex and GPT-5.6 Sol Ultra reproduced\nthese failure modes\n7 Limitations\n7.1 Elicitation threats\n7.2 Implications for accelerating AI research\n8 Potential biases\nReferences\n9 Paper Research Questions\n9.1 Persona Cartography\n9.2 TabPFN\n10 Reviews from paper authors\n10.1 TabPFN review from paper authors\nStrengths\nWeaknesses\n10.2 Personas review from paper authors\nStrengths\nWeaknesses\n11 Pre-experiment expectations survey\n12 Comprehensive survey of prior autonomous-AI R&D experiments\n13 CRUX 2 OpenClaw agent scaffold diagram\nLicense: arXiv.org perpetual non-exclusive license\narXiv:2607.27191v2 [cs.AI] 07 Aug 2026\n1]Princeton University\n2]Cornflower Labs\n3]UK AI Security Institute\n4]University of Toronto\n5]Independent\n6]UC Berkeley\n7]Georgetown University (CSET)\n8]Johns Hopkins University\n9]Golden Gate Institute for AI\n10]AI Digest\n11]Stanford University\n\\contribution [*]Equal contribution\n\\contribution [†]CRUX Core Team\n\\metadata [Materials] https://cruxevals.com/crux/can-ai-agents-conduct-research\n\\metadata [Correspondence] pkirgis@princeton.edu , sayashk@princeton.edu\nCan AI agents conduct open-ended AI research?\nEarly evidence from two case studies\nPeter Kirgis\n  \nSayash Kapoor\n  \nAndrew Schwartz\n  \nStephan Rabanser\n  \nDavid Africa\n  \nKonstantinos Voudouris\n  \nViet Nguyen\n  \nToby Pilditch\n  \nMagda Dubois\n  \nHarry Coppock\n  \nCozmin Ududec\n  \nNitya Nadgir\n  \nMatilda Orona\n  \nTilman Bayer\n  \nDerrick Chan-Sew\n  \nYue Ling\n  \nAbhishek Shetty\n  \nHelen Toner\n  \nGillian Hadfield\n  \nSeth Lazar\n  \nSteve Newman\n  \nShoshannah Tekofsky\n  \nRishi Bommasani\n  \nArvind Narayanan\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAugust 24, 2026\nAbstract\nForecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper’s original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today’s agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.\n\\headingfont\nTable of contents\n1 Introduction\nOne of the most consequential open questions about AI capabilities is whether\nAI agents can conduct AI research. Many forecasts of explosive AI progress\nspeculate that AI systems will soon do AI research\nthemselves  ( Kokotajlo et al., 2025 ; Cunningham et al., 2026 ) . This is also the\nexplicit premise of leading AI labs; in June, Anthropic published a post\nentitled “When AI Builds Itself”  ( Favaro and Clark, 2026 ) , and in July,\nOpenAI advertised that their new model, GPT-5.6 Sol, had helped post-train a\nsmaller model, saving researchers multiple weeks  ( OpenAI, 2026a ) . 1 1\n1\nNotably,\nGPT-5.6 Sol’s contribution to Luna’s\npost-training is not mentioned in the 81-page system card  ( OpenAI, 2026b ) . But despite the\nsignificance of this research direction and the attention paid to these claims,\nthe evidence base on whether agents can solve open-ended research questions is\nthin.\nWhat unifies most of this recent work on autonomous AI R&D is its focus on\nverifiable tasks where agents need to improve a fixed, narrow metric.\nBenchmarks ask agents to improve a known metric, and an automatic verifier\nscores the result. Beyond benchmarks, a number of evaluations have shown AI\nagents beating expert human performance in tasks such as optimizing GPT-2 level\nmodels  ( Prime Intellect Team, 2026 ) , using weaker models to train stronger\nones  ( Wen et al., 2026 ) , and optimizing an autoresearch\nharness  ( Weco Team, 2026 ) . 2 2\n2\nSection   12 describes these and many\nother AI R&D experiments that comprise verifiable tasks.\nBut much AI research goes beyond solving verifiable tasks. Agents can’t hill-climb their way into choosing a set of candidate hypotheses, deciding what evidence would settle a research question, or incorporating feedback effectively and recognizing that an approach has failed and the right move is to start over.\nA small number of projects have adopted a different method for AI R&D\nevaluation: they submit AI-generated papers to blind peer-review processes,\nsuch as to AI conferences and workshops. But peer review is a weak measure of\nresearch ability: conference reviewing is overstretched and highly\nstochastic  ( Cortes and Lawrence, 2021 ; Beygelzimer et al., 2023 ; Shah, 2022 ) , and it does not reveal how many AI-generated submissions were\nrejected before the eventual acceptance.\nIn this paper, we evaluate progress towards automating AI research with a new method, which we call\nshadow evaluation. This involves taking the central research question from a\nhigh-quality research paper that is not yet public, tasking a well-resourced\nfrontier agent with answering it, and asking the paper’s original authors to\ngrade the agent’s output as they would a conference submission. The agent\n“shadows” the original study: it works on the same research question as the\noriginal authors, without access to their paper or findings. This design gives\nus open-ended tasks, uncontaminated questions, and reviewers with deep\nexpertise in the exact question being evaluated. We consider shadow\nevaluations complementary to both narrow, verifiable evaluations and blind\nreview evaluations of automated AI research. We hope future research uses all\nthree to explore different facets of measuring progress towards automating AI r","chars":8000,"total_chars":124208,"offset":0,"truncated":true,"next_offset":8000}