Step 6: tool results

read_page

{
  "url": "https://arxiv.org/html/2607.27191v2",
  "max_chars": 12000
}

{"ok":true,"url":"https://arxiv.org/html/2607.27191v2","text":"Can AI agents conduct open-ended AI research?\nSkip to main content\nReport GitHub Issue\n×\nTitle:\nContent selection saved. Describe the issue below:\nDescription:\nSubmit without GitHub\nSubmit in GitHub\nWhy HTML?\nReport Issue\nBack to Abstract\nDownload PDF\nContents\nReading mode\nColour theme\nAbstract\n1 Introduction\n2 Shadow evaluations: A new method for measuring progress towards automating AI research\n3 The task and the research setup\n4 Results\n4.1 The agents did not produce research at the caliber of a top ML\nconference\n4.2 The agents produced a small number of findings that interested the\nauthors of the original papers\n4.3 The agents committed to unpromising approaches too quickly\n4.4 The agents could not make good use of feedback from AI reviews\n4.5 The agents were capable of all of the engineering steps required\nto conduct the research\n4.6 The agents struggled to effectively present their work\n4.7 We found no significant reward hacking\n4.8 Higher reasoning effort improved the quality of results\n4.9 We tried to use frontier models to improve the harness, with\nlimited success\n5 Log analysis reveals five failure modes\n5.1 Lack of judgment about the bar for high-quality research\n5.2 Lack of creative problem solving\n5.3 Lack of effective backtracking\n5.4 Lack of context awareness\n5.5 Instruction drift\n6 A robustness experiment with Codex and GPT-5.6 Sol Ultra reproduced\nthese failure modes\n7 Limitations\n7.1 Elicitation threats\n7.2 Implications for accelerating AI research\n8 Potential biases\nReferences\n9 Paper Research Questions\n9.1 Persona Cartography\n9.2 TabPFN\n10 Reviews from paper authors\n10.1 TabPFN review from paper authors\nStrengths\nWeaknesses\n10.2 Personas review from paper authors\nStrengths\nWeaknesses\n11 Pre-experiment expectations survey\n12 Comprehensive survey of prior autonomous-AI R&D experiments\n13 CRUX 2 OpenClaw agent scaffold diagram\nLicense: arXiv.org perpetual non-exclusive license\narXiv:2607.27191v2 [cs.AI] 07 Aug 2026\n1]Princeton University\n2]Cornflower Labs\n3]UK AI Security Institute\n4]University of Toronto\n5]Independent\n6]UC Berkeley\n7]Georgetown University (CSET)\n8]Johns Hopkins University\n9]Golden Gate Institute for AI\n10]AI Digest\n11]Stanford University\n\\contribution [*]Equal contribution\n\\contribution [†]CRUX Core Team\n\\metadata [Materials] https://cruxevals.com/crux/can-ai-agents-conduct-research\n\\metadata [Correspondence] pkirgis@princeton.edu , sayashk@princeton.edu\nCan AI agents conduct open-ended AI research?\nEarly evidence from two case studies\nPeter Kirgis\n  \nSayash Kapoor\n  \nAndrew Schwartz\n  \nStephan Rabanser\n  \nDavid Africa\n  \nKonstantinos Voudouris\n  \nViet Nguyen\n  \nToby Pilditch\n  \nMagda Dubois\n  \nHarry Coppock\n  \nCozmin Ududec\n  \nNitya Nadgir\n  \nMatilda Orona\n  \nTilman Bayer\n  \nDerrick Chan-Sew\n  \nYue Ling\n  \nAbhishek Shetty\n  \nHelen Toner\n  \nGillian Hadfield\n  \nSeth Lazar\n  \nSteve Newman\n  \nShoshannah Tekofsky\n  \nRishi Bommasani\n  \nArvind Narayanan\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAffiliation:  [\nAugust 24, 2026\nAbstract\nForecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper’s original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today’s agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.\n\\headingfont\nTable of contents\n1 Introduction\nOne of the most consequential open questions about AI capabilities is whether\nAI agents can conduct AI research. Many forecasts of explosive AI progress\nspeculate that AI systems will soon do AI research\nthemselves  ( Kokotajlo et al., 2025 ; Cunningham et al., 2026 ) . This is also the\nexplicit premise of leading AI labs; in June, Anthropic published a post\nentitled “When AI Builds Itself”  ( Favaro and Clark, 2026 ) , and in July,\nOpenAI advertised that their new model, GPT-5.6 Sol, had helped post-train a\nsmaller model, saving researchers multiple weeks  ( OpenAI, 2026a ) . 1 1\n1\nNotably,\nGPT-5.6 Sol’s contribution to Luna’s\npost-training is not mentioned in the 81-page system card  ( OpenAI, 2026b ) . But despite the\nsignificance of this research direction and the attention paid to these claims,\nthe evidence base on whether agents can solve open-ended research questions is\nthin.\nWhat unifies most of this recent work on autonomous AI R&D is its focus on\nverifiable tasks where agents need to improve a fixed, narrow metric.\nBenchmarks ask agents to improve a known metric, and an automatic verifier\nscores the result. Beyond benchmarks, a number of evaluations have shown AI\nagents beating expert human performance in tasks such as optimizing GPT-2 level\nmodels  ( Prime Intellect Team, 2026 ) , using weaker models to train stronger\nones  ( Wen et al., 2026 ) , and optimizing an autoresearch\nharness  ( Weco Team, 2026 ) . 2 2\n2\nSection   12 describes these and many\nother AI R&D experiments that comprise verifiable tasks.\nBut much AI research goes beyond solving verifiable tasks. Agents can’t hill-climb their way into choosing a set of candidate hypotheses, deciding what evidence would settle a research question, or incorporating feedback effectively and recognizing that an approach has failed and the right move is to start over.\nA small number of projects have adopted a different method for AI R&D\nevaluation: they submit AI-generated papers to blind peer-review processes,\nsuch as to AI conferences and workshops. But peer review is a weak measure of\nresearch ability: conference reviewing is overstretched and highly\nstochastic  ( Cortes and Lawrence, 2021 ; Beygelzimer et al., 2023 ; Shah, 2022 ) , and it does not reveal how many AI-generated submissions were\nrejected before the eventual acceptance.\nIn this paper, we evaluate progress towards automating AI research with a new method, which we call\nshadow evaluation. This involves taking the central research question from a\nhigh-quality research paper that is not yet public, tasking a well-resourced\nfrontier agent with answering it, and asking the paper’s original authors to\ngrade the agent’s output as they would a conference submission. The agent\n“shadows” the original study: it works on the same research question as the\noriginal authors, without access to their paper or findings. This design gives\nus open-ended tasks, uncontaminated questions, and reviewers with deep\nexpertise in the exact question being evaluated. We consider shadow\nevaluations complementary to both narrow, verifiable evaluations and blind\nreview evaluations of automated AI research. We hope future research uses all\nthree to explore different facets of measuring progress towards automating AI research.\nTo carry out shadow evaluations of open-ended AI research, we partnered with the\nauthors of two papers submitted to NeurIPS 2026 that were not yet public.\nUnpublished papers give us a way to test if agents can solve open-ended\nresearch problems. Through their own months-long effort in thinking about the\nquestion, the original authors are uniquely positioned to grade the quality of\nthe agent’s work, and the agent cannot look up the researchers’ findings,\nbecause they are not yet on the web. We gave an agent the paper’s research\nquestion, six days of wall-clock time, $3,000 in Anthropic API credits to\nallow agents to conduct open-ended exploration and run experiments over the\ncourse of a week, GPU credits for experiments, and full access to a VM and the\nopen web. The goal was to produce a paper worthy of publication at a top-tier\nAI conference. The original authors of both papers then graded the results as\nconference reviewers.\nTable 1 : Summary of the original paper authors’ reviews of the agents’\nsubmitted work. Section  3 contains additional details on our setup and Appendix  9 outlines the research questions and relevant context for both papers. \\headingfont\nCriterion\nPaper 1 (Personas)\nPaper 2 (TabPFN)\nSummary of expert comments\nQuality\n2/4\n1/4\nUnprincipled data and experiment choices; conclusions did not follow from\nthe evidence.\nClarity\n1/4\n2/4\nDense, unclear writing; hard to tell what matters.\nSignificance\n2/4\n2/4\nOf limited interest; not well justified over prior and comparable work.\nOriginality\n3/4\n2/4\nNew datasets and some new methods, but built primarily on prior work.\nOverall\n2/6\n1/6\nBoth unambiguous rejections.\nConfidence\n4/5\n5/5\nBoth reviewers were confident or certain in their assessments.\nOur key finding was that while agents could solve the engineering problems\nnecessary to do the research, they failed to produce original research at the\ncaliber of a top ML conference (see Table 1 for details). Our main takeaways:\n1.\nThe agents lacked the judgment to identify when a problem was\nadequately solved. They understood the research questions and proposed\ndirections that closely mirrored those of the authors of the original papers. But they\nthen falsified those hypotheses using small, hand-curated or synthetic\ndatasets. In their final papers, they engaged only shallowly with the\nliterature, and they presented underpowered negative results as substantive findings.\n2.\nThe agents lacked awareness about the resources available to\nthem and the timeline for the project. Both runs ended with less than 50%\nof the API budget spent, even though the agents could monitor their own usage\nin real time, and were encouraged to use their remaining budgets. The agents\ndid not appear to intuitively grasp the meaning of these resource limits,\nparticularly the time limit. They rushed through their initial exploration\nin a number of hours, and finished with hours of clock time remaining despite\npapers that did not meet their own bar for success.\n3.\nThe agents did not creatively respond to feedback about poor\nresearch design. We instructed the agents to send their papers for AI\nreview—both to a subagent and to external tools such as refine.ink—\nto assess the quality and progress of their research. Across dozens of rounds of\nrevision, the agent’s self-review never once returned an acceptance (see Figure   3 ). These\nreviews surfaced many of the issues that the human reviewers later raised\n(see Table   4 ). But the agents did not creatively address\nthe feedback from the AI reviews; when faced with negative feedback they\nresponded by adding caveats to existing findings, and continued to pursue\nunpromising research directions.\n4.\nThe agents did not effectively backtrack from unpromising\napproaches. While the agents initially experimented with multiple distinct\nresearch directions and backtracked locally throughout the process, they\nboth retired their most ambitious research targets within the first ten\nhours, and neither agent fundament","chars":12000,"total_chars":124208,"offset":0,"truncated":true,"next_offset":12000}