Step 6: tool results

read_page

{
  "url": "https://arxiv.org/abs/2607.27191",
  "max_chars": 4000
}

{"ok":true,"url":"https://arxiv.org/abs/2607.27191","text":"[2607.27191] Can AI agents conduct open-ended AI research? Early evidence from two case studies\nSkip to main content\nSearch\nSubmit\nDonate\nLog in\nSearch arXiv\nPress Enter to search &middot; Advanced search\n-->\nComputer Science > Artificial Intelligence\narXiv:2607.27191 (cs)\n[Submitted on 29 Jul 2026 ( v1 ), last revised 7 Aug 2026 (this version, v2)]\nTitle: Can AI agents conduct open-ended AI research? Early evidence from two case studies\nAuthors: Peter Kirgis , Sayash Kapoor , Andrew Schwartz , Stephan Rabanser , David Africa , Konstantinos Voudouris , Viet Nguyen , Toby Pilditch , Magda Dubois , Harry Coppock , Cozmin Ududec , Nitya Nadgir , Matilda Orona , Tilman Bayer , Derrick Chan-Sew , Yue Ling , Abhishek Shetty , Helen Toner , Gillian Hadfield , Seth Lazar , Steve Newman , Shoshannah Tekofsky , Rishi Bommasani , Arvind Narayanan\nView a PDF of the paper titled Can AI agents conduct open-ended AI research? Early evidence from two case studies, by Peter Kirgis and 23 other authors\nView PDF\nHTML (experimental)\nAbstract: Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.\nSubjects:\nArtificial Intelligence (cs.AI) ; Computers and Society (cs.CY); Machine Learning (cs.LG)\nCite as:\narXiv:2607.27191 [cs.AI]\n(or\narXiv:2607.27191v2 [cs.AI] for this version)\nhttps://doi.org/10.48550/arXiv.2607.27191\nFocus to learn more\narXiv-issued DOI via DataCite\nSubmission history\nFrom: Peter Kirgis [ view email ] [v1]\nWed, 29 Jul 2026 17:57:19 UTC (1,453 KB)\n[v2]\nFri, 7 Aug 2026 04:24:54 UTC (1,454 KB)\nFull-text links:\nAccess Paper:\nView a PDF of the paper titled Can AI agents conduct open-ended AI research? Early evidence from two case studies, by Peter Kirgis and 23 other authors\nView PDF\nHTML (experimental)\nTeX Source\nview license\nCurrent browse context:\ncs.AI\n< prev\n|\nnext >\nnew\n|\nrecent\n| 2026-07\nChange to browse by:\ncs\ncs.CY\ncs.LG\nReferences & Citations\nNASA ADS\nGoogle Scholar\nSemantic Scholar\nexport BibTeX citation\nLoading...\nBibTeX formatted citation\n&times;\nloading...\nData provided by:\nBookmark\nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer ( What is the Explorer? )\nConnected Papers Toggle\nConnected Papers ( What is Connected Papers? )\nLitmaps Toggle\nLitmaps ( What is Litmaps? )\nscite.ai Toggle\nscite Smart Citations ( What are Smart Citations? )\nCode, Data, Media\nCode, Data and Media Associated with this Article\nalphaXiv Toggle\nalphaXiv ( What is alphaXiv? )\nLinks to Code Toggle\nCatalyzeX Code Finder for P","chars":4000,"total_chars":5512,"offset":0,"truncated":true,"next_offset":4000}