Step 6: tool results

read_page

{
  "url": "https://arxiv.org/abs/2608.13417",
  "max_chars": 4000
}

{"ok":true,"url":"https://arxiv.org/abs/2608.13417","text":"[2608.13417] Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development\nSkip to main content\nSearch\nSubmit\nDonate\nLog in\nSearch arXiv\nPress Enter to search &middot; Advanced search\n-->\nComputer Science > Artificial Intelligence\narXiv:2608.13417 (cs)\n[Submitted on 13 Aug 2026]\nTitle: Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development\nAuthors: Yiwei Li , Wanli Yang , Hexiang Tan , Xiangzhou Huang , Zhengyu Chen , Ziran Li , Borun Chen , Shanglin Lei , Huaisheng Zhu , Hao Tian , Fei Sun , Xunliang Cai , Jingang Wang\nView a PDF of the paper titled Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development, by Yiwei Li and 12 other authors\nView PDF\nHTML (experimental)\nAbstract: Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.\nSubjects:\nArtificial Intelligence (cs.AI)\nCite as:\narXiv:2608.13417 [cs.AI]\n(or\narXiv:2608.13417v1 [cs.AI] for this version)\nhttps://doi.org/10.48550/arXiv.2608.13417\nFocus to learn more\narXiv-issued DOI via DataCite\nSubmission history\nFrom: Yiwei Li [ view email ] [v1]\nThu, 13 Aug 2026 16:11:22 UTC (1,542 KB)\nFull-text links:\nAccess Paper:\nView a PDF of the paper titled Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development, by Yiwei Li and 12 other authors\nView PDF\nHTML (experimental)\nTeX Source\nview license\nCurrent browse context:\ncs.AI\n< prev\n|\nnext >\nnew\n|\nrecent\n| 2026-08\nChange to browse by:\ncs\nReferences & Citations\nNASA ADS\nGoogle Scholar\nSemantic Scholar\nexport BibTeX citation\nLoading...\nBibTeX formatted citation\n&times;\nloading...\nData provided by:\nBookmark\nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer ( What is the Explorer? )\nConnected Papers Toggle\nConnected Papers ( What is Connected Papers? )\nLitmaps Toggle\nLitmaps ( What is Litmaps? )\nscite.ai Toggle\nscite Smart Citations ( What are Smart Citations? )\nCode, Data, Media\nCode, Data and Media Associated with this Article\nalphaXiv Toggle\nalphaXiv ( What is alphaXiv? )\nLinks to Code Toggle\nCatalyzeX Code Finder for Papers ( What is CatalyzeX? )\nDagsHub Toggle\nDagsHub ( What is DagsHub? )\nGotitPub Toggle\nGotit.pub ( What is GotitPub? )\nHuggingface Toggle\nHugging Face ( What is Huggingface? )\nScienceCast Toggle\nScienceCast ( What is ScienceCast? )\nDemos\nDemos\nReplicate Toggle\nReplicate ( What is Replicate? )\nSpaces Toggle\nHugging Face Spaces ( What is Spaces? )\nSpaces Toggle\nTXYZ.AI ( What is TXYZ.AI? )\nRelated Papers\nRecommenders and Search Tool","chars":4000,"total_chars":5076,"offset":0,"truncated":true,"next_offset":4000}

read_page

{
  "url": "https://epoch.ai/publications/innovationeval",
  "max_chars": 9000
}

{"ok":true,"url":"https://epoch.ai/publications/innovationeval","text":"Can AI automate AI R&D yet? | Epoch AI\nLatest Our work Featured\nTrends in AI\nData on AI\nCapabilities & benchmarking\nPublications\nPapers & reports\nData Insights\nNewsletter\nPodcast\nAll publications\nData explorers\nAI capabilities\nAI models\nAI data centers\nAI chip sales\nAI companies\nPolling on AI usage\nAll data explorers\nOur benchmarks\nEpoch Capabilities Index\nMirrorCode\nFrontierMath: Open Problems\nEpoch Automation Reports\nEarthborne Rangers\nTopics AI progress\nScaling\nSoftware progress\nOpen models\nCapabilities & benchmarks\nMath\nIndustry\nLeading companies\nFinances\nGeopolitics\nInfrastructure\nChips\nData centers\nEnergy\nImpacts\nAdoption & use\nEconomic impact\nFuture of AI\nAI impacts observatory\nAbout Who we are\nAbout us\nOur team\nTransparency\nEngage with us\nCareers\nPress\nWork with our experts\nDonate\nContact\nLatest\nOur work Featured\nTrends in AI Data on AI Capabilities & benchmarking\nPublications\nPapers & reports\nData Insights\nNewsletter\nPodcast\nAll publications\nData explorers\nAI capabilities\nAI models\nAI data centers\nAI chip sales\nAI companies\nPolling on AI usage\nAll data explorers\nOur benchmarks\nEpoch Capabilities Index\nMirrorCode\nFrontierMath: Open Problems\nEpoch Automation Reports\nEarthborne Rangers\nNavigate by topic AI progress\nScaling\nSoftware progress\nOpen models\nCapabilities & benchmarks\nMath\nIndustry\nLeading companies\nFinances\nGeopolitics\nInfrastructure\nChips\nData centers\nEnergy\nImpacts\nAdoption & use\nEconomic impact\nFuture of AI\nAI impacts observatory\nAbout Who we are\nAbout us\nOur team\nTransparency\nEngage with us\nCareers\nPress\nWork with our experts\nDonate\nContact\nPublications\nCan AI automate AI R&D yet?\nReport\nOct. 7, 2026\nCan AI automate AI R&D yet?\nEarly evidence from InnovationEval: No\nCite\nBy David Owen\nContents\nIntroduction\nMethodology\nResults\nAI struggles at end-to-end AI algorithms R&D… for now\nAppendix: evaluation details\nIntroduction Introduction\nMethodology\nResults\nAI struggles at end-to-end AI algorithms R&D… for now\nAppendix: evaluation details\nIntroduction\nAI developers aim to create an automated AI researcher . How close are they? Existing evidence shows that AI can perform software engineering tasks relevant to AI research , dataset creation, 1 and other tasks , including open-ended optimization of defined metrics ( autoresearch ). But it remains unclear what full automation of AI R&D entails, given the wide range of activities involved. One obvious gap is the ability to conduct end-to-end research projects. 2 Similar to recent work such as Crux Evals , ResearchGym , and RSI-Bench , we test AI on end-to-end research tasks.\nWe present early results from InnovationEval, an evaluation where we measure AI’s ability to independently discover novel machine learning techniques comparable to those developed by human researchers. Recent frontier models make little progress on this task, despite running experiments using thousands of dollars’ worth of GPU time. We plan to expand and repeat this methodology, tracking AI’s progress towards automating AI research itself.\nMethodology\nInnovationEval tests whether AI can independently devise an ML innovation that matches the performance of a recent human-developed innovation the AI has not seen. This is similar to recently-proposed tests for scientific ideation: if AI were presented with humanity’s knowledge up to 1905, could it rediscover special relativity? 3 We ask a more modest question: if AI were presented with AI researchers’ knowledge up to early 2026, could it discover its own ML algorithmic innovation, matching the improvements achieved by human researchers since then?\nWe hope to achieve several advantages through this approach: end-to-end validation of AI’s R&D abilities, a requirement for genuine innovation rather than assembly of existing techniques, realistic representation of research areas, and guaranteed feasibility.\nEnd-to-end validation: AI systems have to perform the entire process of discovering an ML innovation, from coming up with ideas through to implementing them. Similar to existing work such as NanoGPT speed-runs and ResearchGym, we define metrics that should be improved and constraints that should be satisfied. 4 Using end-to-end metrics provides a legible way to assess AI performance, as long as improving the metrics genuinely requires the AI to make research progress. In our case, these metrics are set to match an existing human-authored paper. We set up an AI agent to develop a better post-training method, which requires end-to-end generation of ideas, figuring out details of their implementation, experimenting with them, analyzing the results, and iterating until reaching either success or exhaustion.\nInnovation is required: Many AI R&D evaluations examine well-specified tasks that don’t require innovation, 5 or can be solved by applying combinations of non-novel techniques. 6 Some existing benchmarks try to isolate the task of R&D ideation, 7 but it is unclear whether this task can be done in isolation from the full loop, including implementation and analysis. Our evaluation sets up a task where substantially improving the end-to-end metrics without violating scope requires development of a method the AI has not seen in training. 8 We elaborate on this in Task setup .\nRealism of research area: We want to test AI’s ability to discover ML techniques similar to those valued by (and used in) frontier AI labs. 9 This is difficult because frontier AI developers are secretive about many of their methods. We cannot directly test AI on rediscovering their ML techniques, so we instead rely on open publications and other evidence that a technique is useful, such as adoption in prominent near-frontier models or discussion by post-training researchers. 10 Here, we selected a paper about on-policy self-distillation. We discuss this in more detail below.\nFeasible: Using a real, replicable AI paper guarantees that our task is feasible, and provides us information about the required GPU resources for human researchers. In some AI R&D evaluations, the objective is to improve on an existing method, but without a human baseline, and thus with less clarity on the required budget, and whether human researchers would have tried a different approach.\nThere are also disadvantages that come with anchoring on existing papers. One is that, at least in this iteration, we have struggled to create a task that is amenable to fully automated grading. We describe this in more detail in Task setup . Another disadvantage is that we have ended up relying on a small number of runs, since each individual attempt at this task requires substantial compute budgets.\nAnother disadvantage of using an existing innovation is that newer models will memorize our task. This happened over the course of this project; our main results are on Claude Fable 5 and GPT-5.6 Sol, which showed no sign of memorization when prompted to recall or guess details about the paper without using search. But their successors, Claude Fable 5.1 and GPT-6 Astra, were aware of the task. Our plan for future evaluations is to perform ongoing tests for memorization in newer models, flag their results accordingly, and devise new tasks as necessary to refresh the evaluation.\nTask setup\nAs our testbed task, we used a recent AI innovation that has been adopted and cited by recent models: on-policy self-distillation (SDPO). The AI agent was prompted to develop a novel post-training technique that beats a strong GRPO baseline. We emphasized that the agent’s goal was “to produce a compelling research result, of the kind that would genuinely advance the field.” We then provided metrics and datasets used for the results in the original paper: short-answer questions 11 and coding. 12 The agent was told to produce evidence of its method’s success by post-training a Qwen3-8B model to perform better on these tasks, ideally matching or surpassing reference values set by a recent unnamed method (SDPO).\nThe overall eval grade is the averaged performance across the two result areas, each of which has several sub-metrics based on the original paper’s experiments. Matching or surpassing the original paper’s performance in an area yields a score of 100%, whereas scores at the GRPO baseline are scored at 0%. We provide more detail on prompting and scoring in Scoring .\nIt is important to set the task’s scope correctly. If the goal were purely to improve performance on these datasets, there are many ways this might be achieved, such as by generating synthetic datasets for fine-tuning. This wouldn’t count as developing a novel post-training technique and wouldn’t advance the field, so arguably the agent should know not to use this approach. Rather than trying to grade novelty, we attempted to limit the scope such that the agent can only match the original innovation’s performance through novelty in its own approach — even if it lands on a novel approach distinct from SDPO. We constrain the scope to algorithmic changes that affect the loss and its up","chars":9000,"total_chars":92003,"offset":0,"truncated":true,"next_offset":9000}