Step 10: tool results

read_page

{
  "url": "https://arxiv.org/html/2608.13417",
  "offset": 3000,
  "max_chars": 9000
}

{"ok":true,"url":"https://arxiv.org/html/2608.13417","text":" indicate whether accumulated experience improves later decisions.\nWe therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks.\nThe results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare.\nDetailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability.\nThese findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.\n \n* * footnotetext: Equal contribution.\n1 Introduction\nFrontier language models are increasingly capable of conducting long-horizon automated research, repeatedly proposing changes, running experiments, interpreting feedback, and refining executable artifacts  ( Huang et al., 2024 ; Wijk et al., 2025 ; Xu et al., 2026 ) .\nBy requiring agents to optimize models, algorithms, or computing systems, these tasks can provide a measurable form of AI-for-AI and an early window into how close frontier language models are to enabling recursive self-improvement  ( Chan et al., 2025 ; Rank et al., 2026 ; Lyu et al., 2026 ) .\nSystematically evaluating current agents is therefore essential for understanding their research capabilities and guiding targeted improvements to both models and agent systems  ( Wijk et al., 2025 ; Meng et al., 2026 ) .\nHowever, there remains a fundamental mismatch between the process of automated research and its evaluation: agents engage in long-horizon, closed-loop cycles of experimentation and refinement, yet existing benchmarks primarily evaluate them using a single final score, which fails to capture the underlying reasons for model behavior or provide fine-grained diagnostic information  ( Huang et al., 2024 ; Chan et al., 2025 ; Wijk et al., 2025 ) .\nWithin a run, for example, the same final score may come from an effective direction identified early or from one found after extensive trial and error.\nIt also does not show whether proposed ideas are translated into reliable implementations or whether feedback is used effectively to retain progress and recover from failures.\nBeyond the limitations of final-score-based evaluation, conventional evaluation treats each run independently, making capability appear static and obscuring whether accumulated experience improves or misleads subsequent decisions.\nIt also leaves unclear whether the surrounding harness helps the agent sustain effective behavior over a long research run.\nThese gaps make it difficult to determine how far current agents have progressed toward autonomous research and whether further improvements should target model capabilities, experience reuse, or harness design.\nTo address these gaps and provide a systematic evaluation framework for agents engaged in long-horizon AI R&D, we organize the evaluation around four questions:\n❶ How strong are the final results produced by current agents?\n❷ Where is progress gained or lost within the research loop?\n❸ Can accumulated experience improve subsequent decisions?\n❹ How does harness choice affect agent performance?\nSpecifically, final performance is measured directly from task scores.\nTo diagnose behavior within a run, we decompose the research process into three complementary capabilities: Solution Framing (C1), Execution (C2), and Feedback Control (C3).\nFor each capability, we design a rule-based metric computed deterministically from verifier outcomes and recorded trajectory signals rather than LLM judgments.\nBeyond these within-run capabilities, we treat the ability to use accumulated experience as a meta-capability (M) and conduct controlled comparisons to measure its effect on subsequent decisions in intra- and inter-task settings.\nFigure  1 summarizes the proposed process and experience views.\nIn addition, harness effects are examined by comparing alternative harness designs.\nAs a complementary analysis, we use LLM judges to examine whether the solutions produced by agents exhibit genuine methodological novelty. To provide a comprehensive evaluation, we conduct these analyses across seven frontier models and a suite of 36 long-horizon tasks.\nThe full evaluation required approximately one hundred thousand U.S. dollars in model inference.\nFigure 1: Analytical views used to interpret behavior in automated research. The process view covers Solution Framing (C1), Execution (C2), and Feedback Control (C3). The experience view uses controlled comparisons to measure how accumulated experience affects subsequent decisions in intra- and inter-task settings.\nBenefiting from our evaluation design, which enables fine-grained diagnosis of model capabilities, we reach an overall assessment of the current capability stage:\nCurrent automated research agents operate more like engineering optimizers than fully autonomous researchers.\nWithin bounded research loops, they can formulate practical directions, implement working solutions, and improve technical artifacts.\nYet their success varies across runs, genuine algorithmic innovation remains rare, and realized performance is shaped by process bottlenecks, accumulated experience, and harness design.\nThe evidence for this conclusion is threefold:\n•\nReliability separates current models more than peak performance.\nThe gap between the strongest and weakest models is 0.237 0.237 on avg@3 but only 0.122 0.122 on best@3.\nSeveral models can therefore reach competitive solutions, but show substantially different levels of consistency across repeated runs.\nThese results point to headroom in inference-time selection and rollout-relative training to close the gap between observed peak and average performance.\n•\nOutcome scores conceal where research actually fails.\nFor example, GPT-5.5 and Gemini-3.1-Pro achieve similar final scores and identical Solution Framing scores, yet GPT-5.5 is substantially stronger in Execution while Gemini-3.1-Pro is stronger in Feedback Control.\nThe dominant bottleneck also varies by task category: CUDA tasks show the weakest Solution Framing and Execution, whereas Model Development tasks show the strongest Execution but the weakest Feedback Control.\nMore broadly, although Execution scores are high across all seven models, only three of 252 best-seed solutions qualify as novel approaches under our review protocol, revealing a clear gap between optimization performance and methodological novelty.\nOverall, a leaderboard can rank systems, but it cannot diagnose where improvement is needed.\n•\nResearch performance is not fixed by the backbone model alone: experience can improve or degrade performance, while harnesses mainly affect reliability.\nWithin tasks, accumulated experience usually improves the next solution by preserving useful discoveries, but it can also carry forward misleading conclusions or anchor agents to local optima.\nAcross tasks, this dual effect is strong enough to change model ordering: transferred experience raises DeepSeek-V4-Pro’s avg@3 by 0.093 0.093 but lowers Gemini-3.1-Pro’s by 0.017 0.017 .\nBy contrast, using their native harnesses gives GPT-5.5 and Kimi-K2.7-Code greater run-to-run stability than the shared harness without materially changing best@3 or model ordering.\nAutomated harness optimization offers further headroom.\nEvaluating models as static, isolated components therefore misses both their learning dynamics and the system design required to realize their capabilities.\nTaken together, these findings show that autonomous research capability is neither one-dimensional nor static, and that observed performance reflects an interaction among the model, its accumulated experience, and the system around it.\nThe report therefore begins with final performance and cost, then examines within-run process behavior, experience-driven improvement, and harness effects before discussing their implications for model training and agent-system design.\n2 Evaluation Setting and Outcome-Level Landscape\nWe first evaluate seven frontier models on the same tasks with a shared harness and protocol to establish a controlled comparison of final performance and resource use.\nThis outcome-level landscape anchors our subsequent analyses of research behavior, experience reuse, harness effects, and solution novelty.\n2.1 Evaluation Setting\nEvaluation tasks.\nOur evaluation focuses on four workload families that capture distinct demand","chars":9000,"total_chars":131201,"offset":3000,"truncated":true,"next_offset":12000}