Step 10: tool results

read_page

{
  "url": "https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/",
  "max_chars": 14000
}

{"ok":true,"url":"https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/","text":"Independent Assessment of AI R&D Acceleration at Anthropic in October 2026 - METR\nOur Work\nResearch\nNotes\nUpdates\nRisk Assessment\nAbout\nDonate\nCareers\nSearch\n-->\nOur Work\nResearch\nNotes\nUpdates\nRisk Assessment\nAbout\nDonate\nCareers\nMenu\n×\nIndependent Assessment of AI R&D Acceleration at Anthropic in October 2026\nCONTRIBUTORS\nTom Cunningham ,\nNate Rush ,\nParker Whitfill ,\nand\nHjalmar Wijk\nDATE\nNovember 5, 2026\nSHARE\nCopy Link\nCitation\nBibTeX Citation\n×\n@misc { metr-2026-anthropic-ai-rd-acceleration-assessment ,\ntitle = {Independent Assessment of AI R&D Acceleration at Anthropic in October 2026} ,\nauthor = {METR} ,\nhowpublished = {\\url{https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/}} ,\nyear = {2026} ,\nmonth = {11} ,\n}\nCopy\nSubscribe\nRedaction summary: As part of this assessment, Anthropic had rights to redact any non-public information in this post. Per our approach to transparency , we had the right to disclose if Anthropic exercised redaction rights, and whether or not any of our findings depended significantly on redacted information. While we were able to freely share our highest-level conclusions about risk, Anthropic redacted a majority of the evidence we found most informative for these conclusions. Anthropic described the redactions as “aimed at removing confidential information that could accelerate other AI developers.” 1 Some of these redactions are themselves challenging to describe clearly without revealing sensitive information we have been asked to redact. We share a table of important evidence we were and were not able to discuss given the redactions in our process section below. All redactions that were important for our conclusions are covered by our descriptions in that table or other in-line descriptions in the report below.\nMuch of the redacted evidence we viewed had multiple plausible interpretations, and both METR and Anthropic researchers have significant uncertainty about how to synthesize the evidence into a holistic picture. This makes it hard for the public to judge this evidence themselves, reconcile it with evidence from other sources, and come to their own conclusions (i.e. this report does not achieve ‘transparency about evidence’ ). As a consequence, the public has to rely on METR’s judgment much more than would be ideal, given the weight of the questions answered here.\nExecutive summary\nFrom August 12 to October 29, 2026, METR staff performed a high-access evaluation at Anthropic to assess the degree to which AI is accelerating AI R&D. 2 If AI systems started to significantly accelerate further AI development, this could lead to a sudden emergence of extreme and risky capabilities .\nAnthropic offered unprecedented access to internal data on their AI R&D process, including a variety of metrics on pretraining, reinforcement learning (RL), training compute in each domain, benchmark scores for internal models, AI speedup analyses, forecasts of a future model’s metrics, and interviews with relevant staff.\nIn our first section, we share our subjective conclusions about risk . We now believe that:\nAs of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025. This estimate is given with high uncertainty; we would not be surprised if the true acceleration was closer to 1.3x or 3x the historical rate.\nA majority of this acceleration today is likely attributable to Anthropic’s extensive use of AI to do R&D, though there is significant uncertainty and other causes likely played a large role in acceleration in late 2025.\nThe lack of observed bottlenecks or strong diminishing returns means there is a significant risk that further capabilities progress could lead to even faster acceleration, though, as we say below, we may be missing important bottlenecks.\nIn our second section we share some of the key empirical findings from our Full Report :\nSince October 2025, benchmark scores (measured via AECI, Anthropic’s version of the Epoch Capabilities Index) for Anthropic’s models have been growing at an estimated 1.9x the pre-October 2025 rate [95% confidence interval: 1.5–2.8x], implying that Anthropic has made roughly two years of historical progress over the last year.\nWe are confident that the acceleration was not due to an acceleration in the scaling of training compute.\nInterviews and surveys are consistent with AI usage causing a doubling in the pace of capabilities progress. Anthropic staff self-report dramatic uplift from use of AI systems in interviews and surveys, although these are challenging to interpret. We were told that R&D is accelerated primarily via AI assistance rather than fully autonomous contributions, but there are some signs that fully autonomous contributions will play a larger role soon.\nWe did not see evidence for significant bottlenecks which would prevent further capabilities progress due to limits on algorithmic progress, training compute, experiment compute, or data in the next 12 months. However, our understanding of the status of different bottlenecks changed several times as we received more detailed information, so we are not confident in our current assessment.\nSome of the key findings that most informed our conclusions about risk were redacted by Anthropic.\nIn our third section, we describe the assessment process :\nThe scope, setup, and timeline of this assessment.\nDetails of the information shared with METR.\nDetails of what information and evidence we were not able to share given Anthropic’s redactions.\nFinally, we describe the biggest limitations of our analysis:\nWe could be underestimating capability acceleration if we did not assign enough importance to (1) evidence that some growth rates have further increased recently; (2) further automation of AI R&D tasks leading to a further acceleration in progress; (3) evidence that AECI growth has sometimes understated growth in real-world impact.\nWe could be overestimating capability acceleration if we did not assign enough importance to (1) non-AI factors in Anthropic’s research progress; (2) AECI’s focus on verifiable tasks; (3) reasons why Anthropic’s early growth rate (2023–2025) was unusually low.\nThe remainder of the report goes into detailed discussion of each of the key components. We include appendices with supplemental material.\nOur subjective conclusions on risk\nOur conclusions on risk.\nIndependently of this assessment, METR published an October 2026 article proposing a set of high-level questions about AI R&D acceleration we believe the public needs to understand to be able to judge loss-of-control risks from frontier AI development. In this section, we describe how the evidence we reviewed in this assessment changed our subjective conclusions on these high-level questions.\nPace of capabilities progress.\nA significant acceleration of Anthropic’s overall rate of progress would be surprising, since diminishing returns and dependence on compute scaling mean this would likely require remarkable scaling in other inputs. 3 After reviewing the internal evidence, we still have significant uncertainty and internal disagreements on the team, but most METR staff who worked on this project now believe that, as of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025. We would not be surprised if the current rate of growth was closer to 1.3x or 3x the historical rate. We have additional uncertainty in how this acceleration translates to broader and less well-specified tasks.\nThe 1.9x acceleration [95% confidence interval: 1.5–2.8x] in the AECI trend since October 2025 described below significantly informed this view, but could be explained by overfitting to the benchmark distribution or otherwise be an artifact of the AECI methodology. On the other hand, we are not confident that AECI would reliably let us estimate recent R&D acceleration, since the data points are lagging acceleration in research and are somewhat sparse.\nBeyond the AECI evidence, we also saw redacted evidence that significantly informed our view on this question, and made us take this apparent ≈2x acceleration more seriously. It also gave us some indication that acceleration could be above 2x and increasing over time, though we would be surprised if it was significantly higher.\nAttribution to AI.\nWe looked at a variety of factors that could explain the acceleration we observed, including compute scaling, hiring, data improvements, algorithmic discoveries, and resolving other bottlenecks.\nAs we describe below, we were able to confidently determine that the acceleration was not due to an acceleration in the rate of growth of training compute. Beyond that, there was not a consensus among Anthropic’s own staff and we did not find decisive evidence. Our overall subjective estimate is that:\nThe acceleration in late 2025 was likely influenced by non-AI factors. We investigated a variety of factors which could have contributed to the observed acceleration in late 2025, but details of many of these factors have been redacted.\nWe guess that a majority (perhaps ≈80%) of the current (October 2026) acceleration is due to AI. This judgment is given with substantial uncertainty, but is based on a variety of lines of evidence on contributions from AI and other causes. We believe AI’s contribution has grown relatively more important over time, and do not think the other factors we investigated would explain the sustained acceleration. The evidence of speedup from the usage of AI we describe below also seems compatible with a large contribution.\nWe illustrate these points below:\nOur more detailed estimates for the cumulative and instantaneous effects of AI on Anthropic’s rate of progress in AECI, and at various stages of the model training pipeline, were redacted.\nBottlenecks to further acceleration.\nA key concern is that acceleration from the use of AI could start to compound, as the development of more capable models leads to further acceleration. As we describe below, we did not identify any clear bottlenecks (limited inputs that would prevent further growth) or steep diminishing returns to further scaling that would get in the way of such compounding acceleration, though our understanding evolved several times over the course of this assessment, and we have significant uncertainty. Because of this, we believe there is a significant risk that further capabilities progress could lead to even faster acceleration over the coming year.\nKey findings\nIn this section, we summarize the key empirical findings from our Full Report that we can describe without revealing redacted information (see the process section for more details on what was redacted).\nTerminology: We describe growth as annual multiples and acceleration as a ratio of growth rates.\nWe use “acceleration” to refer to an increase in the rate of growth, such that a 2x acceleration means that 2 years of historical progress is made in 1 year.\nIf AECI’s historical trend was 15 points/year, then increasing to 30 points/year implies an acceleration rate of 2 (it takes 1 year to achieve 2 years’ progress at the old rate). For an exponentially growing variable, the acceleration rate is the ratio of growth rates. If a variable is growing 3x/year, then to make 2 years of progress in 1 year the growth rate must be (3 2 )x/yr = 9x/yr (and acceleration can be written as ln(9)/ln(3) = 2).\nSince October 2025, benchmark scores for internal models have been growing at around twice their historical rate.\nBenchmark scores of Anthropic’s models, as measured by AECI (Anthropic’s version of the Epoch Capabilities Index ), have accelerated 1.9x [95% confidence interval: 1.5–2.8x] since October 2025. Put another way, Anthropic appears to have made as much progress on benchmarks in the past year as they did between October 2023 and October 2025 (roughly, Claude 2.1 to Sonnet 4.5):\nOur estimated acceleration is greater than that reported in the Haiku 5.5 system card (1.7x) primarily because of the inclusion of data on newer unreleased models, explained further below.\nWe investigated the robustness of these results to different choices of benchmark inclusion, model dating, model inclusion, and benchmark saturation. Robustness analyses find acceleration rates between 1.1x and 2.9x. We would be surprised if the true evolution of benchmark scores (e.g. as measured by an improved version of AECI) showed no acceleration or acceleration above 3x. See our appendix section for details.\nThe acceleration is not driven by an acceleration in the scale of training compute.\nWe received convincing evidence that the acceleration was not due to an acceleration in the scale of training compute. Further details on this evidence were redacted.\nInterviews and surveys are consistent with AI usage driving this doubling in the pace of capabilities progress.\nWe have many lines of evidence relevant to AI’s contribution to progress in different parts of the model training pipeline, including interviews, self-reported uplift, headcount, lines of code, experiment usage, and qualitative reflections from leadership. We highlight three important unredacted sources.\nFirst, both METR and Anthropic interviewed many researchers in detail about the degree of overall acceleration in their area of work attributable to AI use (see details in the appendix below). Our overall impression was a quite wide range of estimates, but centered around a 2x acceleration. Some quotes, with sensitive details redacted, are in the appendix.\nSecond, Anthropic also shared data on self-reported uplift due to AI from a variety of sources. The most recent data is from a series of Slack polls, where staff use reactions to vote on their estimated speedup from the most recent model at the time. We consider this to be only very loose evidence, as the res","chars":14000,"total_chars":78371,"offset":0,"truncated":true,"next_offset":14000}

read_page

{
  "url": "https://web.archive.org/web/2026/https://openai.com/index/research-acceleration-view-inside-openai/",
  "offset": 14000,
  "max_chars": 9000
}

{"ok":true,"url":"https://web.archive.org/web/2026/https://openai.com/index/research-acceleration-view-inside-openai/","text":"tally, colleagues report that coding agents excel at troubleshooting internal research infrastructure, which addresses one meaningful bottleneck to research progress. Multiple teams which previously held office hours to help researchers troubleshoot their experiments have noted declining attendance in 2026, and one has stopped holding sessions entirely, to focus on making other system improvements instead.\nHere, we plot the number of top-level posts per day to one of the main internal channels where researchers seek technical support from other teams. To our knowledge, the channel’s decrease in activity has not been offset by queries shifting to another technical support channel run by humans. The decline in traffic aligns with this broader shift.\n:nth-child(2),>[data-dotcom-chart-group]>:nth-child(2)]:[--dotcom-chart-edge-gutter:0px] [&_[data-dotcom-chart-logical-container][style*='width:_min-content']]:[--dotcom-chart-edge-gutter:0px] [--dotcom-chart-edge-gutter:var(--container-inline-gutter,0px)] md:[--dotcom-chart-edge-gutter:0px] @md:[--dotcom-chart-edge-gutter:0px] multi-columns:[--dotcom-chart-edge-gutter:0px] shaded-container:[--dotcom-chart-edge-gutter:0px] items-start\">\nView methods\nWe can also study whether coding agents are succeeding at the tasks researchers request. Using an agentic classifier, we find that from January to July, success rates generally increased across several difficulty buckets (proxied as the estimated time a human would take to complete the task) on tasks we can find a ground truth outcome for. However, agents still require significant human steering to be successful, especially as task complexity rises. In the last 6 months, over half of successful 4-8 hour tasks involved 1 or more interventions.\n:nth-child(2),>[data-dotcom-chart-group]>:nth-child(2)]:[--dotcom-chart-edge-gutter:0px] [&_[data-dotcom-chart-logical-container][style*='width:_min-content']]:[--dotcom-chart-edge-gutter:0px] [--dotcom-chart-edge-gutter:var(--container-inline-gutter,0px)] md:[--dotcom-chart-edge-gutter:0px] @md:[--dotcom-chart-edge-gutter:0px] multi-columns:[--dotcom-chart-edge-gutter:0px] shaded-container:[--dotcom-chart-edge-gutter:0px] items-start\">\nSuccess rates on researcher tasks have increased over time. Graph excludes classifications where the outcome was uncertain and points with <50 sessions or <50 unique users.\nView methods\n:nth-child(2),>[data-dotcom-chart-group]>:nth-child(2)]:[--dotcom-chart-edge-gutter:0px] [&_[data-dotcom-chart-logical-container][style*='width:_min-content']]:[--dotcom-chart-edge-gutter:0px] [--dotcom-chart-edge-gutter:var(--container-inline-gutter,0px)] md:[--dotcom-chart-edge-gutter:0px] @md:[--dotcom-chart-edge-gutter:0px] multi-columns:[--dotcom-chart-edge-gutter:0px] shaded-container:[--dotcom-chart-edge-gutter:0px] items-start\">\nTask success and intervention rate from Jan to July, broken out by time horizon. Excludes classifications where the outcome was uncertain.\nView methods\n4. Pacing model development\nProgress toward more capable systems for safe and beneficial AGI will also depend on the safeguards needed for such work. Our assessment of the needed safeguards may change as we learn more about the risks.\nAs we have described, ⁠ we have recently updated our standards for monitoring, alignment, and security. Here, we show how recent restrictions have affected one aspect of research activity.\n:nth-child(2),>[data-dotcom-chart-group]>:nth-child(2)]:[--dotcom-chart-edge-gutter:0px] [&_[data-dotcom-chart-logical-container][style*='width:_min-content']]:[--dotcom-chart-edge-gutter:0px] [--dotcom-chart-edge-gutter:var(--container-inline-gutter,0px)] md:[--dotcom-chart-edge-gutter:0px] @md:[--dotcom-chart-edge-gutter:0px] multi-columns:[--dotcom-chart-edge-gutter:0px] shaded-container:[--dotcom-chart-edge-gutter:0px] items-start\">\n* The majority of Astra compute shown here between July 20 and August 6 was intended to test the implementation of safety and security improvements.\nView methods\nOn July 20, following the discovery that agents had compromised our research infrastructure, we temporarily shut down the container service used for training, and then restored it with significant additional restrictions.\nThis led to a sharp decline in RL training compute while teams reconfigured their workflows to operate within the hardened research environment. The plot above includes the two week pause in reinforcement learning on our latest models intended for deployment. Astra-class RL experiments between July 20 and August 6 include a majority of runs (by GPU allocation) intended to test the implementation of safety and security improvements.\nOn August 7, preliminary evidence that Astra may have critical cyber capabilities ⁠ under our Preparedness Framework ⁠ (opens in a new window) led to additional model-specific security restrictions which required the Astra model to be run in higher security research environments. In the following week, Astra-class GPU allocation fell a further 59.2 percent, but allocation to other model classes rose 17.2 percent. That increase offset about 85 percent of the Astra-class decline, leaving total allocation in the analyzed RL workloads largely unchanged. This pattern is consistent with substitution of some training and experimentation to non-Astra models while work involving Astra was restricted, and comports with anecdotal reports of researchers finding other uses for compute that could no longer be leveraged for workloads covered by the new constraints.\nThis data provides a useful signal for ongoing conversations about training and safety: When new controls are introduced, compute remains valuable and flexible, and will naturally be channeled into alternative uses within the research enterprise. Longer term, discussions about the pace of AI progress should also extend to the question of how compute that is subject to new or proposed controls can best be used.\n5. The path ahead\nMaking and understanding progress toward aligned RSI is important for our mission. We will continue to refine our methods, report on our evolving understanding, and work toward an informed public debate and meaningful democratic governance of frontier systems.\nAppendix: Our methods for this post\nAgent-powered AI research is still new, and we are still learning how to measure it. Some indicators, such as the amount of code our research teams generate, are relatively easy to gather, but hard to interpret because their relationship to research progress is uncertain. Metrics that focus more directly on research progress—such as how often agents succeed at the tasks researchers give them—could be more useful, but are complex to develop and validate. Furthering the difficulty, the tools and systems researchers rely on are evolving rapidly. Deepening our understanding of research acceleration is a significant focus area across OpenAI.\nAcross these analyses, unless otherwise noted:\np:only-of-type]:mb-0\"> “Researcher” is a broad term for any member of our research organization, including some who build research infrastructure, manage research projects, or otherwise support the enterprise.\np:only-of-type]:mb-0\"> Metrics of coding agent use cover most, but not all, usage given rapid evolution in the tools and systems researchers rely on.\n2026\nEconomic Research\nAuthor\nOpenAI\nKeep reading\nView all\nControl Room: A new way to track and stop AI workloads\nSafety Nov 9, 2026\nMeasuring how AI agents handle everyday wet-lab science\nPublication Nov 6, 2026\nGPT‑6 Cyber: Advancing the frontier of aligned cybersecurity agents\nProduct Nov 5, 2026\nResearch Research Index\nResearch Overview\nEconomic Research\nLatest Advancements GPT-6.1 Sol\nGPT-6 Astra\nGPT-5.6\nGPT-5.5\nSafety Safety Approach\nDeployment Safety (opens in a new window)\nSecurity & Privacy\nTrust & Transparency\nProducts ChatGPT (opens in a new window)\nChatGPT Business (opens in a new window)\nChatGPT Enterprise (opens in a new window)\nChatGPT for Education (opens in a new window)\nCodex\nDots (opens in a new window)\nRelease Notes\nAPI Platform Overview\nAPI Log In (opens in a new window)\nDocs (opens in a new window)\nBusiness Overview\nSolutions\nResources\nPlugins\nCustomer Stories\nPartner Network\nContact Sales\nDevelopers Plugins (opens in a new window)\nModels (opens in a new window)\nDocs (opens in a new window)\nResources (opens in a new window)\nDeveloper Forum (opens in a new window)\nCompany About Us\nOur Charter\nCareers\nNews\nSupport Help Center (opens in a new window)\nOpenAI Status (opens in a new window)\nMore Stories\nAcademy\nSupply Co.\nLivestreams\nPodcast\nRSS\nTerms & Policies Terms of Use\nPrivacy Policy\nOther Policies\n(opens in a new window) (opens in a new window) (opens in a new window) (opens in a new window) (opens in a new window) (opens in a new window) (opens in a new window)\nOpenAI © 2015–2026 Your privacy choices\nEnglish United States","chars":8881,"total_chars":22881,"offset":14000,"truncated":false,"next_offset":null}

read_page

{
  "url": "https://metr.org/notes/2026-02-10-simpler-ai-timelines-model/",
  "max_chars": 5000
}

{"ok":true,"url":"https://metr.org/notes/2026-02-10-simpler-ai-timelines-model/","text":"A simpler AI timelines model predicts 99% AI R&D automation in ~2032 - METR\nOur Work\nResearch\nNotes\nUpdates\nRisk Assessment\nAbout\nDonate\nCareers\nSearch\n-->\nOur Work\nResearch\nNotes\nUpdates\nRisk Assessment\nAbout\nDonate\nCareers\nMenu\n×\nResearch note: A simpler AI timelines model predicts 99% AI R&D automation in ~2032\nCONTRIBUTORS\nThomas Kwa\nDATE\nFebruary 10, 2026\nSHARE\nCopy Link\nCitation\nBibTeX Citation\n×\n@misc { metr-2026-simpler-ai-timelines-model ,\ntitle = {A simpler AI timelines model predicts 99% AI R&D automation in ~2032} ,\nauthor = {Thomas Kwa} ,\nhowpublished = {\\url{https://metr.org/notes/2026-02-10-simpler-ai-timelines-model/}} ,\nyear = {2026} ,\nmonth = {02} ,\n}\nCopy\nThomas Kwa\nSubscribe\nNote\nNotes are unpolished research updates directly from METR researchers, and have gone through less review than our research posts . They do not necessarily reflect the views of METR as a whole.\nIn this post, I describe a simple model for forecasting when AI will automate AI development. It is based on the AI Futures model , but more understandable and robust, and has deliberately conservative assumptions.\nAt current rates of compute growth and algorithmic progress, this model’s median prediction is >99% automation of AI R&D in late 2032. Most simulations result in a 1000x to 10,000,000x increase in AI efficiency and 300x-3000x research output by 2035. I therefore suspect that existing trends in compute growth and automation will still produce extremely powerful AI on “medium” timelines, even if the full coding automation and superhuman research taste that drive the AIFM’s “fast” timelines (superintelligence by ~mid-2031) don’t happen.\nWhy make this?\nThe AI Futures Model (AIFM) has 33 parameters; this has 8.\nI previously summarized the AIFM on LessWrong and found it to be very complex. Its philosophy is to model AI takeoff in great detail, which I find admirable and somewhat necessary given the inherent complexity in the world. More complex models can be more accurate, but they can also be more sensitive to modeling assumptions, prone to overfitting, and harder to understand.\nAIFM is extremely sensitive to time horizon in a way I wouldn’t endorse.\nIn particular, the “doubling difficulty growth factor”, which measures whether time horizon increases superexponentially, could change the date of automated coder from 2028 to 2049! I suspect that time horizon is too poorly defined to nail down this parameter, and rough estimates of more direct AI capability metrics like uplift can give much tighter confidence intervals.\nScope and limitations\nFirst, this model doesn’t treat research taste and software engineering as separate skills/tasks. As such, I see it as making predictions about timelines (time to Automated Coder or Superhuman AI Researcher), not takeoff (the subsequent time from SAR to ASI and beyond). The AIFM can model takeoff because it has a second phase where the SAR’s superhuman research taste causes further AI R&D acceleration on top of coding automation. If superhuman research taste makes AI development orders of magnitude more efficient, takeoff could be faster than this model predicts.\nSecond, this model, like AIFM, doesn’t track effects on the broader economy that feed back into AI progress the way Epoch’s GATE model does.\nThird, we deliberately make two conservative assumptions:\nNo full automation: as AIs get more capable, they never automate 100% of AI R&D work, just approach it. In the AIFM, automation of coding follows a logistic curve that saturates above 100% (by default 105%), meaning that there is a capability level where they automate all coding.\nNo substitutability: Automation follows Amdahl’s law (speedup = $1/(1-f)$ when automated tasks are much faster than manual tasks)\nThis was constructed and written up fairly quickly (about 15 hours of work), so my opinions on parameters and some of the modeling assumptions could change in the future.\nThe model\nWe assume that AI development has the following dynamics:\nResearch progress is Cobb-Douglas between labor and compute\nSoftware efficiency S follows a Jones model\nThe key metric we want to predict, fraction of automatable tasks $f$ , increases as a sigmoid in log(effective compute)\nZero substitution between tasks\nLabor\nHumans work on ONLY non-automated tasks\nHuman labor on each task is $L/(1-f)$\nAI labor on each task is $CS/f$, but this doesn’t matter because we assume human labor is the bottleneck (since humans work slower than AIs)\nThis implies the following model:\n\\[S'(t) = R(t) S^{1 - \\beta} = \\left(\\frac L {1-f}\\right)^\\alpha C^\\zeta S^{1 - \\beta}\\]\n\\[f(t) = \\sigma(v(\\log (C(t)S(t)) - \\log E_{hac}))\\]\nwhere\n$S(t)$ is the algorithmic efficiency multiplier (I assume that training and inference efficiency improve at the same rate)\nso $C(t)S(t)$ is the effective compute of the best AI\n$f(t)$ is the fraction of automated tasks at time $t$\n$R(t)$ is research production at time $t$\n$L(t)$ is human labor, specified as an input time series\n$C","chars":5000,"total_chars":17741,"offset":0,"truncated":true,"next_offset":5000}

read_page

{
  "url": "https://arxiv.org/abs/2608.13417",
  "max_chars": 3500
}

{"ok":true,"url":"https://arxiv.org/abs/2608.13417","text":"[2608.13417] Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development\nSkip to main content\nSearch\nSubmit\nDonate\nLog in\nSearch arXiv\nPress Enter to search &middot; Advanced search\n-->\nComputer Science > Artificial Intelligence\narXiv:2608.13417 (cs)\n[Submitted on 13 Aug 2026]\nTitle: Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development\nAuthors: Yiwei Li , Wanli Yang , Hexiang Tan , Xiangzhou Huang , Zhengyu Chen , Ziran Li , Borun Chen , Shanglin Lei , Huaisheng Zhu , Hao Tian , Fei Sun , Xunliang Cai , Jingang Wang\nView a PDF of the paper titled Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development, by Yiwei Li and 12 other authors\nView PDF\nHTML (experimental)\nAbstract: Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.\nSubjects:\nArtificial Intelligence (cs.AI)\nCite as:\narXiv:2608.13417 [cs.AI]\n(or\narXiv:2608.13417v1 [cs.AI] for this version)\nhttps://doi.org/10.48550/arXiv.2608.13417\nFocus to learn more\narXiv-issued DOI via DataCite\nSubmission history\nFrom: Yiwei Li [ view email ] [v1]\nThu, 13 Aug 2026 16:11:22 UTC (1,542 KB)\nFull-text links:\nAccess Paper:\nView a PDF of the paper titled Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development, by Yiwei Li and 12 other authors\nView PDF\nHTML (experimental)\nTeX Source\nview license\nCurrent browse context:\ncs.AI\n< prev\n|\nnext >\nnew\n|\nrecent\n| 2026-08\nChange to browse by:\ncs\nReferences & Citations\nNASA ADS\nGoogle Scholar\nSemantic Scholar\nexport BibTeX citation\nLoading...\nBibTeX formatted citation\n&times;\nloading...\nData provided by:\nBookmark\nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer ( What is the Explorer? )\nConnected Papers Toggle\nConnected Papers ( What is Connected Papers? )\nLitmaps Toggle\nLitmaps ( What is Litmaps? )\nscite.ai Toggle\nscite Smart Citations ( What are Smart Citations? )\nCode, Data, Media\nCode, Data and Media Associated with this Article\nalphaXiv Toggle\nalphaXiv ( What","chars":3500,"total_chars":5076,"offset":0,"truncated":true,"next_offset":3500}

read_page

{
  "url": "https://arxiv.org/abs/2610.12466",
  "max_chars": 3500
}

{"ok":true,"url":"https://arxiv.org/abs/2610.12466","text":"[2610.12466] On the estimation and validity of AI time horizons---a statistical look at the METR plot\nSkip to main content\nSearch\nSubmit\nDonate\nLog in\nSearch arXiv\nPress Enter to search &middot; Advanced search\n-->\nComputer Science > Artificial Intelligence\narXiv:2610.12466 (cs)\n[Submitted on 8 Oct 2026]\nTitle: On the estimation and validity of AI time horizons---a statistical look at the METR plot\nAuthors: Drew T. Nguyen , William Fithian\nView a PDF of the paper titled On the estimation and validity of AI time horizons---a statistical look at the METR plot, by Drew T. Nguyen and William Fithian\nView PDF\nHTML (experimental)\nAbstract: METR's 50\\% time horizon measures the human completion time of software tasks that an AI solves with 50\\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that \\emph{converts} human time to AI difficulty; it is nearly flat in a region from 2--30 min but close to linear elsewhere. Hence, a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of $10 \\times$. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons' construct validity. We suggest that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks.\nSubjects:\nArtificial Intelligence (cs.AI) ; Machine Learning (cs.LG); Machine Learning (stat.ML)\nCite as:\narXiv:2610.12466 [cs.AI]\n(or\narXiv:2610.12466v1 [cs.AI] for this version)\nhttps://doi.org/10.48550/arXiv.2610.12466\nFocus to learn more\narXiv-issued DOI via DataCite\nSubmission history\nFrom: Drew Nguyen [ view email ] [v1]\nThu, 8 Oct 2026 17:59:50 UTC (330 KB)\nFull-text links:\nAccess Paper:\nView a PDF of the paper titled On the estimation and validity of AI time horizons---a statistical look at the METR plot, by Drew T. Nguyen and William Fithian\nView PDF\nHTML (experimental)\nTeX Source\nview license\nAdditional Features\nAudio Summary\nCurrent browse context:\ncs.AI\n< prev\n|\nnext >\nnew\n|\nrecent\n| 2026-10\nChange to browse by:\ncs\ncs.LG\nstat\nstat.ML\nReferences & Citations\nNASA ADS\nGoogle Scholar\nSemantic Scholar\nexport BibTeX citation\nLoading...\nBibTeX formatted citation\n&times;\nloading...\nData provided by:\nBookmark\nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer ( What is the Explorer? )\nConnected Papers Toggle\nConnected Papers ( What is Connected Papers? )\nLitmaps Toggle\nLitmaps ( What is Litmaps? )\nscite.ai Toggle\nscite Smart Citations ( What are Smart Citations? )\nCode, Data, Media\nCode, Data and Media Associated with this Article\nalphaXiv Toggle\nalphaXiv ( What is alphaXiv? )\nLinks to Code Toggle\nCatalyzeX Code Finder for Papers ( What is CatalyzeX? )\nDagsHub Toggle\nDagsHub ( What is DagsHub? )\nGotitPub Toggle\nGotit.pub ( What is GotitPub? )\nHuggingface Toggle\nHugging Face ( What is Huggingface? )\nScienceCast Toggle\nScienceCast ( What is ScienceCast? )\nDemos\nDemos\nReplicate Toggle\nReplicate ( What is Replicate? )\nSpaces Toggle\nHugging Face Spaces ( What is Spaces? )\nSpaces Toggle\nTX","chars":3500,"total_chars":4646,"offset":0,"truncated":true,"next_offset":3500}

read_page

{
  "url": "https://www.anthropic.com/institute/recursive-self-improvement",
  "max_chars": 12000
}

{"ok":true,"url":"https://www.anthropic.com/institute/recursive-self-improvement","text":"When AI builds itself \\ Anthropic\nSkip to main content News Research Policy Commitments\nCompany\nTry Claude\nWhen AI builds itself\nOur progress toward recursive self-improvement, and its implications.\nFor most of AI’s history, humans drove every step in its development cycle. But at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work.\nTaken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self-improvement . We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for.\nUsing public benchmarks and previously unreported data from within Anthropic, The Anthropic Institute is showing that AI is already accelerating the development of AI systems. To take just one example: today, Anthropic engineers on average ship 8x as much code per quarter as they did from 2021-2025.\nThe technical trends discussed in this piece suggest that AI systems are going to become much more capable in coming years. These trends have huge implications. AI that can build itself would be a major development in the history of technology—one that could bring enormous good for the world in science, healthcare, and beyond. But full recursive self-improvement also might increase the risks of humans losing control over AI systems. If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important.\n2021–2023\nBuilding the first Claude\nIn the early days, work at Anthropic looked like work at any other tech company: people writing code and docs on laptops.\n2023–2025\nChatbots\nPeople used early chatbots to help with parts of the process, like generating short code snippets and copying the output into text editors.\n2025–2026\nCoding agents\nAs the agents became more capable, they were able to write and edit code on their own, sometimes entire files.\nToday\nAutonomous agents\nAgents can now run code themselves and delegate hours of work to other agents.\n20XX?\nClosing the loop\nIn the future, agents could become capable enough to build and train models themselves. If this happens, future versions of Claude could be continuously improved by Claude itself.\nEvidence from the outside world\nThe rate at which AI models improve is accelerating. The length of tasks that they can reliably complete on their own has been doubling roughly every four months, up from an earlier trend of doubling every seven months. In March 2024, Claude Opus 3 could complete software tasks that take humans about four minutes to complete. A year later, Claude Sonnet 3.7 managed tasks that took about an hour and a half. A year after that, Claude Opus 4.6 managed 12-hour tasks. 1 If this trend holds, tasks that take a skilled person days could come into range this year. In 2027, AI systems could be capable of tasks that take a person weeks.\nThe same pattern appears on coding and research benchmarks. Benchmarks measure the performance of models in a given domain, and they’re “saturated” when models achieve close to 100% performance. 2 SWE-bench is a standard test of real-world software engineering: it hands a model an actual open-source codebase and a real bug report, and asks it to write a code change that fixes the issue and passes the project’s own tests. Models have gone from scoring in the low single digits to saturating the benchmark in two years.\nCORE-Bench tests whether a model can reproduce existing research, a prerequisite for them to conduct original research. It gives an AI model the code and data behind a published paper, and asks it to rerun everything and confirm it can replicate the paper’s results. AI systems went from succeeding at reproducing the results roughly 20% of the time in 2024 to saturating the benchmark fifteen months later. METR, which runs the benchmark measuring how well models can complete long-duration tasks, found that Claude Mythos Preview could work for “at least” 16 hours and was “at the upper end of what [METR] can measure without new tasks.”\nPublic benchmarks say a lot about the capabilities of these systems. But they can’t reveal the impact AI systems are having on speeding up AI development itself. For that, we need direct evidence from within AI companies like Anthropic.\nEvidence from within Anthropic\nBuilding a frontier model takes two broad categories of work. There is engineering : writing the code, standing up the infrastructure, and overseeing the model training. And there is research : deciding what experiments to run, interpreting what comes back, and figuring out which ideas to try next.\nAcross both engineering and research, the picture is consistent. In engineering, Claude can be handed an underspecified problem and figure out how to solve it; humans supply the goal, but they no longer need to supply the method. In research, Claude can already match or outperform skilled humans at executing a well-specified experiment. However, large performance gaps persist when it comes to Claude exercising judgement in choosing goals in both engineering and research. That’s the gap between AI today and a future system that could autonomously design its own successor.\nIt’s common for employees at Anthropic to receive more open-ended and important tasks as they gain more experience. Early on, they execute a task someone else specified, like, “The export button isn’t working, please fix it.” With experience, they’re handed a goal and design the approach themselves, such as, “Investigate why the network slows down under heavy load.” At the most senior levels, they are deciding which problems are worth working on at all: “What should the team build next quarter?” We can use internal Anthropic data to see how far Claude has come in being able to handle these different kinds of tasks.\nClaude writes a significant proportion of Anthropic’s code. As of May 2026, more than 80% of the code we merge into Anthropic’s codebase was authored by Claude. 3 Before Claude Code launched in research preview in February 2025, this number was in the low single digits. That shift also shows up in the amount of output per engineer. Lines of code merged per engineer per day stayed constant through Anthropic’s first four years (2021-2024), then began to climb upward in 2025 when Claude began to run code rather than just suggesting it for an engineer to copy and paste. The slope steepened again in 2026 when models began to work autonomously over longer time horizons. These two inflection points are shown in the chart below. In the second quarter of 2026, the typical engineer was merging 8× as much code per day as they were in 2024. 4 This is because much of the code is written by Claude, with the engineer directing and reviewing, rather than typing it themselves.\nA caveat: Lines of code is an imperfect measure, as it measures quantity over quality. So 8 × lines of code/engineer/day in the second quarter of 2026 is almost certainly an overstatement of the true productivity gain. Nonetheless, it indicates an acceleration. At Anthropic, we don’t reward people for how many lines of code they write; rather, team members are producing more code simply because they’re using AI systems to write more code.\nThe increase in lines of code written lines up with subjective impressions of large productivity increases. In a March 2026 poll of 130 employees from across Anthropic research teams, the median respondent estimated that they produced around 4x as much output with Mythos Preview as they would have without access to any AI models, on the kinds of projects they would have been working on regardless. 5 We expect that the true degree of uplift in March was somewhat lower. 6 Nevertheless, we find the overall claim plausible, and in line with our other observations: a significant fraction of Anthropic technical staff is accomplishing their core work multiple times faster than they could without AI assistance.\nWe also see evidence that people at Anthropic are using Claude to do work that simply wouldn’t have happened otherwise, like building exploratory tooling and addressing long-deferred cleanup. For example, in April 2026, Claude shipped over 800 fixes that reduced a class of API errors by a factor of one thousand. The engineer overseeing Claude estimated that a human would have taken four years to complete this work; solving other people’s bugs is slow and painstaking, and humans struggle to hold that much unfamiliar context in their head at once.\n“ I started leaning hard into Claudifying about a year ago. That’s been a crazy adventure and it’s now been ~5 months since I last wrote any code myself. ” Anthropic employee*\nThe code that Claude writes is “good” and improving. “Good code” means two things: it works, and it is written in a manner that allows another engineer to understand it and build upon it. On the first criterion, the evidence is clear. The rate at which Anthropic staff correct, redirect, or take over mid-task from Claude has been falling steadily for a year, including on the most complex and open-ended tasks. This means problems with no clear specification, where the engineer isn’t sure what the answer looks like. This is evident in Claude’s success rate over time on tasks of different difficulties, as shown in the graph below. Claude writes code that works.\nHow to read this: Session success is determined by a Claude judge; a session is deemed successful if the Claude Code agent clearly succeeded at the user’s tasks without requiring corrections. Changes in workloads can lead to short-term fluctuations in success rates.\nOn the most open-ended tasks, Claude’s success rate reached 76% in May 2026, up 50 percentage points in six months. To give an example of tasks in this difficulty tier, a routine upgrade began crashing tens of thousands of training jobs. An engineer pointed Claude at the live incident with little more than some text content and cluster access. Working through the running jobs and testing one environment setting at a time, Claude isolated the single obscure debugging flag that was triggering the crash, reproduced it reliably, and confirmed a fix. In about two hours, Claude delivered what would normally be two to three days of work.\nThe second criterion is writing code that another engineer can understand and build on. Here the gap between humans and AI persists, but is closing fast. There isn’t full consensus among staff at Anthropic, but many believe that the Claude-written code was still worse in quality than human-written code at Anthropic in late 2025, and is roughly at parity today. We expect it to be better within the year.\nThis has changed the way that Anthropic now reviews its own code. Proposed changes to our codebase are now read by an automated Claude reviewer that looks for bugs, security flaws, and other defects before it can merge. Using this tool, we ran a retrospective analysis, and found that an automated Claude review of every change to our codebase would have caught roughly a third of the bugs behind past incidents on claude.ai before they ever reached production. The engineers who wrote that code are among the best in the world at building these systems. Claude is now catching the mistakes that they missed.\n“ Claude-written code was somewhat worse than human-written code at Anthropic in late 2025, is roughly at parity today, and we expect it to be strictly better within the year. ”\nClaude is good at running experiments to hit a goal that someone else has set. Every time Anthropic releases a model, we run the same test: we give Claude some code that trains a small AI model, and ask it to make that code run as fast as possible while still passing the same correctness checks. The goal and the success metrics are fixed in advance, so Claude’s job is to find speedups by","chars":12000,"total_chars":34056,"offset":0,"truncated":true,"next_offset":12000}