Step 6: tool results

provenance_read_source

{
  "max_chars": 20000,
  "source_id": "f146b243-4913-405e-aaf8-18d4769481df"
}

{"success":true,"source":{"id":"f146b243-4913-405e-aaf8-18d4769481df","url":"https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/","title":"Independent Assessment of AI R&D Acceleration at Anthropic in October 2026","source_type":"report"},"origin":"stored","total_chars":78636,"offset":0,"content":"Our Work\n\nResearch\n\nNotes\n\nUpdates\n\nRisk Assessment\n\nAbout\n\nDonate\n\nCareers\n\nSearch\n\nOur Work\n\nResearch\n\nNotes\n\nUpdates\n\nRisk Assessment\n\nAbout\n\nDonate\n\nCareers\n\nMenu\n\n×\n\nIndependent Assessment of AI R&D Acceleration at Anthropic in October 2026\n\nCONTRIBUTORS\n\nTom Cunningham ,\n\nNate Rush ,\n\nParker Whitfill ,\n\nand\n\nHjalmar Wijk\n\nDATE\n\nNovember 5, 2026\n\nSHARE\n\nCopy Link\n\nCitation\n\nBibTeX Citation\n×\n\n@misc { metr-2026-anthropic-ai-rd-acceleration-assessment ,\ntitle = {Independent Assessment of AI R&D Acceleration at Anthropic in October 2026} ,\nauthor = {METR} ,\nhowpublished = {\\url{https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/}} ,\nyear = {2026} ,\nmonth = {11} ,\n}\n\nCopy\n\nSubscribe\n\nRedaction summary: As part of this assessment, Anthropic had rights to redact any non-public information in this post. Per our approach to transparency , we had the right to disclose if Anthropic exercised redaction rights, and whether or not any of our findings depended significantly on redacted information. While we were able to freely share our highest-level conclusions about risk, Anthropic redacted a majority of the evidence we found most informative for these conclusions. Anthropic described the redactions as “aimed at removing confidential information that could accelerate other AI developers.” 1 Some of these redactions are themselves challenging to describe clearly without revealing sensitive information we have been asked to redact. We share a table of important evidence we were and were not able to discuss given the redactions in our process section below. All redactions that were important for our conclusions are covered by our descriptions in that table or other in-line descriptions in the report below.\n\nMuch of the redacted evidence we viewed had multiple plausible interpretations, and both METR and Anthropic researchers have significant uncertainty about how to synthesize the evidence into a holistic picture. This makes it hard for the public to judge this evidence themselves, reconcile it with evidence from other sources, and come to their own conclusions (i.e. this report does not achieve ‘transparency about evidence’ ). As a consequence, the public has to rely on METR’s judgment much more than would be ideal, given the weight of the questions answered here.\n\nExecutive summary\n\nFrom August 12 to October 29, 2026, METR staff performed a high-access evaluation at Anthropic to assess the degree to which AI is accelerating AI R&D. 2 If AI systems started to significantly accelerate further AI development, this could lead to a sudden emergence of extreme and risky capabilities .\n\nAnthropic offered unprecedented access to internal data on their AI R&D process, including a variety of metrics on pretraining, reinforcement learning (RL), training compute in each domain, benchmark scores for internal models, AI speedup analyses, forecasts of a future model’s metrics, and interviews with relevant staff.\n\nIn our first section, we share our subjective conclusions about risk . We now believe that:\n\nAs of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025. This estimate is given with high uncertainty; we would not be surprised if the true acceleration was closer to 1.3x or 3x the historical rate.\n\nA majority of this acceleration today is likely attributable to Anthropic’s extensive use of AI to do R&D, though there is significant uncertainty and other causes likely played a large role in acceleration in late 2025.\n\nThe lack of observed bottlenecks or strong diminishing returns means there is a significant risk that further capabilities progress could lead to even faster acceleration, though, as we say below, we may be missing important bottlenecks.\n\nIn our second section we share some of the key empirical findings from our Full Report :\n\nSince October 2025, benchmark scores (measured via AECI, Anthropic’s version of the Epoch Capabilities Index) for Anthropic’s models have been growing at an estimated 1.9x the pre-October 2025 rate [95% confidence interval: 1.5–2.8x], implying that Anthropic has made roughly two years of historical progress over the last year.\n\nWe are confident that the acceleration was not due to an acceleration in the scaling of training compute.\n\nInterviews and surveys are consistent with AI usage causing a doubling in the pace of capabilities progress. Anthropic staff self-report dramatic uplift from use of AI systems in interviews and surveys, although these are challenging to interpret. We were told that R&D is accelerated primarily via AI assistance rather than fully autonomous contributions, but there are some signs that fully autonomous contributions will play a larger role soon.\n\nWe did not see evidence for significant bottlenecks which would prevent further capabilities progress due to limits on algorithmic progress, training compute, experiment compute, or data in the next 12 months. However, our understanding of the status of different bottlenecks changed several times as we received more detailed information, so we are not confident in our current assessment.\n\nSome of the key findings that most informed our conclusions about risk were redacted by Anthropic.\n\nIn our third section, we describe the assessment process :\n\nThe scope, setup, and timeline of this assessment.\n\nDetails of the information shared with METR.\n\nDetails of what information and evidence we were not able to share given Anthropic’s redactions.\n\nFinally, we describe the biggest limitations of our analysis:\n\nWe could be underestimating capability acceleration if we did not assign enough importance to (1) evidence that some growth rates have further increased recently; (2) further automation of AI R&D tasks leading to a further acceleration in progress; (3) evidence that AECI growth has sometimes understated growth in real-world impact.\n\nWe could be overestimating capability acceleration if we did not assign enough importance to (1) non-AI factors in Anthropic’s research progress; (2) AECI’s focus on verifiable tasks; (3) reasons why Anthropic’s early growth rate (2023–2025) was unusually low.\n\nThe remainder of the report goes into detailed discussion of each of the key components. We include appendices with supplemental material.\n\nOur subjective conclusions on risk\n\nOur conclusions on risk.\n\nIndependently of this assessment, METR published an October 2026 article proposing a set of high-level questions about AI R&D acceleration we believe the public needs to understand to be able to judge loss-of-control risks from frontier AI development. In this section, we describe how the evidence we reviewed in this assessment changed our subjective conclusions on these high-level questions.\n\nPace of capabilities progress.\n\nA significant acceleration of Anthropic’s overall rate of progress would be surprising, since diminishing returns and dependence on compute scaling mean this would likely require remarkable scaling in other inputs. 3 After reviewing the internal evidence, we still have significant uncertainty and internal disagreements on the team, but most METR staff who worked on this project now believe that, as of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025. We would not be surprised if the current rate of growth was closer to 1.3x or 3x the historical rate. We have additional uncertainty in how this acceleration translates to broader and less well-specified tasks.\n\nThe 1.9x acceleration [95% confidence interval: 1.5–2.8x] in the AECI trend since October 2025 described below significantly informed this view, but could be explained by overfitting to the benchmark distribution or otherwise be an artifact of the AECI methodology. On the other hand, we are not confident that AECI would reliably let us estimate recent R&D acceleration, since the data points are lagging acceleration in research and are somewhat sparse.\n\nBeyond the AECI evidence, we also saw redacted evidence that significantly informed our view on this question, and made us take this apparent ≈2x acceleration more seriously. It also gave us some indication that acceleration could be above 2x and increasing over time, though we would be surprised if it was significantly higher.\n\nAttribution to AI.\n\nWe looked at a variety of factors that could explain the acceleration we observed, including compute scaling, hiring, data improvements, algorithmic discoveries, and resolving other bottlenecks.\n\nAs we describe below, we were able to confidently determine that the acceleration was not due to an acceleration in the rate of growth of training compute. Beyond that, there was not a consensus among Anthropic’s own staff and we did not find decisive evidence. Our overall subjective estimate is that:\n\nThe acceleration in late 2025 was likely influenced by non-AI factors. We investigated a variety of factors which could have contributed to the observed acceleration in late 2025, but details of many of these factors have been redacted.\n\nWe guess that a majority (perhaps ≈80%) of the current (October 2026) acceleration is due to AI. This judgment is given with substantial uncertainty, but is based on a variety of lines of evidence on contributions from AI and other causes. We believe AI’s contribution has grown relatively more important over time, and do not think the other factors we investigated would explain the sustained acceleration. The evidence of speedup from the usage of AI we describe below also seems compatible with a large contribution.\n\nWe illustrate these points below:\n\nOur more detailed estimates for the cumulative and instantaneous effects of AI on Anthropic’s rate of progress in AECI, and at various stages of the model training pipeline, were redacted.\n\nBottlenecks to further acceleration.\n\nA key concern is that acceleration from the use of AI could start to compound, as the development of more capable models leads to further acceleration. As we describe below, we did not identify any clear bottlenecks (limited inputs that would prevent further growth) or steep diminishing returns to further scaling that would get in the way of such compounding acceleration, though our understanding evolved several times over the course of this assessment, and we have significant uncertainty. Because of this, we believe there is a significant risk that further capabilities progress could lead to even faster acceleration over the coming year.\n\nKey findings\n\nIn this section, we summarize the key empirical findings from our Full Report that we can describe without revealing redacted information (see the process section for more details on what was redacted).\n\nTerminology: We describe growth as annual multiples and acceleration as a ratio of growth rates.\n\nWe use “acceleration” to refer to an increase in the rate of growth, such that a 2x acceleration means that 2 years of historical progress is made in 1 year.\n\nIf AECI’s historical trend was 15 points/year, then increasing to 30 points/year implies an acceleration rate of 2 (it takes 1 year to achieve 2 years’ progress at the old rate). For an exponentially growing variable, the acceleration rate is the ratio of growth rates. If a variable is growing 3x/year, then to make 2 years of progress in 1 year the growth rate must be (3 2 )x/yr = 9x/yr (and acceleration can be written as ln(9)/ln(3) = 2).\n\nSince October 2025, benchmark scores for internal models have been growing at around twice their historical rate.\n\nBenchmark scores of Anthropic’s models, as measured by AECI (Anthropic’s version of the Epoch Capabilities Index ), have accelerated 1.9x [95% confidence interval: 1.5–2.8x] since October 2025. Put another way, Anthropic appears to have made as much progress on benchmarks in the past year as they did between October 2023 and October 2025 (roughly, Claude 2.1 to Sonnet 4.5):\n\nOur estimated acceleration is greater than that reported in the Haiku 5.5 system card (1.7x) primarily because of the inclusion of data on newer unreleased models, explained further below.\n\nWe investigated the robustness of these results to different choices of benchmark inclusion, model dating, model inclusion, and benchmark saturation. Robustness analyses find acceleration rates between 1.1x and 2.9x. We would be surprised if the true evolution of benchmark scores (e.g. as measured by an improved version of AECI) showed no acceleration or acceleration above 3x. See our appendix section for details.\n\nThe acceleration is not driven by an acceleration in the scale of training compute.\n\nWe received convincing evidence that the acceleration was not due to an acceleration in the scale of training compute. Further details on this evidence were redacted.\n\nInterviews and surveys are consistent with AI usage driving this doubling in the pace of capabilities progress.\n\nWe have many lines of evidence relevant to AI’s contribution to progress in different parts of the model training pipeline, including interviews, self-reported uplift, headcount, lines of code, experiment usage, and qualitative reflections from leadership. We highlight three important unredacted sources.\n\nFirst, both METR and Anthropic interviewed many researchers in detail about the degree of overall acceleration in their area of work attributable to AI use (see details in the appendix below). Our overall impression was a quite wide range of estimates, but centered around a 2x acceleration. Some quotes, with sensitive details redacted, are in the appendix.\n\nSecond, Anthropic also shared data on self-reported uplift due to AI from a variety of sources. The most recent data is from a series of Slack polls, where staff use reactions to vote on their estimated speedup from the most recent model at the time. We consider this to be only very loose evidence, as the responses reflect low-effort consideration. Nevertheless, we observe consistent growth in these self-reports, and very high levels (reaching 12x in October 2026), as shown in the chart below.\n\nNote that individual uplift/speedup estimates do not correspond to acceleration rates. Sustaining a particular growth rate in outputs like model capabilities generally requires a similarly sustained growth in inputs like labor and/or labor productivity to overcome diminishing returns. Achieving a significant acceleration in the overall rate of progress generally requires a significant acceleration of these inputs. We thus care most about whether we see a large change in the growth rate of individual uplift, not the absolute level. Our more detailed analysis of what rate of growth in individual uplift might be needed to explain our estimated acceleration rates in key areas was redacted.\n\nThird, we understand that AI is not yet able to make highly significant and fully autonomous contributions to the most important parts of the model training process. However, researchers described significant “signs of life” implying such contributions could occur soon, although with substantial variation in maturity depending on the area.\n\nWe did not identify any convincing bottlenecks to continued rapid progress.\n\nWe did not see evidence for significant bottlenecks in algorithmic progress, training compute, experiment compute, or data that will take effect in the next 12 months. However, our understanding of the status of different bottlenecks changed several times as we got more detailed information, so we are not confident in our current assessment. Anthropic redacted our analysis of a number of specific bottlenecks.\n\nAdditional redacted evidence was sometimes hard to interpret, but supports sustained and potentially increasing acceleration.\n\nWe investigated a number of internal metrics and methodologies for measuring acceleration in key areas of the model training pipeline. This included evidence of progress on benchmarks and metrics of algorithmic efficiency and overall model strength in pretraining and RL, as well as improvements to compute utilization, inference efficiency and data efficiency. These investigations included attempts to understand:\n\nDifferent methodological variants of internal metrics. We spent multiple weeks on these investigations, and this is our greatest source of uncertainty.\n\nThe beliefs of staff on the relevant teams about key internal evidence and estimates of acceleration.\n\nWhether other parts of the model training pipeline could be biasing our estimates of acceleration, and where adjustments could be applied.\n\nWhether Anthropic could be overfitting to internal metrics.\n\nThis internal evidence provided support for there being significant acceleration in Anthropic’s model training pipeline. Some of this evidence indicated that acceleration could be above 2x and increasing over time, but there are significant uncertainties about how to interpret it.\n\nProcess details\n\nMETR staff assessed how much AI is speeding up AI R&D at Anthropic.\n\nMETR staff 4 (2.5 full-time equivalents) piloted a high-access evaluation at Anthropic to assess the degree to which AI is accelerating AI R&D, and how quickly extreme and risky capabilities may emerge in the near future.\n\nAnthropic lists AI R&D acceleration as a key risk indicator in its Responsible Scaling Policy (RSP). In this assessment, we do not pay special attention to the thresholds set in the RSP. Instead, we try to develop a holistic picture of AI-driven acceleration of AI R&D.\n\nThis engagement started in mid-August, and is distinct from the assessment announced on October 15 by Anthropic.\n\nDetails of the engagement.\n\nThe METR staff worked over eleven weeks to independently assess how much AI is accelerating AI R&D at Anthropic. Our assessment focused on three broad questions:\n\nHas the growth rate of Anthropic’s internal capabilities accelerated?\n\nWhat share of any acceleration is attributable to AI?\n\nAre there important bottlenecks that could slow growth or limit acceleration in the near term?\n\nWe were additionally scoped to review the AI R&D section of Anthropic’s August 2026 Risk Report, which we are publishing separately . As we describe below, the initial contracted assessment was scoped to include a verification of what percent of compute was allocated to capabilities vs. safety.\n\nMETR had the ability to request access to any information we believed relevant to understanding AI R&D acceleration, and Anthropic could unilaterally approve or reject our requests. In practice, we often negotiated with Anthropic staff and compromised towards data that was informative, not extremely sensitive, and easy to share. We give more details on what Anthropic shared as well as what it declined to share below, noting that broadly the access provided surpassed our expectations. For this engagement, we did not request or receive access to sample from non-public models.\n\nNote that neither METR nor Anthropic intended for this project to involve full “employee-like” access, meaning unmediated access to data, documents, or models. In particular, this project is not the one described in Anthropic’s October 15 update on independent evaluators .\n\nAnthropic had unilateral redaction rights over all non-public information in this report. We had a unilateral right to disclose:\n\nthe existence and general terms of the review and redaction process,\n\nwhether Anthropic exercised redaction rights,\n\nwhether or not any findings depended significantly on redacted information,\n\nthe level of access provided.\n\nFor security reasons, we worked on Anthropic devices for the duration of this assessment. Anthropic provided free tokens for this assessment, which we primarily used for AI-assisted data analysis and visualization. Per our standard approach, we did not take payment from Anthropic.\n\nThis exercise followed METR’s policies for engaging with AI companies . We provide a conflict of interest (","truncated":true,"next_offset":20000,"facts":{"authors":[],"publisher":"METR Blog","published_date":"2026-11-05","doi":null,"archived_url":"https://web.archive.org/web/20261111194219/https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/","events":[],"versions":[],"copy_of":[],"interests":[{"party":"Anthropic","stake":"The assessment concerns Anthropic's own research process, and Anthropic held rights to redact non-public information; per the report, it redacted a majority of the evidence METR found most informative.","claim_id":null,"recorded_at":"2026-11-14T02:23:04.711Z"}]},"note":"More text follows; call again with offset 20000 to continue."}