Step 6: tool results

provenance_read_source

{
  "offset": 20000,
  "max_chars": 20000,
  "source_id": "f146b243-4913-405e-aaf8-18d4769481df"
}

{"success":true,"source":{"id":"f146b243-4913-405e-aaf8-18d4769481df","url":"https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/","title":"Independent Assessment of AI R&D Acceleration at Anthropic in October 2026","source_type":"report"},"origin":"stored","total_chars":78636,"offset":20000,"content":"COI) disclosure and excerpted contract language in the process transparency disclosures appendix plus the AEF-1 checklist .\n\nDetails of Anthropic’s redactions.\n\nOur ~100-page Full Report attempts to answer the key questions we recently described as crucial to answer in future AI R&D assessments in detail. We summarize the evidence sources we were given, and which we were able to include in this public report, in the table below.\n\nTable of shared and redacted evidence sources\n\nType of information\nShared with METR?\nResults described in public report\n\nMeasurements of internal capabilities: benchmark capabilities of internal models\nYes\nYes\n\nMeasurements of internal capabilities: key intermediate measures\nYes\nNo\n\nAlgorithmic progress estimates\nYes\nNo\n\nEvidence of AI’s contribution\nYes\nPartly\n\nMeasurements of key R&D inputs\nMostly\nNo\n\nDetails of bottlenecks and forecasts\nPartly\nNo\n\nInterviews with staff\nYes\nPartly\n\nAnthropic’s redactions also meant that we could not communicate:\n\nOur more detailed estimates of acceleration and attribution to AI usage for key areas of the model training pipeline.\n\nOur more detailed breakdown of progress attributable to compute, data and algorithmic progress, other than the key takeaway that “We are confident that the acceleration was not due to acceleration in training compute.”\n\nMore detailed answers to the limitations questions in our proposal, which would require more discussion of methodological details.\n\nForecasts of metrics for future models.\n\nThey also made some other more minor redactions throughout, which we attempt to describe in each section of this report.\n\nAnthropic described the redactions as “aimed at removing confidential information that could accelerate other AI developers.”\n\nKey dates.\n\nAugust 12: Anthropic begins sharing confidential information with METR.\n\nAugust 14: Anthropic shares the unredacted version of its August AI R&D Risk Report with METR.\n\nSeptember 12: METR delivers a first draft of its Full Report to Anthropic for feedback.\n\nSeptember 18: METR delivers a first draft of its Risk Report Review to Anthropic for feedback.\n\nSeptember 22–October 22: METR iterates on the Risk Report Review and the Full Report with further information.\n\nOctober 22: METR delivers its ~100-page Full Report to Anthropic.\n\nOctober 25: Anthropic shares redaction guidance, summarized above.\n\nOctober 26: METR delivers a draft of this public report to Anthropic.\n\nOctober 27–29: Anthropic tightens the redaction guidance, and METR iterates on the public report as a result.\n\nOctober 29: METR submits a draft public report.\n\nOctober 29: Anthropic and METR agree on an ~5 day extension to more carefully accommodate the new redaction guidance.\n\nNovember 3: METR submits the final public report.\n\nMETR received a significant amount of private data from Anthropic.\n\nMETR made extensive requests for information on every part of the AI R&D pipeline, and Anthropic shared more information than was initially scoped for this project, surpassing our expectations. Information was typically shared within a few days of requesting it.\n\nA detailed description of the data we had access to was redacted. A summarized version is as follows:\n\nInternal benchmark scores and AECI for internal models, plus sample tasks, rubric answers and agent expenditure on those tasks.\n\nHistorical data on algorithmic efficiency and overall model strength in pretraining and RL, as well as improvements to compute utilization, inference efficiency and data efficiency.\n\nHeadcount and attribution of progress to specific staff.\n\nTraining compute metrics in many relevant domains.\n\nInternal analysis on relevant open questions.\n\nInternal surveys of self-reported AI uplift by team.\n\nLines of code and pull requests across the company and some research teams.\n\nAI agent inference spend over time.\n\nInternal documents on autonomous research contributions.\n\n~15 interviews with key staff, covering a predefined but wide scope that excluded particularly sensitive topics.\n\nIn a small number of cases, Anthropic did not provide relevant information when they deemed it too sensitive.\n\nAnthropic also offered extensive staff time and expertise to this project. We believe the two staff we worked most closely with are the two highest-context researchers on AI R&D acceleration measurement at Anthropic. Both of these staff met with us frequently for live discussion, and spent substantial time collecting data, doing analysis, and providing feedback.\n\nAnthropic canceled a compute allocation component of this assessment.\n\nThe contracted scope of this project originally included a verification of Anthropic’s compute allocation, especially the allocation between capabilities and safety . However, Anthropic changed the scope of this project midway to exclude compute verification and instead just focus on AI R&D acceleration. This component was canceled before METR came to any substantive conclusions on compute usage at Anthropic. Anthropic stated this was due to wanting to keep the engagement focused on risk assessment for the time being, and that it would revisit compute allocation verification later on.\n\nLimitations\n\nWe report our best guesses at ways in which we could be misinterpreting the evidence.\n\nWe discuss the main ways we could be making mistakes in our interpretation of the evidence. We highlight:\n\nLimitations that result from the structure of this assessment.\n\nLimitations that result from incorrect analysis.\n\nLimited access may have led us to miss important context or data.\n\nDespite the access provided to METR, it is clear that staff-level access would enable a significantly deeper assessment. Our pipeline of information was relatively narrow, and a small number of our requests for data were denied. Being able to read documents on demand, observe internal discussion and debate, and have frequent and diverse conversations would have allowed for a richer view. While we believe we had access to the bulk of relevant information, it is possible we missed important information because we did not know it existed or could not look for it ourselves.\n\nOver the course of our exercise, we substantially changed our interpretation of key data as more context was shared with us, in some cases leading to multiple reversals on critical questions. We believe fuller access would have dramatically sped up that process of proper interpretation, and we think there likely remain points which we may have misinterpreted due to insufficient context.\n\nAcceleration could be higher than this report estimates.\n\nOur best guesses as to how we might be underestimating acceleration:\n\nWe may have missed recent further acceleration. The rate of growth could be undergoing continuous acceleration, e.g. hyperbolic growth. While we do not see clear evidence of this in the AECI trends, there is other evidence that makes this seem more plausible which Anthropic redacted. If growth is hyperbolic, we are likely underestimating instantaneous acceleration and substantially understating acceleration in the near future. In addition, we observe model-level outcomes only after a model is trained, meaning we observe acceleration in effective R&D with a lag roughly equal to the model’s training time.\n\nAutomation could be closer than we estimated. We understand that AI cannot yet fully automate substantial parts of Anthropic’s model training pipeline; however, there are “signs of life,” discussed below. It is possible that automation is close, and leads to very rapid acceleration once human bottlenecks are removed.\n\nAECI could understate real-world capability impact. We have sparse data on the relationship between benchmark scores and real-world impact. As one example, ECI scores on math have not accelerated substantially, but we are now seeing the resolution of Millennium Prize Problems that have evaded decades of human effort. Real-world impacts may be downstream of capability thresholds that models suddenly pass, leading to sudden impacts that are hard to predict.\n\nAcceleration could be lower than this report estimates.\n\nOur best guesses as to how we might be overestimating acceleration:\n\nNon-AI factors could explain most of the acceleration. It’s possible that the non-AI factors explain all or most of the recent acceleration. This implies we have less reason to expect high future growth or future accelerations.\n\nAcceleration might be limited to well-specified tasks. Much of the evidence we looked at measures performance on well-specified tasks that are easy to grade automatically or with a language model. If recent models have been growing more quickly on “clean” tasks than on “messy” tasks (relative to historical relative growth rates), then we may be overstating the acceleration in real-world impact. For instance, recent progress may be overfitting to the types of tasks which are included in AECI.\n\nOur acceleration numbers could be overstated if the historical rate of growth was unusually low. If progress historically was below its normal or natural rate due to temporary factors, this could lead us to overestimate the amount of acceleration. As we describe in an appendix , Anthropic models had a lower historical growth rate on AECI than the broader industry frontier before late 2025, which could explain some of the acceleration.\n\nAppendices\n\nMethodology\n\nRedactions in this section.\n\nNone of this section was redacted, but note that some of the methodological discussion may only apply to the Full Report.\n\nWe include data from Anthropic internal-only models.\n\nThis includes the following models (these are the codenames shared with us; we understand they have different internal codenames):\n\nUnreleased 2022, which finished training in 2022\n\nModel 2, which finished training in July 2026\n\nModel 3, which finished training in August–September 2026\n\nModel 4, which finished training in September–October 2026\n\nMETR’s Full Report includes a forecast for a future model, but this was redacted.\n\nWe use piecewise linear fits to detect acceleration.\n\nWe typically estimate time trends by an ordinary least squares fit with a single estimated breakpoint; we then report acceleration as the ratio of the two estimated linear trends. We adopt the best-fitting breakpoint on standard AECI data as our default breakpoint on other graphs, to create a consistent breakpoint throughout this report. However, in some places we consider deviations from this, and test estimated acceleration as per the best fit on each graph.\n\nWe organize our analysis around 3 growth rates: the historical growth rate, the current growth rate and the counterfactual growth rate.\n\nWe want to assess the degree to which AI automation is speeding up AI progress. For any measure of progress (e.g. model capabilities as measured by AECI), we can estimate three numbers:\n\nHistorical growth rate: the growth rate prior to a detected trend change.\n\nCurrent growth rate: the growth rate as of October 2026.\n\nCounterfactual growth rate: the growth rate as of October 2026, if Anthropic could not use AI.\n\nThe first two are observable quantities. The counterfactual growth rate we can only guess at. The following plot illustrates these measures on illustrative data:\n\nIt is important to distinguish between instantaneous acceleration due to AI, as we have defined it, and cumulative acceleration due to AI, i.e. what fraction of overall progress since some point in time (e.g. October 2025) can be attributed to AI. Some interviews and documents are implicitly or explicitly discussing cumulative acceleration, though we generally expect cumulative acceleration to be lower than instantaneous acceleration, due to non-AI acceleratory factors in 2025, and due to the growing influence of AI over time.\n\nIn the Full Report, we organize indirect evidence with a simple accounting framework.\n\nWe want to understand if there has been an acceleration in overall capabilities due to AI R&D. To reach our subjective conclusions, we organized our evidence using the following accounting framework, but all relevant usage of it was redacted.\n\nArrows into a box indicate that box is a function of the preceding boxes, so for example, pretraining strength is a function of algorithms, data quality, and training compute. We use the term RL to include all contributions not included in pretraining.\n\nThis framework lets us start at the right (capabilities) and determine whether there has been an acceleration. Then we can go back one level to see if there has been an acceleration in pretraining or RL, and then go back one further level to see what inputs are causing that acceleration. This allows us to get a deeper understanding of what is happening to the inputs.\n\nIn the Full Report, analyses were all documented in an accompanying notebook and dashboard.\n\nFor reproducibility we maintained a notebook and dashboard with all results in the Full Report. Where feasible we included references to the notebook or dashboard, and most charts had a “stamp” which identified their provenance.\n\nBenchmark scores (AECI)\n\nAECI appendix overview.\n\nWe expand on the main AECI analysis in a few ways.\n\nWe explain the methodology behind our main estimation of a 1.9x acceleration in AECI.\n\nWe show the degree of estimated acceleration varies based on reasonable alternative analysis choices:\n\nVarying the link function: 1.1x–2.9x.\n\nVarying the snapshots used: 1.9x–2.2x.\n\nVarying the benchmarks used: 1.9x–2.2x.\n\nVarying the regularizer used: 1.9x–2.1x.\n\nWe explain why these results differ from AECI estimates in previous Anthropic system cards.\n\nWe go into detail on potential problems with AECI such as:\n\nAnthropic’s internal benchmarks are increasingly saturated: Model 4 achieves greater than 50% on all the AECI benchmarks on which it was tested. However, measured acceleration appears reasonably robust to saturation concerns.\n\nToken spend may be too low to maximally elicit models.\n\nThe logistic link function could be misspecified on a few benchmarks. However, on the whole we think it is a reasonable approximation.\n\nThere are various reasons why an acceleration in benchmark scores could overstate or understate an acceleration in real-world impacts.\n\nRedactions in this section.\n\nThis section was very minimally redacted by Anthropic. The only information redacted was:\n\nThe names of specific benchmarks.\n\nOne graph that shows AECI results for a subset of models chosen on the basis of information that is redacted.\n\nMethodological details on AECI estimation.\n\nWe use Epoch’s public code to estimate AECI values for each model. We generate 95% confidence intervals via bootstrap by drawing benchmark scores with replacement, stratified by model.\n\nOur underlying data includes the AECI scores of many snapshots per model. These snapshots differ in both dates and AECI scores.\n\nFor publicly released models, we include the snapshot that was publicly released and the date that snapshot was available for use by most Anthropic employees. For example, if a model has two snapshots: one available to Anthropic employees on November 10 and one available to Anthropic employees on November 20, and the second snapshot was released publicly on December 1, then our procedure would assign November 20 as the release date. We think there are many reasonable alternative dating procedures (e.g. at first snapshot release instead of last snapshot), which are shown below in one of the robustness exercises.\n\nFor models that are never released, we include snapshots that received substantial internal usage and the date most Anthropic employees could use them.\n\nWe use the date at which the model could be used by most Anthropic employees instead of public release dates; this allows us to make apples-to-apples comparisons with unreleased models. These in general show slightly higher degrees of acceleration as Anthropic is holding onto more recent models for slightly longer before their public release.\n\nWhen we fit trends over time we use ordinary least squares with equal weight to each snapshot on the frontier. The confidence interval in the trend estimation only accounts for uncertainty in the AECI scores for each model, and does not include sampling uncertainty over which models are included in the fit.\n\nWe estimate a learned breakpoint in the trend, though we exogenously impose that there is 1 breakpoint. We estimate the breakpoint is October 7, 2025 in our main AECI specification. For our robustness checks, we maintain this October 7, 2025 breakpoint to ensure comparability.\n\nRobustness: Estimates of AECI acceleration vary 1.1x–2.9x depending on the link function.\n\nWe show below that estimates of AECI acceleration are somewhat sensitive to assumptions about the distribution of task difficulty within each benchmark. In principle, we could discriminate between these assumptions based on the data, but in practice most functions fit the data about equally well. We acknowledge useful suggestions from Anthropic staff on analysis here.\n\nAECI assumes that a model’s score on a benchmark is a function of the model’s latent strength, and specifically that the function is logistic (shown below in black). This can be interpreted as an assumption on the distribution of difficulty of tasks in a benchmark, including how many tasks are broken or what fraction can be guessed. The following figure illustrates a few different link functions. The logistic function assumes a symmetric distribution with exponentially declining tails. The log-log function assumes a long tail of hard tasks; the complementary log-log function assumes a long tail of easy tasks.\n\nThe following table shows the estimated acceleration depending on various assumptions of the link function. Note all these link functions are global, i.e. not benchmark-specific.\n\nLink function\nIntuitive assumption on task distribution\nAcceleration\nOut-of-sample R²\nFigure\n\nStandard logistic\nBenchmarks have a symmetric difficulty distribution of problems with exponential tails.\n1.9x [1.5–2.8]\n87.2%\nabove\n\nProbit\nBenchmarks have a smaller tail of both easy and hard problems.\n1.8x [1.3–2.7]\n87.1%\n\nCauchit\nBenchmarks have a larger tail of both easy and hard problems.\n2.8x [2.0–4.1]\n86.1%\n\nComplementary log-log (heavy left tailed)\nBenchmarks have a larger tail of only easy problems (for example, if there are many easily guessable tasks).\n1.1x [0.8–1.5]\n87.1%\n\nLog-log (heavy right tailed)\nBenchmarks have a larger tail of only hard problems (for example, if there are a few extremely hard problems).\n2.9x [2.1–4.9]\n86.9%\n\nLearned link function (Stukel two-tail family, 2 globally learned parameters)\nLength of both tails is globally estimated from data.\n2.6x [1.8–4.1]\n87.0%\n\nThe quality of fit is fairly similar across each of these link functions (except cauchit), and we do not have strong a priori reasons to prefer one. We continue to use logistic for our main results since it is the default link function used by Epoch and Anthropic.\n\nRobustness: An acceleration in AECI of around 2x appears robust to which snapshots are included.\n\nWe consider various alternative choices for which models to use when estimating AECI growth:\n\nParameterization\nAcceleration\nFigure\n\nStandard AECI (snapshots that got substantial internal usage)\n1.9x [1.5–2.8]\nabove\n\nOnly the highest snapshot per model\n1.9x [1.4–2.8]\n\nOnly the first snapshot per model\n2.2x [1.3–3.2]\n\nA subset of models (for redacted reasons)\n1.9x [1.4–3.3]\n[this graph was redacted by Anthropic]\n\nAll snapshots\n1.9x [1.4–2.6]\n\nRobustness: An acceleration in AECI of around 2x appears robust to which benchmarks are included.\n\nWe consider various alternative choices for which benchmarks to use when estimating AECI growth:\n\nParameterization\nAcceleration\nFigure\n\nStandard AECI\n1.9x [1.5–2.8]\nabove\n\nDrop vision benchmarks (e.g. OSWorld, MMMU).\n2.2x [1.7–3.5]\n\nKeep only benchmarks that Claude labels as relevant to AI R&D (e.g. drop multilingual MMLU, honesty benchmarks).\n2.0x [1.5–2.9]\n\nDrop benchmarks that on average decrease a","truncated":true,"next_offset":40000,"note":"More text follows; call again with offset 40000 to continue."}