Step 10: tool results

read_page

{
  "url": "https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/",
  "offset": 40000,
  "max_chars": 20000
}

{"ok":true,"url":"https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/","text":"t has lower than 50% R². (This drops benchmarks where the logistic link function is a bad fit.)\n1.9x [1.4–2.6]\nStop using benchmarks after they hit 85% (for saturation robustness).\n1.9x [1.4–2.3]\nRobustness: An acceleration in AECI of around 2x appears robust to regularization.\nParameterization\nAcceleration\nFigure\nStandard AECI\n1.9x [1.5–2.8]\nabove\nDecrease L2 regularization by 10x.\n1.9x [1.4–2.8]\nIncrease L2 regularization by 10x.\n2.1x [1.6–3.0]\nRemove regularization entirely.\n1.9x [1.4–2.8]\nRe-center the L2 penalty on a linear time trend instead of towards 0: each model’s capability score and benchmark’s difficulty is pulled towards a line in its release date. Each benchmark’s discriminability is pulled towards 1.\n1.9x [1.4–2.8]\nRobustness: AECI shows less acceleration when we include non-Anthropic models.\nIf we take our main AECI results and plot all frontier models instead of just Anthropic models, we estimate a lower degree of acceleration.\nNotice the historical trend is estimated mostly from non-Anthropic models, while the current trend is estimated from mostly Anthropic models.\nThere are a couple of plausible interpretations of this data:\nAnthropic models were subfrontier in 2025, so Anthropic’s historical progress up to 2025 was understated. Therefore, the 1.9x acceleration numbers in the rest of the report are overstated, and 1.4x is more accurate.\nIf Anthropic models are less benchmark-optimized compared to other models, this could mean the historical trend is overstated as the earlier points are more overfit to benchmarks. Therefore, the 1.4x acceleration in the graph above is understated and the 1.9x we quote elsewhere is closer to accurate.\nNew internal models make a permanent slope change look more plausible than a one-time jump at Mythos Preview.\nPrior Anthropic system cards have posited the hypothesis that there was a one-time jump in capabilities at Mythos Preview after which there was a return to the previous trend. Recent models make this hypothesis look less likely than a permanent trend change.\nComparing our AECI estimates to various public estimates.\nThe following chart compares our acceleration estimate to various public estimates of acceleration.\nThe rows denote different methodologies for calibrating (A)ECI scores for each model, e.g. by changing benchmarks, which models are included in the score calibration, etc. The columns denote different methodologies for converting those scores into an acceleration over time, e.g. by subsetting to certain models, changing the release dates, etc.\nOur main takeaways are that:\nThe most important difference between the latest Anthropic model release, the Haiku system card (1.7x acceleration), and our estimate (1.9x acceleration) is the inclusion of Anthropic’s recent internal-only models.\nThe most important differences between ECI and our estimates are subsetting the trend over time to only Anthropic models and the inclusion of Anthropic’s recent internal-only models.\nModel 4 achieves >50% on all of its benchmarks.\nThe plot below shows all benchmark scores for frontier models in the AECI, scaled on a log-odds scale, implying that they should evolve linearly with a model’s latent AECI score (if the logistic assumption holds). The highlighted lines represent the 15 benchmarks used for Model 4. Notably, Model 4 scores above 50% on all these benchmarks. We can see that the distribution of benchmark difficulty was very different just 9 months ago: Opus 4.6 had many benchmarks on which it scored below 50%. This lack of benchmarks which are difficult for Model 4 makes us somewhat nervous about our ability to measure AECI now and in the future.\nMost of the benchmarks used for Model 4 appear reasonably linear in log-odds space.\nA specific concern in measuring acceleration with AECI is benchmarks with many broken tasks, or benchmarks with a fat-tailed distribution of task difficulty. The logistic link function assumes exponential tails of task difficulty; if we instead had a fat-tailed distribution of task difficulty, this would mean that going from 98% to 99% requires more AECI points than going from 80% to 90% (the logistic assumes they are approximately equally difficult). A fat-tailed distribution of task difficulty could cause us to underestimate acceleration; as described above, using cauchit or log-log link functions (which model the last few tasks in a benchmark as particularly difficult) results in acceleration estimates of 2.8x and 2.9x, respectively.\nWe perform a simple diagnostic test for task difficulty. For each of the 15 benchmarks used for Model 4 in AECI, we compared a model’s estimated AECI against the model’s log-odds score on that benchmark. If the AECI assumptions are satisfied, meaning task difficulty is logistic in a model’s latent ability, then we would expect each plot to be linear. If there are many broken tasks or a fat-tailed distribution of difficulty, then we would expect these fits to be concave (we would expect a convex fit if there are disproportionately many easy tasks).\nWe noticed there are 3 of 15 benchmarks with notably flat or concave fits. When we re-estimate AECI for all models without those 3 suspicious benchmarks, Model 4’s score increases by 1 point, from 178.6 to 179.7. We thus think there is some weak evidence for fat tails causing understatement in acceleration, but the magnitude is not large.\nTokens per benchmark may be too low.\nUsing a low expenditure in benchmarking could bias estimates of capability growth downwards if later models show higher returns to expenditure (i.e. if early models plateau with expenditure, but later models show high returns).\nAnthropic shared with us data on tokens used on benchmarks in the AECI. A typical task (one attempt) used about 2.5M input and 72k output tokens across internal benchmarks, and about 10M input and 94k output on agentic coding benchmarks. For a loose comparison, Epoch AI uses roughly 0.5M input and 10k output tokens on a typical task across a few public benchmarks (GPQA, AIME, SWE-bench), and 1M input and 8k output tokens on a specific agentic coding benchmark (SWE-bench). For another loose comparison, METR uses roughly 650k output tokens for its latest long-running software benchmarks, though it is possible the token accounting is not comparable.\nMoreover, it would be informative to see returns-to-expenditure curves for each model, i.e. scores elicited at different levels of expenditure. Anthropic’s system cards do report expenditure curves for some benchmarks (e.g. Opus 5.5 system card ). There is some recent evidence that it is possible to use very large amounts of inference compute to solve difficult problems (e.g. Navier–Stokes ). Overall, our guess is that the AECI scores for each model could be increased by a handful of points if inference tokens were scaled further. Our rough guess is that this would have similar returns across models, so the degree of estimated acceleration would be the same, though this is not something we looked into deeply and it is possible that recent models have better token scaling properties.\nAn acceleration in benchmark scores may fail to track acceleration in real-world impact.\nIdeally we want to know the real-world value, or potential impact, of each model, not its benchmark scores. If real-world impact is smoothly related to benchmark scores, then we should expect an acceleration in AECI to be associated with an equivalent acceleration in real-world effects. 5\nThere are a number of reasons why acceleration in AECI may fail to track acceleration in the real-world impact of model capabilities:\nIf real-world impact is a convex function of benchmark scores, e.g. if impact depends on crossing a capability threshold, then AECI would understate real-world acceleration. In some domains we have seen a steady increase in benchmark scores accompanied by sudden accelerations in real-world impacts, e.g. impacts on the rate of discoveries in cybersecurity or math ( ref ).\nIf the benchmarks in AECI cover only a subset of capabilities, and the real-world impact has diminishing returns in those capabilities, then AECI could overstate real-world acceleration. For example, suppose AECI primarily measures a model’s ability to solve formal problems, but there are diminishing real-world returns in that ability; then we might observe an acceleration in AECI without an acceleration in real-world impact.\nOverall, we are confident there has been at least a modest acceleration in the growth of benchmark scores. We believe there is significant risk that acceleration in AECI may not map cleanly to acceleration in real-world impact.\nEvidence of AI speedup\nThere is considerable evidence of AI speedup.\nWe define AI speedup as the ratio between a staff member or team’s output with AI and without AI, noting that output can often be difficult to measure or define.\nWe now briefly survey the overall evidence for AI speedup, concluding:\nThere is some evidence for high speedup, though the precise amount of speedup is highly uncertain.\nFully autonomous contributions to AI R&D are not currently a large contributor to overall progress, but they may be in the near future.\nRedactions in this section.\nAnthropic redacted evidence of speedup that was closely connected to specific parts of their model training pipeline, as well as our discussion of how the evidence in this section related to our estimates of acceleration in various parts of the model training pipeline. Moreover, a claim about inference compute usage at Anthropic was redacted because it was priced in compute terms and we did not find it informative enough to request it in API prices. They also redacted sensitive details from the quotes from researchers below.\nInformal self-reports of uplift have been steadily growing.\nAnthropic shared data on self-reported uplift due to AI from a variety of sources. The most recent data is from a series of Slack polls, where staff use reactions to vote on their estimated speedup from the most recent model at the time. We consider this to be only very loose evidence: we expect the responses come from an unrepresentative subset of staff and the responses reflect fairly low-effort consideration. Additionally, there is some evidence that self-report survey estimates of uplift can overestimate true productivity impact.\nNevertheless, we observe consistent growth in these self-reports, shown in the chart below. For the most capable internal model in October 2026, average self-reported uplift was ≈12x.\nThese self-reported estimates have grown but it is difficult to know how to extrapolate them. The plot above fits uplift against time, using both a log-linear fit and a superexponential fit.\nInterviews broadly support AI doubling the rate of growth.\nBoth METR and Anthropic interviewed many researchers (see details in the following section ). We used Claude to classify the quotes according to whether they support 2x or more acceleration from AI in their area of work, relative to working without AI. We interpret the results as mildly in favor of 2x or greater acceleration:\nYes: 2\nLeans yes: 3\nBorderline/unclear: 4\nLeans no: 0\nNo: 2\nThe quotes, with sensitive details redacted, are in the following section .\nThere are signs of life for fully autonomous contributions in some parts of the pipeline.\nIn addition to the uplift of human labor, we are also interested in autonomous AI R&D. We understand that AI is not yet able to make highly significant and fully autonomous contributions to the most important parts of the model training process. However, researchers described significant “signs of life” implying such contributions could occur soon, although with substantial variation depending on the area. Further details were redacted. This evidence came from:\nMultiple interviews from researchers working to automate different parts of the training pipeline.\nInternal reports on the degree of automation in different parts of the pipeline.\nQuantitative analysis on the percent of tasks in a relevant pipeline that were built by humans or AIs.\nInterview quotes on AI acceleration\nAnthropic and METR staff conducted interviews with a number of researchers about AI R&D acceleration in their work. Interviews vary in length and depth, but we expect researchers spent significantly more time on these than answering the surveys above.\nTo classify interviews, we had Claude look over all our notes and take all relevant quotes and code them as supporting >2x acceleration due to AI in their area of work, ambiguous or against. Each row in the following table corresponds to a single person and all their relevant quotes. Note that each interviewee often interpreted the question in somewhat different ways, e.g. whether they are referring to cumulative or instantaneous speedup, or whether they are referring to personal or org-wide speedup.\nWe used the following prompt:\n“Can you please look through all the interview tabs and all relevant interview docs outside of this google doc. [redacted context]. Can you make a table where each row is a person and then we put all their quotes together. Then please add a column which is your overall determination if this person supports a > 2x acceleration due to Claude in their work.”\nPerson\nQuotes\nSupports >2x acceleration in their area of work from Claude?\nResearcher A\n“Could definitely endorse a greater than 2X speedup” — METR interview notes “My inside view is by far the largest effect is model acceleration” — METR interview notes “The AI acceleration is larger than all the other effects.” — METR interview notes “Now only bottlenecked on ideas & compute, no longer bottlenecked on code.” — METR interview notes “I would definitely endorse >2 years of progress in 1 year and I don’t think 4 is crazy.” — METR interview notes “I think there’s some truth to the ‘we just started trying harder’ story, but IMO the bulk of the effect is recent models.” — METR interview notes “Like 80% reduction in time between idea and experiment, other 20% some mix of making fewer mistakes and being able to measure fiddly things that would’ve required too much one-off work before.” — METR interview notes “I think there was a medium bump when it was good enough to code at all and a big bump with Opus-4.5/Opus 4.6 and an even bigger bump with mythos-preview” — Anthropic interview notes\nYes\nResearcher B\n“It is hard to see how I am not being 3x more productive. I feel like I have postdocs and grad students, and we are working towards a larger project. I think the simplest explanation is Claude—just from my individual perspective that feels sensible.” — METR interview notes\nYes\nResearcher C\n“this is probably the biggest factor imo. When people present experiments, they most often just point to an [agent scaffold] thread. Experimentation overhead is driven towards zero super fast.” — Anthropic interview notes “I think CC was already good enough before mythos-preview to implement most ideas around Opus 4.5” — Anthropic interview notes\nLeans yes\nResearcher D\n“It is just much easier to [perform] lots of [redacted] experiments that would have taken days. People don’t try [redacted] things bc it takes 1-2 days to implement per idea, but now its ~free with mythos-preview” — Anthropic interview notes “I personally think this is a large lift and has led to me trying far more ideas.” — Anthropic interview notes “On a linear scale, ~0 uplift pre Opus 4.6/Opus-4.5, ~some uplift that was small with them, large uplift with mythos-preview + [agent-scaffold]. I would guess we’re just starting to fully switch over … and we’ll see a continued slope increase / inflection.” — Anthropic interview notes\nLeans yes\nResearcher E\n“I do genuinely think Claude (claude code, [agent-scaffold]) has been fairly significant. I’ve seen us discovering very promising directions ~entirely because people can spam 10 ideas at [agent-scaffold] and see what sticks.” — Anthropic interview notes “A lot of [redacted] research work is just implementation, babysitting, etc. Opus-4.5 in claude code is already pretty helpful with this, even if it mostly can’t do end to end.” — Anthropic interview notes\nLeans yes\nResearcher F\n“The slope change is 50-80% Claude” — Anthropic interview notes “Change is especially notable for this manager, who can de-risk ideas with Claude enough to get someone else excited, and would not have been able to do that at all as a manager of 10 before.” — Anthropic interview notes (paraphrase)\nBorderline\nResearcher G\n“Inflection point matches his vibes as to what’s been happening. Two biggest contributors here are Claude automation and hiring.” — Anthropic interview notes (paraphrase) “Claude lets researchers try many things quickly, most of which won’t work. …” — Anthropic interview notes\nUnclear\nResearcher H\n“I think it’s true [that] we’re now trying a lot more experiments at the same time because of claudification.” — Anthropic interview notes\nUnclear\nResearcher J\n“Before Opus-4.5 -> Claude not useful for autonomous code / Opus-4.5 -> first time useful for autonomous code / mythos-preview + [agent-scaffold] -> first time useful for autonomous experiment” — Anthropic interview notes\nUnclear\nResearcher K\n“IMO [hiring is] probably the biggest reason …” — Anthropic interview notes\nNo\nResearcher L\n“Guesses 15-20% is due to AI. Don’t believe in self-reported productivity increase.” — METR interview “My favored interpretation: [redacted non-AI factor]” — METR interview “(1) opus45 couldn’t write code; (2) opus46 could write a little; (3) Mythos Preview a sharp inflection; (4) Model 2 is another really large improvement in how useful these things are.” — METR interview “definitely the case that we’re accelerated by claude .. Even though this rate of progress by end of year will be even faster … my attribution [mostly not to Claude] and uncertainty remain.” — METR interview “One impact is SWE acceleration, which is extremely substantial. We’re definitely benefiting from that differentially between 2025 and 2026.” — METR interview\nNo\nAdversarial robustness\nWe believe this exercise was reasonably adversarially robust.\nAs this is an independent assessment, it is important for us to do our due diligence and understand how hard it would have been for Anthropic to systematically mislead us, i.e., how likely it is that we could have been led to an incorrect conclusion intentionally.\nAdversarial robustness weakness 1: We relied on data from a few Anthropic employees. Most of our data was received through two technical staff members; however, we interviewed around 15 other Anthropic employees on a range of topics. We did not notice any discrepancies between what our primary contacts said and what our interviewees said. Given that many of these Anthropic employees were chosen by us, not Anthropic, we think this would have been challenging to pull off if we were being given purposefully misleading information.\nAdversarial robustness weakness 2: Most analysis relied on Claude. Claude was involved in many stages of this project, and so in principle could be distorting our perception of the data. We believe Claude was deeply involved in:\nthe majority of data collection and analysis that we received,\nwriting a significant share of the documents we received,\nthe majority of our own analysis.\nWe did not detect any significant discrepancies consistent with a systematic bias even when checking against human interviews. We suspect this is not a significant risk.\nOverall, we think it is extremely unlikely Anthropic was actively trying to mislead us. However, we believe greater access would allow us to increase the degree of adversarial robustness.\nProcess transparency disclosures\nCOI disclosure.\nWe applied v1.0 of our conflict of interest policy when selecting the team for this project, which was the live policy at the start of this project.\nAt the start of the project, one participating team member was in discussions with Anthropic about a possible role. They receiv","chars":20000,"total_chars":78371,"offset":40000,"truncated":true,"next_offset":60000}