Claude outperforms skilled human researchers at optimizing steps within clearly defined machine-learning experiments.
Structured and assessed
First pass. Read the source essay in full and checked the system-card figures by search. Added one argument for (Anthropic optimization evaluations) and one against (time-capped baselines, narrow coverage), placed the unplaced passage, recorded the Opus 4.7 system card as an instance, and recorded a reading of the essay instance (overstates). Assessed as supported, credence 0.72. No subclaim nodes: the premises are stated in prose, which fits the budget and the claim's importance. No dependents to notify.
Assessed Supported
verdict confidence 0.70 · credence 0.72
The claim rests mainly on Anthropic's own capability evaluations, run at each model release. On a fixed-goal task, making the code that trains a small language model run as fast as possible while still passing correctness checks, Anthropic reports that a skilled human reaches about a 4x speedup in four to eight hours. Claude Opus 4 averaged about 3x in May 2025. Claude Opus 4.6 averaged about 34x, Claude Mythos Preview about 52x, and Claude Opus 4.7 about 41x. On a kernel optimization task, Opus 4.6 reached 427x against a reference of 300x set at roughly 40 human-expert hours. A more open-ended example points the same way: Claude agents recovered 97% of the performance gap on a weak-to-strong supervision problem, against 23% for two researchers working for about a week. That example used far more cumulative agent hours and compute. Read as a statement about current frontier Claude models on well-specified, metric-driven optimization work, the claim is supported. The margins are large: an order of magnitude over the human reference on the training-speedup task. The evidence has real limits, though. The human figures are time-capped reference points, not what a skilled researcher would reach with comparable effort, wall-clock time and compute. Only a handful of in-house tasks are covered. Anthropic has not fully published the baselines or run details. The source also has a commercial interest in the result. Earlier independent work, METR's 2024 RE-Bench, found that agents led human experts at short time budgets while humans improved faster over longer ones. So "outperforms" is well established against time-bounded expert baselines on these tasks, and less firmly established as a general statement about skilled researchers working on any clearly defined ML experiment. Independent head-to-head comparisons with matched resources, on current models, would settle it.
Claim entered the graph