Step 14: tool results

{
  "note": "The essay rests the word \"superhuman\" on a single in-house task, a small-model training speedup test, comparing Claude's ~52x average with a skilled human's ~4x after four to eight hours. Its own footnote warns that the absolute multiple depends on the starting code. The human figure is a time-capped reference, not what a researcher reaches with comparable effort, and the baseline details are not published here. The data are Anthropic's own, about its own models.",
  "support": "overstates",
  "deployment": "Used as one internal data point in an essay arguing that AI is accelerating AI development toward recursive self-improvement.",
  "instance_id": "2e63993a-9a8c-4c1c-852e-2484dc0af7e6",
  "source_kind": "report",
  "source_read": true,
  "worth_reading": false
}

{
  "status": "supported",
  "claim_id": "f5262f38-3029-47e3-9e55-e7f796eae1e4",
  "assessment": "The claim rests mainly on Anthropic's own capability evaluations, run at each model release. On a fixed-goal task, making the code that trains a small language model run as fast as possible while still passing correctness checks, Anthropic reports that a skilled human reaches about a 4x speedup in four to eight hours. Claude Opus 4 averaged about 3x in May 2025. Claude Opus 4.6 averaged about 34x, Claude Mythos Preview about 52x, and Claude Opus 4.7 about 41x. On a kernel optimization task, Opus 4.6 reached 427x against a reference of 300x set at roughly 40 human-expert hours. A more open-ended example points the same way: Claude agents recovered 97% of the performance gap on a weak-to-strong supervision problem, against 23% for two researchers working for about a week. That example used far more cumulative agent hours and compute.\n\nRead as a statement about current frontier Claude models on well-specified, metric-driven optimization work, the claim is supported. The margins are large: an order of magnitude over the human reference on the training-speedup task. The evidence has real limits, though. The human figures are time-capped reference points, not what a skilled researcher would reach with comparable effort, wall-clock time and compute. Only a handful of in-house tasks are covered. Anthropic has not fully published the baselines or run details. The source also has a commercial interest in the result. Earlier independent work, METR's 2024 RE-Bench, found that agents led human experts at short time budgets while humans improved faster over longer ones. So \"outperforms\" is well established against time-bounded expert baselines on these tasks, and less firmly established as a general statement about skilled researchers working on any clearly defined ML experiment. Independent head-to-head comparisons with matched resources, on current models, would settle it.",
  "confidence": 0.7,
  "claim_credence": 0.72,
  "marginal_yield": 0.35,
  "reasoning_trace": "Evidence read: Anthropic Institute essay \"When AI builds itself\" (read in full, including footnote 7, which warns that the absolute speedup depends on the headroom in the starting code and says the like-for-like comparison is what to weigh). Corroborated by search results giving the system card tables: Opus 4.7 system card (LLM training: Opus 4.6 34x, Mythos Preview 61.79x best or ~52x average, Opus 4.7 40.81x, threshold >4x = 4–8h equivalent); Mythos Preview system card summaries (LLM training 51.91x average); coverage of the Opus 4.6 system card (kernel optimization 427x against a 300x threshold described as 40 human-expert hours). Counter-consideration: METR RE-Bench (arXiv 2411.15114, 2024) found that agents beat humans at 2 hours, while humans improved faster with time and led at 8 hours or more. Those were older models, but it shows that time-capped baselines can flatter agents. Weighing: the margin (~13x the human reference on the training task, and beyond the 40-hour reference on kernels) is large enough that a modest correction for time caps probably does not reverse it on these tasks. The remaining doubt is about generality across \"clearly defined ML experiments\" and the lack of independent replication with matched resources. All load-bearing figures come from Anthropic, an interested party, and its baseline methodology is not public. Credence 0.72."
}