Step 17: assistant

text

I've assessed the claim as **supported**: I put the chance it is true at about 72%, with 0.7 confidence that "supported" is the right reading of the evidence. Every load-bearing figure comes from Anthropic itself, which has a commercial stake in the result.

**The evidence for it:**
- **Training-speedup test.** Anthropic gives each new model code that trains a small model and asks it to make the code run faster while still passing correctness checks. A skilled human reaches about a 4x speedup in four to eight hours. Claude Opus 4 averaged about 3x in May 2025. Claude Opus 4.6 averaged about 34x, Claude Mythos Preview about 52x and Claude Opus 4.7 about 41x.
- **Kernel optimization.** Opus 4.6 reached 427x, against a 300x reference that Anthropic sets at about 40 hours of expert human work.
- **A weak-to-strong supervision problem.** Claude agents closed 97% of the performance gap, against 23% for two researchers working about a week. The agents used far more time and compute: 800 cumulative hours and about $18,000.

I read Anthropic's "When AI builds itself" essay in full, including its footnote warning that the size of the speedup depends on how much room the starting code leaves. I checked the 4.6, 4.7 and Mythos Preview figures against search results that quote the system cards' tables, not the PDFs themselves.

**The evidence against it:**
- The human figures are time-capped reference points, not what a researcher would reach with comparable time and compute.
- Only a few in-house tasks are covered, and Anthropic hasn't fully published its human baselines or run details.
- METR's 2024 RE-Bench study found that AI agents beat human experts at short time budgets, but humans improved faster and led at 8 hours or more. Those were older models, but it shows how time-capped baselines can flatter agents.

The margins are large enough that I don't think a fair correction for time caps reverses the result on these specific tasks. The open question is whether it holds across clearly defined machine-learning experiments generally. Independent head-to-head comparisons with matched time and compute would settle it.

**What I recorded:**
- **Arguments:** one for the claim, from Anthropic's evaluations, rated "likely" if its premises hold. One against, from the time-capped baselines and narrow coverage, rated "slight".
- **Sources:** the passage the system had left unattached is now placed on the "for" argument. I added the Opus 4.7 system card as a source. My note on the essay says it overstates its own evidence: it rests the word "superhuman" on one in-house task.
- **Settings:** importance 0.6, contestation 0.4. I added no subclaim nodes; the premises are stated in the arguments' text.
- **Housekeeping:** I logged the decision and answered both inbox messages. Nothing depends on this claim, so no one needed notifying.

The pass used about $0.61 of the $1.00 budget.