Step 29: assistant

text

I've given the claim its first structure and assessment: **inconclusive**, with 60% confidence in that verdict and a 35% probability that the claim is true. Each piece of evidence has a real weakness, so neither side wins. The pass cost $0.79 of the $1.00 budget, and everything is recorded.

**What the claim is about.** Since October 2025, Anthropic's scores on its internal benchmarks (AECI) have been improving about 1.9 times faster than before (95% interval 1.5–2.8x). METR reported this from inside access in November 2026. The claim says this speed-up, measured on well-specified tasks that can be checked, will not carry over to AI research and development as a whole.

**Evidence for the claim (that it won't carry over):**
- METR itself says the benchmark trend could be overfitting or a quirk of how AECI is built. It also says it is unsure how the speed-up applies to broader, less well-specified tasks.
- Getting better scores on well-specified tasks is an easier target than getting better overall. Several AI Futures Project authors suggest the jump may come from moving compute into reinforcement learning, which raises benchmark scores more directly.
- An existing claim in the graph, already rated supported, says benchmarks overstate how well models do on real-world tasks.

**Evidence against the claim (that it will carry over):**
- Beyond the benchmark trend, METR cites staff reports that AI speeds up their work and internal evidence Anthropic redacted, attributing roughly 80% of the current speed-up to AI use.
- Maths and vulnerability discovery, two other fields where answers can be checked, have already fallen to AI.
- The problem is the quality of this evidence: most of it is redacted or self-reported, or comes from parties with a stake in the answer.

My reading is that a speed-up that stays entirely within benchmarks is unlikely. A weaker carry-over, especially to the hardest-to-measure parts of research, is quite possible. Public data on how fast algorithms are improving, or on other research output that isn't a benchmark score, would settle it.

**Structure:**
- I added two arguments. The one for the claim (well-specified tasks are easier to improve) I rated as giving slight support even if its premises hold. The one against it (gains are already carrying over to real research work) I rated likely if its premises hold, but those premises are weakly established.
- I linked two existing claims: "benchmarks overstate real-world capability" as support, and "benchmark software trends carry over to real-world software work" as a consideration against.
- I recorded METR's report as a source that raises the question without taking a side, and filled in the author and date for the AI Futures post. Importance is set to 0.6 and contestation to 0.7.

**Curator's suggestion.** The Curator suggested rewording the claim the other way round, so it says the speed-up does carry over. I kept the current wording. The claim's note on its direction says to keep it, and the change would reverse what the claim says, which the rules hand to the Curator rather than to me. The Curator can still make the change.

I did not notify any other claim's steward, because no other claim depends on this one yet.