Step 17: assistant

thinking

text

I've assessed the claim as **supported**, with confidence 0.72. I gave no credence because the claim is a judgement about adequacy rather than a plain fact. The claim is that a single 50% task-completion horizon is a poor summary of AI-agent capability for forecasting.

**Why I think it holds:**
- The 50% horizon is one point read off a fitted curve of success rate against task length, on one suite of mostly software tasks. METR's own paper (Kwa et al., 2025) reports that the horizon at 80% success is 4 to 6 times shorter. A forecast of when agents can do real work therefore can't be read off the 50% figure.
- That one figure also says nothing about how scaffolding, inference budgets or repeated attempts change results, or how well the task suite carries over to other domains.
- The claim's existing source, a 2026 audit preprint by Costa, reaches the same conclusion. It recommends replacing the single figure with success curves across task length, attempts and time budgets, though it finds no single replacement number.

**The best case against:** METR found the 80% horizon doubles at about the same rate as the 50% horizon (204 versus 207 days). If the whole curve moves in parallel, the 50% trend is a fair proxy for the *rate* of progress, though not for capability at the reliability real work needs. So the claim is strongest for forecasts of absolute capability or readiness to deploy, and weaker for forecasts of growth rate only. Whether the parallel pattern holds as models and task suites change is the main open question.

I only skimmed the opening of the audit preprint. The METR figures come from a search-result excerpt of the paper (Section 3.2.1), not a full reading of it.

**What I added to the page:**
- **Arguments:** two named arguments with written forms and ratings. The argument from information lost by a one-number summary is rated "likely" if its premises hold. The argument from parallel trends at other reliability levels, against the claim, is rated "slight".
- **Supporting claim:** I linked the existing claim that METR's 50% horizons above roughly 16 human-hours are unreliable. That claim hasn't been assessed yet.
- **See-also links:** to the claim that the horizon keeps doubling about every seven months, and to the claim that agent performance varies a lot across repeated runs.
- **Source:** the METR paper, recorded as mentioning the claim without taking a side.
- **Importance and contestation:** importance 0.4 and contestation 0.35, since METR itself partly concedes the point.

I've logged the decision and answered both inbox messages. The claim has no dependents, so I sent no notifications. This pass used about $0.42 of the $1.00 budget, and I put the likely value of another pass at 0.3 out of 1.