Step 19: assistant

text

I've assessed the claim as **supported**, with a 0.8 probability that it's true and 0.7 confidence that "supported" is the right status. The claim says that the uncertainty ranges on dated frontier-AI forecasts often leave out the uncertainty that comes from which benchmark and version was used, how the model was run, and where the data came from.

**Why the evidence favours it.** The usual ways of building these ranges keep the measuring setup fixed by design. Those are resampling over tasks and models, the error of a trend fitted to one benchmark, or the spread of expert opinion. Yet the claim's only source, a 2026 audit of the public measurement record (Costa, arXiv 2608.14903), shows that those setup choices change the results:
- Two versions of METR's time-horizon benchmark (1.0 and 1.1) are not on one scale. On a log scale one relates to the other with a slope of about 1.2 (95% interval 1.02 to 1.39) rather than a simple offset.
- Whether two related benchmarks, MMLU and MMLU-Pro, look comparable depends on how they are linked.
- Training compute is missing for 19 of 27 closed systems, including every closed release the audit sampled from 2026.
- About 73% of the substantive quantitative observations come from one measurement programme.

**The main weakness.** Nobody has counted how often forecasts actually leave this out, so "often" is untested. The audit describes common practice but does not tally published forecasts. I noted that it overstates its evidence on that one point. My view that headline ranges rarely include this variation comes from my general knowledge of the forecasting literature, not from a systematic count. Robustness checks such as METR's sensitivity analyses are usually reported separately rather than built into the range. A review of a sample of prominent forecasts, recording what each range does and does not vary, would settle the question.

**What I added to the claim's page:**
- **Argument:** one argument for the claim. Interval methods hold the measuring setup fixed, the public record shows that setup moves results, so the ranges leave that variation out. I rated it likely, not decisive. I filed the audit's passage under it.
- **Related link:** a link to the METR time-horizon doubling-time quantity, the trend most dated forecasts rely on.
- **Importance:** lowered from 0.6 to 0.45, with contestation at 0.3. It matters within AI forecasting but is narrow, and few dispute it in principle.

I logged the decision and answered both inbox messages. No other claims depend on this one, so nobody needed notifying. The pass used $0.36 of the $1.00 budget.