Dated frontier-AI forecast intervals often omit uncertainty from instrument, protocol, and data provenance choices.
Structured and assessed
First pass. Read the sole source (Costa 2026 audit) in its relevant sections; recorded a reading (overstates: frequency not tallied). Created one argument with no subclaims, placed the ingestion passage on it, linked the METR doubling-time quantity as related. Assessed supported, credence 0.8; importance lowered to 0.45, contestation 0.3. No dependents to notify.
Assessed Supported
verdict confidence 0.70 · credence 0.80
The claim is that the uncertainty ranges attached to dated forecasts of frontier AI capability (for example, the date at which a benchmark threshold or task horizon will be reached) commonly reflect only sampling or fitting error within one fixed measurement setup, and leave out the further uncertainty from which benchmark and version were used, how the model was prompted or scaffolded and at what inference budget, and where the data came from. The evidence favours it. The standard ways such intervals are produced, bootstrapping over tasks and models, regression error on a trend fitted to one benchmark, or the spread of expert opinion, hold the instrument fixed by construction. Meanwhile the public record shows that instrument choices move results: a 2026 audit of the frontier-AI measurement record found that METR's Time Horizon 1.0 and 1.1 suites are related by a log-scale slope of about 1.2 (95% CI 1.02 to 1.39) rather than a simple shift, that whether MMLU and MMLU-Pro look comparable depends on the linking function chosen, that training compute is missing for most closed 2026 systems, and that about 73% of substantive quantitative observations come from a single measurement programme. Forecasters such as METR do report robustness checks across task subsets and methods, but these are usually discussed beside the headline interval rather than folded into it. The main weakness is that "often" has not been counted. No systematic survey of published forecasts tallies how many intervals include or omit these sources of uncertainty, and the audit that states the claim most directly characterises practice rather than measuring it. A review of a sample of prominent dated forecasts, recording what each interval does and does not vary, would settle the frequency.
Claim entered the graph