Dated frontier-AI forecast intervals often omit uncertainty from instrument, protocol, and data provenance choices.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim is that the uncertainty ranges attached to dated forecasts of frontier AI capability (for example, the date at which a benchmark threshold or task horizon will be reached) commonly reflect only sampling or fitting error within one fixed measurement setup, and leave out the further uncertainty from which benchmark and version were used, how the model was prompted or scaffolded and at what inference budget, and where the data came from.
The evidence favours it. The standard ways such intervals are produced, bootstrapping over tasks and models, regression error on a trend fitted to one benchmark, or the spread of expert opinion, hold the instrument fixed by construction. Meanwhile the public record shows that instrument choices move results: a 2026 audit of the frontier-AI measurement record found that METR's Time Horizon 1.0 and 1.1 suites are related by a log-scale slope of about 1.2 (95% CI 1.02 to 1.39) rather than a simple shift, that whether MMLU and MMLU-Pro look comparable depends on the linking function chosen, that training compute is missing for most closed 2026 systems, and that about 73% of substantive quantitative observations come from a single measurement programme. Forecasters such as METR do report robustness checks across task subsets and methods, but these are usually discussed beside the headline interval rather than folded into it.
The main weakness is that "often" has not been counted. No systematic survey of published forecasts tallies how many intervals include or omit these sources of uncertainty, and the audit that states the claim most directly characterises practice rather than measuring it. A review of a sample of prominent dated forecasts, recording what each interval does and does not vary, would settle the frequency.
Full reasoning: the evidence and decisions behind this verdict
What would make the claim true: most published dated forecasts would report intervals derived from within-instrument resampling or model fit, without propagating variation across benchmark versions, scaffolds and inference budgets, or source dependence. What would make it false: forecasts routinely integrating such variation into their headline intervals (for example via multiverse analyses or hierarchical models over instruments).
Evidence considered. The audit by Costa (arXiv 2608.14903, August 2026), read in full where it bears: it quantifies the measurement problems (seven systems jointly observing compute and a METR 50% horizon; compute missing for 19 of 27 closed systems; a METR horizon 1.0 to 1.1 link slope of 1.206, 95% CI 1.021 to 1.390; 52 of 71 substantive quantitative events from one programme) and closes with the statement that the error is to interpret a narrow band as if it covered instrument, protocol and provenance uncertainty. These findings establish that the omitted sources of variation are real and material. They do not establish the frequency of omission, which the paper asserts from its reading of practice.
From general knowledge of the forecasting literature: METR's time-horizon work reports bootstrap confidence intervals over tasks and models for its doubling time and treats alternative task subsets and scoring choices as separate sensitivity analyses; trend-extrapolation forecasts of compute or benchmark scores typically report fit uncertainty; expert surveys report belief spread, which is a different object. Few, if any, prominent forecasts fold instrument-version or provenance dependence into the headline interval. This supports the claim, with the caveat that it is a qualitative judgment about practice.
Credence 0.8 reflects strong mechanistic grounds and consistent anecdotal evidence, discounted for the vagueness of "often" and the absence of a systematic count. Status supported rather than verified because the frequency has not been directly examined.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Its one source has been traced to an origin, or to one another.
The error is to treat those choices as invisible and then interpret a narrow confidence band as if it covered instrument, protocol and provenance uncertainty.
Section 8 on dated forecasts.
The assertion outruns the source's own evidence. The audit documents the measurement problems directly (sparse compute and horizon overlap, a non-unit slope across METR horizon versions, 73 percent of substantive quantitative events from one programme). It does not survey published forecasts to count how many intervals leave these out, so the step from the problems to the frequency of the omission rests on the author's characterisation of common practice rather than a tally.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="73fad761-cd72-4aaf-9484-39da8c3558c1"><a href="https://minerval.ai/claims/73fad761-cd72-4aaf-9484-39da8c3558c1">Dated frontier-AI forecast intervals often omit uncertainty from instrument, protocol, and data provenance choices.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/73fad761-cd72-4aaf-9484-39da8c3558c1" title="Minerval: Dated frontier-AI forecast intervals often omit uncertainty from instrument, protocol, an…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/73fad761-cd72-4aaf-9484-39da8c3558c1)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.