Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence

Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence Fabricio F. Costa, PhD, MBA, PMP1,2,3,4∗ 1AIx4All, LLC, Sunnyvale, CA, USA 2HCLTech, Santa Clara, CA, USA 3Genomic Sciences and Biotechnology Program, UCB, Bras´ılia, Brazil 4Cancer Biology and Epigenomics Program, Stanley Manne Children’s Research Institute, Ann & Robert H. Lurie Children’s Hospital of Chicago, Northwestern University Feinberg School of Medicine, Chicago, IL, USA Evidence cutoff: 12 August 2026. Preprint, comments welcome. Abstract Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time or expert belief. This paper audits whether the public measurement record supports those connections before another trend is fitted. I construct a frozen, event-centric record through 12 August 2026 with 62 selected systems, 12 versioned benchmarks, seven capability or impact criteria, 144 graded events, 27 source records and 408 typed relations. The record is an audit sample, not a census. Only seven systems jointly observe estimated training compute and a METR 50% task horizon. Training compute is absent for 19 of 27 closed systems, including every selected closed release from 2026, while none of the 35 open-weight systems has a METR horizon observation. Benchmark succession creates a second break: a seven-system link from METR Time Horizon 1.0 to 1.1 has a log-scale slope of 1.206 (95% CI 1.021–1.390), whereas a six- system MMLU–MMLU-Pro comparison appears approximately shift-like under logit and probit links but not under linear or logarithmic links. The observed bridges have about 80% power only for slope departures near 25%. Provenance is concentrated: 52 of 71 sub- stantive quantitative events (73.2%) come from one measurement programme, and 76.1% are laboratory releases. A review of 56 additional methodological and empirical sources identifies 16 complementary measurement directions spanning resources, inference budgets, reliability, agentic work, safety, human preference, field outcomes and forecast backtesting. No direction supplies a replacement scalar. The result is not that frontier-AI forecasting is impossible, but that a defensible dated forecast is a claim about a versioned measure- ment system—its joins, protocols, links and source dependence—not merely a fitted curve or calendar date. Keywords: AI evaluation; measurement; benchmark linking; forecast validation; training compute; provenance; metascience 1 Introduction Forecasts of advanced AI now appear as expert probability distributions, benchmark extrapola- tions, compute trends and backtested models of agent performance [18, 19, 45]. The quantitative form is attractive: select an indicator, estimate its rate of change and solve for a date at which ∗Correspondence: fcosta@aix4all.com. This independent scholarly work uses only public sources, received no external funding, and represents the author’s views rather than those of the listed affiliations. The author declares no financial interest in organisations whose systems or forecasts are discussed. 1arXiv:2608.14903v1 [cs.AI] 14 Aug 2026

a threshold is crossed. Yet the date is meaningful only if the target is operational, the obser- vations are comparable, the progression variable is jointly observed and the uncertainty model includes changes in the measuring instrument. This paper audits those preconditions. It began as an attempt to construct an event-based forecast of frontier-AI capability milestones. The supporting infrastructure could be built; the intended progression analysis could not be identified without strong additional assumptions. The failure was not a software failure. It was a measurement finding: capability and resource records were sparsely joined, benchmark versions did not always preserve scale, evaluation protocols changed, and repeated observations were heavily dependent on a small number of sources. The distinction matters. A criterion can be well defined while its public evidence is not forecastable. Conversely, a record can contain many scores yet fail to measure one stable quantity. Existing work has proposed taxonomies of performance, generality and autonomy [40], holistic evaluation suites [31], ability-oriented measurement [22], explicit evaluation estimands [6] and statistical models that distinguish fixed-benchmark accuracy from generalized accuracy [26]. The contribution here is to connect those ideas to the actual longitudinal record used in frontier-AI forecasting. The paper makes three contributions. First, it provides a reproducible event-centric audit in which releases, results, revisions, corrections, contamination findings, field experiments and source assertions are separately timestamped and versioned. Second, it quantifies three fore- castability bottlenecks in that record: sparse joint observability, benchmark-linking uncertainty and provenance concentration. Third, it translates a deep source review into a measurement portfolio and a set of study-design requirements rather than proposing another universal leader- board score. The semantic graph is therefore infrastructure, not the main scientific claim. Its role is to make questions such as “which systems connect compute to capability?”, “which result changed after a benchmark revision?” and “which conclusions depend on one programme?” answerable from the same evidence ledger. 2 From a Score to a Forecastable Quantity A benchmark result is not an intrinsic property of a model. It is an observation generated by a model, an instrument and a protocol at a time. A compact representation is Ym,b,v,p,r,j,t = hb,v(θm,t; p, r, j) + εm,b,v,p,r,j,t, (1) where m is the model, b the benchmark family, v its version, p the prompting or agent protocol, r the inference-resource budget, j the judge or scoring procedure and t the observation date. The latent quantity θ may itself be multidimensional. Equation 1 is not fitted as a complete latent-variable model here; it is an accounting identity that makes the assumptions behind a trend explicit. A dated forecast adds at least three more objects: a target estimand, a progression axis and a threshold rule. If training compute is the progression axis, compute and the outcome must be co-observed or missingness must be modelled. If benchmark version v is replaced by v + 1, a linking function must place the two instruments on a defensible common scale. If inference budget or scaffolding changes, the score may move even when the underlying system does not. If many observations share a source, their errors need not be independent. This accounting view is increasingly reflected in measurement science. Statistical evaluation can distinguish performance on one fixed item set from a generalized-accuracy estimand over item and trial populations [26]. Psychometric aggregation can place models and items on a latent scale when the response design and anchors support it; CAISI, for example, has combined item-response modelling with fixed scaffolding, weighted-token budgets and cost measurement 2

systems lack a compute estimate, while all 35 open-weight systems in the sample have one. METR p50 and p80 measurements exist for 26 systems, all in the closed block. Consequently, only seven systems jointly observe training compute and METR p50. No open-weight system in the sample supplies the intended compute–horizon join.Compute only 36 Horizon only 19 Both 7 Neither 0 Compute Horizon (a) The intended join exists for only 7 of 62 systems Training compute Horizon Other capability Peer-reviewed measurement Closed Open-weight 8/27 30% 26/27 96% 3/27 11% 1/27 4% 35/35 100% 0/35 0% 0/35 0% 0/35 0% (b) Missingness follows a closed/open-weight split 1021 1022 1023 1024 1025 1026 Estimated training compute (FLOP) 10−2 10−1 100 101 102 METR p50 task horizon (minutes) GPT-2 GPT-3 GPT-3.5 GPT-4 Claude 3.5 S. Claude 3.7 S. GPT-5 Descriptive only: the sample is not a census, compute is estimated, and no recent open-weight system carries a METR horizon. (c) The apparent relationship is identified by seven selected systems descriptive log-log slope 0.73 Figure 2: Observability of the selected 62-system record. Panel (a) shows the intended compute–horizon join as four mutually exclusive observation patterns: compute only, horizon only, both and neither. Panel (b) shows that missingness follows the closed/open-weight split and extends to non-METR and peer-reviewed measurements. Panel (c) displays the seven jointly observed systems; the fitted line is descriptive and is not interpreted causally or as a stable scaling law. This pattern is more consequential than a low marginal coverage rate. A compute-indexed capability model requires the intersection, not the union, of the two records. The seven systems are also historically clustered and not a designed sample across developers, architectures or access regimes. A descriptive log–log slope can be fitted, but its apparent smoothness does not identify whether compute, algorithms, data, inference effort, model family or evaluation protocol caused the change. The missingness is unlikely to be ignorable. Public compute disclosure is related to access regime, developer practice and time. Treating absent compute as random would therefore turn an institutional disclosure pattern into a statistical assumption. Controlled training suites such as Pythia demonstrate the value of dense checkpoints and fixed data order for identifying training dynamics [5]; frontier-release records do not offer the same design. 5 Result 2: Changing Instruments, Concentrated Provenance Benchmark results are versioned events, not timeless labels. The audit records methodology changes, corrections, contamination findings, reward-hacking findings, measurement ceilings and replacements across METR, ARC-AGI, SWE-bench, MMLU and other instruments (Figure 3). Dynamic and contamination-resistant benchmarks are promising responses [57, 62], but refreshes and repairs still need explicit linking if the objective is longitudinal inference. Recent work on 5

saturation and verified benchmark revision likewise shows that item lifecycle is part of the measurement process rather than a footnote [1, 44, 61].2019 2020 2021 2022 2023 2024 2025 2026 Calendar year LifeSciBench 1 CHC AGI Score 1 Humanity's Last Exam 1 MMLU 1 SWE-bench Pro SWE-bench Verified ARC-AGI 3 ARC-AGI 2 ARC-AGI 1 METR Time Horizon 80pct 1.1 METR Time Horizon 1.1 METR Time Horizon 1.0 cutoff (a) A benchmark is a versioned instrument with a lifecycle MC ! L 52/71 (73.2%) from METR (b) Measurement provenance is concentrated METR TH1.1: 52 Other lab releases: 2 Preprints: 10 Peer-reviewed: 6 Vendor: 1 M/C/!/L mark lifecycle events. Each square is one of 71 quantitative events; administrative release records are excluded. Figure 3: Benchmark lifecycle and source dependence. Panel (a) places benchmark versions and recorded methodology, correction, contamination and measurement-ceiling events on a common timeline. Panel (b) combines a Pareto view of the 71 substantive quantitative events, their cumulative source share and their venue-class composition. METR supplies 52 events (73.2%); the source Herfindahl index is 0.542, or 1.85 effective equally represented sources. Repeated rows from one programme are not independent replication. The provenance distribution is similarly uneven. The 71 substantive events consist of 62 benchmark results, four field experiments, four substrate estimates and one efficiency observa- tion. Fifty-two events (73.2%) come from the METR Time Horizon 1.1 programme. By venue class, 54 events (76.1%) are laboratory releases, ten (14.1%) preprints, six (8.5%) peer-reviewed publications and one (1.4%) a vendor source. The corresponding source Herfindahl index is 0.542, equivalent to only 1.85 equally represented sources, even though 16 source identifiers appear at least once. Protocol history can matter even without an explicit benchmark revision. Repeated behav- ioral measurements have been shown to change with item order, reasoning mode, persona and conversation history [53]; a fixed candidate-response set can also receive different scores when the LLM judge is replaced [60]. These are not side effects to be averaged away after the fact. They are versioned measurement events that belong in the evidence record. These percentages describe this audit, not the full evaluation literature. They neverthe- less bound what can be claimed from this record. Fifty-two measurements generated by one task collection and adjudication process are valuable observations, but they do not provide 52 independent replications. Shared tasks, protocols and model-fitting choices can induce corre- lated error. The concentration is not a criticism of METR; its documentation of limitations, task changes and failure modes makes the analysis possible [35, 36]. It is evidence that the ecosystem lacks overlapping longitudinal programmes with which to estimate programme-specific bias. 6 Result 3: Benchmark Links Depend on Scale and Design Figure 4 separates a within-family revision from a cross-benchmark stress test. For seven systems measured on METR Time Horizon 1.0 and 1.1, the fitted log2 slope is 1.206 (95% CI 1.021–1.390). Leave-one-out slopes range from 1.126 to 1.275. In this bridge, the new version 6

is therefore not represented well by a pure additive shift on the log scale.101 102 TH1.0 horizon (min) 101 102 TH1.1 horizon (min) log2 slope 1.206 95% CI [1.021, 1.390] (a) Within-family revision 60 65 70 75 80 85 90 MMLU accuracy (%) 30 40 50 60 70 80 90 MMLU-Pro accuracy (%) logit slope 0.976 95% CI [0.793, 1.159] (b) Cross-benchmark stress test 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Linking slope (95% CI) METR log2 MMLU logit MMLU probit MMLU linear MMLU log2 1.21 0.98 1.04 1.31 1.98 (c) The diagnosis depends on scale 10 20 30 40 50 60 Common systems (illustrative) 0.0 0.2 0.4 0.6 0.8 1.0 Power for |slope - 1| = 0.10 n=31 n=23 (d) Anchor count and placement determine detectability METR-like bridge MMLU-like bridge Power curves hold residual noise and transformed-score dispersion at their observed values; they are design illustrations, not universal sample-size rules. Figure 4: Linking diagnostics. Panel (a) links METR Time Horizon 1.0 to 1.1 on a logarithmic scale. Panel (b) shows the six common MMLU/MMLU-Pro systems with paired uncertainty intervals in raw percentage space; MMLU-Pro changes content, choice structure and reasoning demands, so this is a cross-benchmark stress test rather than a claim of construct invariance. Panel (c) shows how the fitted slope changes with score representation; the vertical line marks a shift-like slope of one. Panel (d) shows that detectability depends jointly on anchor count and the observed score dispersion. For six common systems on MMLU and MMLU-Pro, the logit slope is 0.976 (95% CI 0.793– 1.159), the probit slope is 1.043, the linear slope is 1.312 and the logarithmic slope is 1.984. Logit and probit are natural bounded-response links, but boundedness alone does not make either uniquely correct. More importantly, MMLU-Pro is not simply a new form of the same test: it changes item selection, answer options and reasoning demands [55]. The comparison shows why an analyst must state the measurement model; it does not establish strict invariance. Table 1: Power of the observed linking designs. The final column is an illustrative projection assuming the same residual noise and transformed-score dispersion. Deliberately spanning the old score range can materially change the required sample. Bridge and scale n 95% CI 80% MDE Power observed n for 10% METR TH1.0 → TH1.1, log2 7 [1.021, 1.390] 0.252 63% 31 MMLU → MMLU-Pro, logit 6 [0.793, 1.159] 0.248 6% 23 The designs have roughly 80% power only for slope departures near 25% (Table 1). The METR bridge has about 63% power at its observed departure; the MMLU logit bridge has about 6%. Under the strong same-noise and same-spread assumption, detecting a 10% departure would require approximately 23–31 common systems. This is why benchmark revisions need designed anchor panels: common systems or common items chosen across the previous score range and model families, with uncertainty on both axes. Psychometric linking, benchmark- agreement analysis and fixed-parameter calibration provide more appropriate foundations than concatenating versioned leaderboards [21, 24, 25, 27, 43]. 7

7 Beyond One Ruler: A Measurement Portfolio The source audit was expanded to ask what else could be measured. The answer is not another benchmark that replaces all others. Frontier-AI progress has several scientifically distinct layers, and the most defensible record is a portfolio in which each layer has a stated role. Resources and mechanisms. Training FLOP, parameters, tokens and data describe inputs, not capability by themselves. Algorithmic efficiency asks how much resource is needed to reach a fixed performance level [23]. Inference-compute response curves then measure how achieved per- formance changes with tokens, attempts, feedback, parallelism or tool use; fixed-budget scores can understate or reorder systems [34, 51]. Price-performance and energy provide deployment- relevant resource frontiers rather than assuming the cost of eliciting a score is constant [20, 39]. A recent CAISI evaluation demonstrates the combined design: latent-capability aggregation, controlled agent scaffolds and token budgets, held-out tasks and end-to-end cost are reported together rather than as separate leaderboards [10]. Behavior as a distribution, not a point. Psychometric capability profiles and efficient item selection can estimate multiple abilities while retaining item difficulty and uncertainty [26, 30, 32]. Dynamic streams address freshness and contamination, while anchor-based sys- tems preserve longitudinal comparability as instruments grow [11, 21, 24]. Reliability curves replace a single p50 horizon with success probability across task duration, repeated attempts and time budgets. Agentic evaluations such as RE-Bench and PaperBench add realistic long-form work, human baselines and decomposed rubrics [52, 58]. Capability alone is still incomplete: propensity profiles ask whether systems tend toward or away from behaviors that matter for performance and safety [49]. Human-preference systems such as Chatbot Arena measure a dif- ferent endpoint again: perceived comparative usefulness under a changing prompt population [12]. Human ratings are themselves an instrument: multifaceted item-response models can sep- arate rater severity and centrality from output quality, while safety-profile IRT can preserve multidimensional tendencies rather than collapsing them into one score [9, 48]. Validate the construct and the human reference. Two further directions become im- portant whenever a forecast uses phrases such as “reasoning”, “autonomy”, “human-level” or “superhuman”. Measurement-theory work distinguishes the construct from the particular score used to operationalize it, and empirical benchmark audits show that item defects or nuisance phrasing can move results without the intended capability changing [2, 41, 59]. The appropri- ate response is triangulation: pre-specify the construct, seek convergent evidence from more than one instrument, test discriminant and criterion validity, and inspect item- or process-level failure modes. Human thresholds require the same discipline. Human baselines should define the participant population and expertise, use matched tools and time budgets, retain repeated observations and uncertainty, and document exclusions rather than treating a single historical score as a fixed species-level constant [56, 63]. Risk and deployment. Evaluations of dangerous capability and robust refusal are policy- relevant even when they do not track general benchmark averages [33, 50]. Applied-work bench- marks such as LifeSciBench can better represent realistic domain workflows, but still require controls for protocol, judge and provenance [42]. Field experiments measure productivity, qual- ity, heterogeneous effects and task boundaries that laboratory scores cannot supply [4, 7, 13, 14]. Post-deployment monitoring is therefore a measurement layer, not merely operational housekeep- ing [3, 47]. Figure 5 shows why these directions should not be averaged into one “AGI score.” Public availability, joinability, protocol control and external validity are different properties. Human 8

8 Implications for Dated Forecasts The full argument is a chain (Figure 6). Resources are converted by a deployed system config- uration into behavior; behavior interacts with people and institutions to produce outcomes; a forecast maps a versioned endpoint to a date. Four failure modes occur at the links: structured missingness, protocol dependence, benchmark drift and source dependence.A forecast is the end of a measurement chain, not the beginning Every dated claim inherits choices about the target, ruler, version link, data join, source dependence and validation rule. 1 Define the estimand What population, task and quantity? 2 Choose the ruler Score, horizon, cost, profile or outcome? 3 Link versions What stays invariant when instruments change? 4 Join the records Which systems carry both variables? 5 Test independence How many programmes actually replicated? 6 Forecast and backtest Was the dated target frozen and scored? AUDIT FAILURE DESIGN RESPONSE Vague or shifting target Operational criterion and variance model One-point score as intrinsic ability Response curve or multidimensional profile Successive scores concatenated Designed anchors and link uncertainty Sparse or MNAR overlap Sentinel panel and explicit missingness Repeated rows treated as replication Multi-lab reruns and assertion provenance Date published; measurement untested Archived target, monitoring and scoring Measurement portfolio: choose the ruler that matches the research question Linked latent capability Task duration x reliability Resource, cost and energy frontier Construct-valid capability profile Matched human reference Field and deployment outcomes Cross-cutting controls fresh item lifecycles | construct validation | matched human baselines | judge calibration | provenance | uncertainty and missingness The research question should choose the ruler. The ruler should not choose the conclusion. Figure 6: Synthesis of the paper. A dated frontier-AI forecast links resources, the system as used, measured behavior and deployment outcomes. Missingness, protocol dependence, benchmark drift and source dependence can break that chain. The corresponding remedies are disclosure with uncertainty bounds, response curves at matched budgets, versioned anchors and independent replication with back- testing. A defensible forecast packet should therefore report, at minimum: 1. the operational target, population and estimand; 2. the benchmark, version, item lifecycle and linking function; 3. the exact model, scaffold, tools, judge and inference-resource budget; 4. the joint observations supporting the progression relationship and the missingness model; 5. provenance, dependence among sources and independent replications; and 6. a frozen backtest protocol using proper scoring rules or interval coverage after measurement changes. This standard is compatible with provocative forecasting. It does not require waiting for perfect measurement. It requires that the forecast expose where extrapolation begins. A release- date model may be useful even when compute is missing; an agent benchmark may be informa- tive even when it changes; an expert survey may be decision-relevant even when criteria differ. The error is to treat those choices as invisible and then interpret a narrow confidence band as if it covered instrument, protocol and provenance uncertainty. 10

9 Limitations The 62 systems form a curated analytic sample, not a population estimate of all frontier re- leases through 12 August 2026. Coverage percentages must therefore be read as properties of this record. The package explicitly lists known pre-cutoff omissions, but that list is itself not guaranteed exhaustive. The benchmark bridges are small and observational. OLS treats the old transformed score as measured without error; a Bayesian or Deming-style errors-in-variables model would be preferable if comparable uncertainty were available on both versions. The MMLU/MMLU-Pro case changes construct-relevant content and protocol, so it is a stress test of linking assumptions rather than a psychometric equating study. The portfolio review is structured and source-audited but is not a registered systematic review or meta-analysis. Its zero-to-three ratings are reasoned design assessments documented in CSV form. They are intended to make tradeoffs inspectable, not to rank research programmes. Only public evidence is represented. Private evaluations, internal training records, undis- closed inference budgets and proprietary deployment outcomes may be much denser. Their existence would not repair the public record used for independent forecasting, but it limits claims about what developers themselves can infer. Finally, event semantics improve traceability, not causal identification. A graph can reveal that two variables do not join or that one source dominates; it cannot supply missing counter- factuals, invariant constructs or independent replications. 10 Conclusion Frontier AI is being forecast without a common ruler. In the audited public record, the resource and capability tables meet on seven systems; missing compute is structured by access regime and time; benchmark versions can change scale; the diagnosis depends on the chosen link; and most substantive measurements come from one programme. Those are not reasons to abandon forecasting. They are reasons to move the measurement system into the forecast rather than leaving it in the background. The expanded research review also changes the remedy. The field does not need one more scalar presented as the answer. It needs a versioned measurement portfolio: resources and algo- rithmic efficiency; inference-budget response curves; reliability and agentic work; dynamic and psychometric instruments; construct validity and matched human references; safety, preference and field outcomes; and forecast backtesting. Each layer answers a different question and carries a different uncertainty structure. A dated forecast should therefore be read as a composite scientific claim: that the target is operational, the observations are joined, the ruler has been linked across revisions, the protocol is controlled, and the evidence is sufficiently independent. Until those claims are made explicit, the most precise part of many frontier-AI forecasts may be the date—and the least precise part may be what, exactly, is being measured on the way there. Data and Code Availability The reproducibility package contains 26 CSV tables and an RDF graph in Turtle, together with a 27-source verification log and cutoff audit. It also includes the 56-source literature table, the normalized direction–source map, the 16-direction measurement portfolio, nine cross-cutting design requirements, analysis and figure scripts, and one reproduction entry point. A clean run rebuilds the normalized data, graph, linking and power analyses, all six figures, bibliography and manuscript PDF without author-specific paths. The arXiv bundle includes the compiled bibliography and the ancillary audit under anc/. 11