Step 6: tool results

provenance_read_source

{
  "max_chars": 8000,
  "source_id": "0a99fd17-2b29-45c8-9ab6-6afe1fdbc286"
}

{"success":true,"source":{"id":"0a99fd17-2b29-45c8-9ab6-6afe1fdbc286","url":"https://arxiv.org/pdf/2608.14903","title":"Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence","source_type":"preprint"},"origin":"stored","total_chars":53644,"offset":0,"content":"Frontier AI Forecasting Has\na Measurement Problem:\nAn Audit of Progress Evidence\nFabricio F. Costa, PhD, MBA, PMP1,2,3,4∗\n1AIx4All, LLC, Sunnyvale, CA, USA\n2HCLTech, Santa Clara, CA, USA\n3Genomic Sciences and Biotechnology Program, UCB, Bras´ılia, Brazil\n4Cancer Biology and Epigenomics Program, Stanley Manne Children’s Research Institute,\nAnn & Robert H. Lurie Children’s Hospital of Chicago, Northwestern University Feinberg\nSchool of Medicine, Chicago, IL, USA\nEvidence cutoff: 12 August 2026. Preprint, comments welcome.\nAbstract\nQuantitative forecasts of frontier artificial intelligence often connect dated targets to trends\nin benchmark scores, training compute, release time or expert belief. This paper audits\nwhether the public measurement record supports those connections before another trend is\nfitted. I construct a frozen, event-centric record through 12 August 2026 with 62 selected\nsystems, 12 versioned benchmarks, seven capability or impact criteria, 144 graded events,\n27 source records and 408 typed relations. The record is an audit sample, not a census.\nOnly seven systems jointly observe estimated training compute and a METR 50% task\nhorizon. Training compute is absent for 19 of 27 closed systems, including every selected\nclosed release from 2026, while none of the 35 open-weight systems has a METR horizon\nobservation. Benchmark succession creates a second break: a seven-system link from METR\nTime Horizon 1.0 to 1.1 has a log-scale slope of 1.206 (95% CI 1.021–1.390), whereas a six-\nsystem MMLU–MMLU-Pro comparison appears approximately shift-like under logit and\nprobit links but not under linear or logarithmic links. The observed bridges have about\n80% power only for slope departures near 25%. Provenance is concentrated: 52 of 71 sub-\nstantive quantitative events (73.2%) come from one measurement programme, and 76.1%\nare laboratory releases. A review of 56 additional methodological and empirical sources\nidentifies 16 complementary measurement directions spanning resources, inference budgets,\nreliability, agentic work, safety, human preference, field outcomes and forecast backtesting.\nNo direction supplies a replacement scalar. The result is not that frontier-AI forecasting\nis impossible, but that a defensible dated forecast is a claim about a versioned measure-\nment system—its joins, protocols, links and source dependence—not merely a fitted curve\nor calendar date.\nKeywords: AI evaluation; measurement; benchmark linking; forecast validation; training\ncompute; provenance; metascience\n1 Introduction\nForecasts of advanced AI now appear as expert probability distributions, benchmark extrapola-\ntions, compute trends and backtested models of agent performance [18, 19, 45]. The quantitative\nform is attractive: select an indicator, estimate its rate of change and solve for a date at which\n∗Correspondence: fcosta@aix4all.com. This independent scholarly work uses only public sources, received\nno external funding, and represents the author’s views rather than those of the listed affiliations. The author\ndeclares no financial interest in organisations whose systems or forecasts are discussed.\n1arXiv:2608.14903v1 [cs.AI] 14 Aug 2026\n\na threshold is crossed. Yet the date is meaningful only if the target is operational, the obser-\nvations are comparable, the progression variable is jointly observed and the uncertainty model\nincludes changes in the measuring instrument.\nThis paper audits those preconditions. It began as an attempt to construct an event-based\nforecast of frontier-AI capability milestones. The supporting infrastructure could be built; the\nintended progression analysis could not be identified without strong additional assumptions.\nThe failure was not a software failure. It was a measurement finding: capability and resource\nrecords were sparsely joined, benchmark versions did not always preserve scale, evaluation\nprotocols changed, and repeated observations were heavily dependent on a small number of\nsources.\nThe distinction matters. A criterion can be well defined while its public evidence is not\nforecastable. Conversely, a record can contain many scores yet fail to measure one stable\nquantity. Existing work has proposed taxonomies of performance, generality and autonomy [40],\nholistic evaluation suites [31], ability-oriented measurement [22], explicit evaluation estimands\n[6] and statistical models that distinguish fixed-benchmark accuracy from generalized accuracy\n[26]. The contribution here is to connect those ideas to the actual longitudinal record used in\nfrontier-AI forecasting.\nThe paper makes three contributions. First, it provides a reproducible event-centric audit\nin which releases, results, revisions, corrections, contamination findings, field experiments and\nsource assertions are separately timestamped and versioned. Second, it quantifies three fore-\ncastability bottlenecks in that record: sparse joint observability, benchmark-linking uncertainty\nand provenance concentration. Third, it translates a deep source review into a measurement\nportfolio and a set of study-design requirements rather than proposing another universal leader-\nboard score.\nThe semantic graph is therefore infrastructure, not the main scientific claim. Its role is to\nmake questions such as “which systems connect compute to capability?”, “which result changed\nafter a benchmark revision?” and “which conclusions depend on one programme?” answerable\nfrom the same evidence ledger.\n2 From a Score to a Forecastable Quantity\nA benchmark result is not an intrinsic property of a model. It is an observation generated by a\nmodel, an instrument and a protocol at a time. A compact representation is\nYm,b,v,p,r,j,t = hb,v(θm,t; p, r, j) + εm,b,v,p,r,j,t, (1)\nwhere m is the model, b the benchmark family, v its version, p the prompting or agent protocol,\nr the inference-resource budget, j the judge or scoring procedure and t the observation date.\nThe latent quantity θ may itself be multidimensional. Equation 1 is not fitted as a complete\nlatent-variable model here; it is an accounting identity that makes the assumptions behind a\ntrend explicit.\nA dated forecast adds at least three more objects: a target estimand, a progression axis and\na threshold rule. If training compute is the progression axis, compute and the outcome must\nbe co-observed or missingness must be modelled. If benchmark version v is replaced by v + 1,\na linking function must place the two instruments on a defensible common scale. If inference\nbudget or scaffolding changes, the score may move even when the underlying system does not.\nIf many observations share a source, their errors need not be independent.\nThis accounting view is increasingly reflected in measurement science. Statistical evaluation\ncan distinguish performance on one fixed item set from a generalized-accuracy estimand over\nitem and trial populations [26]. Psychometric aggregation can place models and items on a\nlatent scale when the response design and anchors support it; CAISI, for example, has combined\nitem-response modelling with fixed scaffolding, weighted-token budgets and cost measurement\n2\n\n[10]. Adaptive testing and fixed-parameter calibration can reduce evaluation cost [21, 30], but\nsimulation evidence also shows that small, clustered or non-normal model samples can make\nitem and ranking inferences unstable [25]. The method is therefore not “use IRT” or “use a\nbetter benchmark.” It is to state the estimand, design the linking observations and preserve\nthe protocol that makes the scale interpretable.\nThe audit therefore asks five questions:\n1. Is the target construct and intended population explicit?\n2. Are the proposed predictor and outcome jointly observed on enough systems?\n3. Are benchmark versions linked rather than concatenated?\n4. Are protocol and resource choices part of the measurement record?\n5. Are provenance, replication and source dependence represented in uncertainty?\nThese are deliberately stricter than asking whether a regre","truncated":true,"next_offset":8000,"facts":{"authors":["Costa, Fabricio F"],"publisher":"arXiv.org","published_date":"2026-08-14","doi":null,"archived_url":"https://web.archive.org/web/20261001072508/https://arxiv.org/pdf/2608.14903","events":[],"versions":[],"copy_of":[]},"note":"More text follows; call again with offset 8000 to continue."}