Step 8: tool results

provenance_read_source

{
  "offset": 30000,
  "max_chars": 12000,
  "source_id": "0a99fd17-2b29-45c8-9ab6-6afe1fdbc286"
}

{"success":true,"source":{"id":"0a99fd17-2b29-45c8-9ab6-6afe1fdbc286","url":"https://arxiv.org/pdf/2608.14903","title":"Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence","source_type":"preprint"},"origin":"stored","total_chars":53644,"offset":30000,"content":"recasts\nThe full argument is a chain (Figure 6). Resources are converted by a deployed system config-\nuration into behavior; behavior interacts with people and institutions to produce outcomes; a\nforecast maps a versioned endpoint to a date. Four failure modes occur at the links: structured\nmissingness, protocol dependence, benchmark drift and source dependence.A forecast is the end of a measurement chain, not the beginning\nEvery dated claim inherits choices about the target, ruler, version link, data join, source dependence and validation rule.\n1\nDefine the\nestimand\nWhat population, task\nand quantity?\n2\nChoose the\nruler\nScore, horizon, cost,\nprofile or outcome?\n3\nLink\nversions\nWhat stays invariant\nwhen instruments change?\n4\nJoin the\nrecords\nWhich systems carry\nboth variables?\n5\nTest\nindependence\nHow many programmes\nactually replicated?\n6\nForecast and\nbacktest\nWas the dated target\nfrozen and scored?\nAUDIT\nFAILURE\nDESIGN\nRESPONSE\nVague or shifting\ntarget\nOperational criterion\nand variance model\nOne-point score\nas intrinsic ability\nResponse curve or\nmultidimensional profile\nSuccessive scores\nconcatenated\nDesigned anchors and\nlink uncertainty\nSparse or MNAR\noverlap\nSentinel panel and\nexplicit missingness\nRepeated rows treated\nas replication\nMulti-lab reruns and\nassertion provenance\nDate published;\nmeasurement untested\nArchived target,\nmonitoring and scoring\nMeasurement portfolio: choose the ruler that matches the research question\nLinked latent\ncapability\nTask duration x\nreliability\nResource, cost and\nenergy frontier\nConstruct-valid\ncapability profile\nMatched human\nreference\nField and deployment\noutcomes\nCross-cutting controls fresh item lifecycles | construct validation | matched human baselines | judge calibration | provenance | uncertainty and missingness\nThe research question should choose the ruler. The ruler should not choose the conclusion.\nFigure 6: Synthesis of the paper. A dated frontier-AI forecast links resources, the system as used,\nmeasured behavior and deployment outcomes. Missingness, protocol dependence, benchmark drift and\nsource dependence can break that chain. The corresponding remedies are disclosure with uncertainty\nbounds, response curves at matched budgets, versioned anchors and independent replication with back-\ntesting.\nA defensible forecast packet should therefore report, at minimum:\n1. the operational target, population and estimand;\n2. the benchmark, version, item lifecycle and linking function;\n3. the exact model, scaffold, tools, judge and inference-resource budget;\n4. the joint observations supporting the progression relationship and the missingness model;\n5. provenance, dependence among sources and independent replications; and\n6. a frozen backtest protocol using proper scoring rules or interval coverage after measurement\nchanges.\nThis standard is compatible with provocative forecasting. It does not require waiting for\nperfect measurement. It requires that the forecast expose where extrapolation begins. A release-\ndate model may be useful even when compute is missing; an agent benchmark may be informa-\ntive even when it changes; an expert survey may be decision-relevant even when criteria differ.\nThe error is to treat those choices as invisible and then interpret a narrow confidence band as\nif it covered instrument, protocol and provenance uncertainty.\n10\n\n9 Limitations\nThe 62 systems form a curated analytic sample, not a population estimate of all frontier re-\nleases through 12 August 2026. Coverage percentages must therefore be read as properties of\nthis record. The package explicitly lists known pre-cutoff omissions, but that list is itself not\nguaranteed exhaustive.\nThe benchmark bridges are small and observational. OLS treats the old transformed score\nas measured without error; a Bayesian or Deming-style errors-in-variables model would be\npreferable if comparable uncertainty were available on both versions. The MMLU/MMLU-Pro\ncase changes construct-relevant content and protocol, so it is a stress test of linking assumptions\nrather than a psychometric equating study.\nThe portfolio review is structured and source-audited but is not a registered systematic\nreview or meta-analysis. Its zero-to-three ratings are reasoned design assessments documented\nin CSV form. They are intended to make tradeoffs inspectable, not to rank research programmes.\nOnly public evidence is represented. Private evaluations, internal training records, undis-\nclosed inference budgets and proprietary deployment outcomes may be much denser. Their\nexistence would not repair the public record used for independent forecasting, but it limits\nclaims about what developers themselves can infer.\nFinally, event semantics improve traceability, not causal identification. A graph can reveal\nthat two variables do not join or that one source dominates; it cannot supply missing counter-\nfactuals, invariant constructs or independent replications.\n10 Conclusion\nFrontier AI is being forecast without a common ruler. In the audited public record, the resource\nand capability tables meet on seven systems; missing compute is structured by access regime\nand time; benchmark versions can change scale; the diagnosis depends on the chosen link; and\nmost substantive measurements come from one programme. Those are not reasons to abandon\nforecasting. They are reasons to move the measurement system into the forecast rather than\nleaving it in the background.\nThe expanded research review also changes the remedy. The field does not need one more\nscalar presented as the answer. It needs a versioned measurement portfolio: resources and algo-\nrithmic efficiency; inference-budget response curves; reliability and agentic work; dynamic and\npsychometric instruments; construct validity and matched human references; safety, preference\nand field outcomes; and forecast backtesting. Each layer answers a different question and carries\na different uncertainty structure.\nA dated forecast should therefore be read as a composite scientific claim: that the target is\noperational, the observations are joined, the ruler has been linked across revisions, the protocol\nis controlled, and the evidence is sufficiently independent. Until those claims are made explicit,\nthe most precise part of many frontier-AI forecasts may be the date—and the least precise part\nmay be what, exactly, is being measured on the way there.\nData and Code Availability\nThe reproducibility package contains 26 CSV tables and an RDF graph in Turtle, together with\na 27-source verification log and cutoff audit. It also includes the 56-source literature table, the\nnormalized direction–source map, the 16-direction measurement portfolio, nine cross-cutting\ndesign requirements, analysis and figure scripts, and one reproduction entry point. A clean run\nrebuilds the normalized data, graph, linking and power analyses, all six figures, bibliography\nand manuscript PDF without author-specific paths. The arXiv bundle includes the compiled\nbibliography and the ancillary audit under anc/.\n11\n\nReferences\n[1] Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammana-\nmanchi, Ruchit Rawal, Vil’em Zouhar, et al. When AI benchmarks plateau: A sys-\ntematic study of benchmark saturation. arXiv preprint arXiv:2602.16763, 2026. doi:\n10.48550/arXiv.2602.16763.\n[2] Ahmed Alaa, Thomas Hartvigsen, Niloufar Golchini, Shiladitya Dutta, Frances Dean, In-\nioluwa Deborah Raji, and Travis Zack. Position: Medical large language model benchmarks\nshould prioritize construct validity. In Proceedings of the 42nd International Conference\non Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages\n80991–81004, 2025. URL https://proceedings.mlr.press/v267/alaa25a.html.\n[3] Razvan Amironesei, Afzal Godil, Craig Greenberg, Kristen Greene, Johnston Patrick Hall,\nTheodore Jensen, Jonathan Fiscus, and Noah Schulman. Assessing risks and impacts of\nAI (ARIA): Pilot evaluation report. Technical Report NIST AI 700-2, National Institute\nof Standards and Technology, 2025.\n[4] Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein. Measuring the impact\nof early-2025 AI on experienced open-source developer productivity. arXiv preprint\narXiv:2507.09089, 2025.\n[5] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle\nO’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth,\nEdward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite\nfor analyzing large language models across training and scaling. In Proceedings of the 40th\nInternational Conference on Machine Learning, volume 202 of Proceedings of Machine\nLearning Research, pages 2397–2430, 2023.\n[6] Olivier Binette and Jerome P. Reiter. Improving the validity and practical usefulness of\nAI/ML evaluations using an estimands framework. arXiv preprint arXiv:2406.10366, 2024.\n[7] Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond. Generative AI at work. The\nQuarterly Journal of Economics, 140(2):889–942, 2025. doi: 10.1093/qje/qjae044.\n[8] Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan\nJurafsky. With little power comes great responsibility. In Proceedings of the 2020 Confer-\nence on Empirical Methods in Natural Language Processing, pages 9263–9274, 2020. doi:\n10.18653/v1/2020.emnlp-main.745.\n[9] Jodi M. Casabianca and Maggie Beiting-Parrish. Correcting human labels for rater effects\nin AI evaluation: An item response theory approach. arXiv preprint arXiv:2602.22585,\n2026. doi: 10.48550/arXiv.2602.22585.\n[10] Center for AI Standards and Innovation. CAISI evaluation of DeepSeek V4 Pro. Na-\ntional Institute of Standards and Technology, May 2026. URL https://www.nist.gov/\nnews-events/news/2026/05/caisi-evaluation-deepseek-v4-pro. Released 1 May\n2026; updated 2 May 2026.\n[11] Simin Chen, Pranav Pusarla, and Baishakhi Ray. Dycodeeval: Dynamic benchmark-\ning of reasoning capabilities in code large language models under data contamination.\nIn Proceedings of the 42nd International Conference on Machine Learning, volume 267\nof Proceedings of Machine Learning Research, pages 8890–8909, 2025. URL https:\n//proceedings.mlr.press/v267/chen25ba.html.\n12\n\n[12] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li,\nDacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.\nChatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings\nof the 41st International Conference on Machine Learning, volume 235 of Proceedings of\nMachine Learning Research, pages 8359–8388, 2024.\n[13] Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz.\nThe effects of generative AI on high-skilled work: Evidence from three field experiments\nwith software developers. Management Science, 2026. doi: 10.1287/mnsc.2025.00535.\n[14] Fabrizio Dell’Acqua, Edward McFowland, Ethan Mollick, Hila Lifshitz, Katherine C. Kel-\nlogg, Saran Rajendran, Lisa Krayer, Fran¸cois Candelon, and Karim R. Lakhani. Navigating\nthe jagged technological frontier: Field experimental evidence of the effects of artificial intel-\nligence on knowledge worker productivity and quality. Organization Science, 37(2):403–423,\n2026. doi: 10.1287/orsc.2025.21838.\n[15] Epoch AI. Data on AI models. https://epoch.ai/data/ai-models-documentation,\n2026. Notable AI Models database; frozen for this audit on 12 August 2026.\n[16] Epoch AI, with major processing by Our World in Data. Computation used to\ntrain notable artificial intelligence systems. https://ourworldindata.org/grapher/\ncomputation-used-to-train-notable-artificial-intelligence-systems, 2026. Re-\ntrieved 12 August 2026.\n[17] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna\nWallach, Hal Daum´e III, and Kate Crawford. Datasheets for datasets. Communications of\nthe ACM, 64(12):86–92, 2021. doi: 10.1145/3458723.\n[18] Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans.","truncated":true,"next_offset":42000,"note":"More text follows; call again with offset 42000 to continue."}