Surveyed AI researchers' aggregate forecasts are a reliable guide to when AI will reach human-level performance
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
Large surveys of published AI researchers, chiefly the AI Impacts Expert Survey on Progress in AI (2016, 2022, 2023), are the most-cited source of "expert" timelines for human-level AI. The evidence weighs against treating their aggregate forecasts as a reliable guide to timing, though it supports treating them as a rough indicator of what the field expects.
The strongest evidence against is that the answers are unstable. In the 2023 survey the aggregate 50% date was 2047 for "high-level machine intelligence" but 2116 for "full automation of labor". The two framings describe nearly the same capability, yet their dates differ by decades. Asking for probabilities by fixed years produces systematically later answers than asking for the year by which a probability is reached. The aggregate moved 13 years earlier between the 2022 and 2023 waves, after barely moving between 2016 and 2022, which tracks the salience of recent events. Individual forecasts vary widely. A measuring instrument whose output shifts this much with wording cannot be giving a stable estimate. The survey authors themselves say there are signs that respondents are not accurate forecasters across the range of questions asked. A second line of evidence is a record of underestimating recent capability progress, which is consistent with a bias toward lateness.
The best evidence in favour is a scoring of the 2016 survey's near-term milestones. It found reasonable calibration: a Brier score of 0.21, with about nine milestones expected and ten met after 6.5 years. That result is small, rests on subjective resolution judgments, and mixes faster-than-expected language and coding progress with slower progress in robotics. It also bears on multi-decade forecasts only if short-range accuracy predicts long-range accuracy, and almost no evidence on that exists. Researchers are well placed to judge technical feasibility, and alternative sources such as forecasting platforms and superforecaster panels have no long-horizon track record either. That supports the weaker claim that the surveys are informative about expert opinion, not the claim that they reliably guide timing. A scored track record would settle the question: a later survey wave, or a resolved long-horizon milestone set, could show whether aggregates stay stable and calibrated.
Full reasoning: the evidence and decisions behind this verdict
Prior model before searching: survey timelines are known to suffer framing effects and sharp revisions between waves, and expert long-range forecasting in general (Tetlock) degrades beyond a few years. I expected "doubtful", with the main counter-evidence being any scored near-term track record.
Evidence checked: - The 2023 ESPAI paper (Grace et al., jair.org/index.php/jair/article/download/19087/27230; arXiv 2401.02843) gives HLMI 50% at 2047 (2060 in 2022) and FAOL 50% at 2116 (2164 in 2022). It reports that between 2016 and 2022 the HLMI date moved only from 2061 to 2060. Its authors concede signs of inaccuracy, while arguing the respondents are among the best placed to make educated guesses. The extractor recorded that concession as a denial; I added the speaker and date. - AI Impacts' scoring of the 2016 narrow-task forecasts (aiimpacts.org/scoring-forecasts-from-the-2016-expert-survey-on-progress-in-ai/): Brier 0.21 versus 0.25 for uniform 50% guesses. That is better than the 1.2x and 1.5x shifted variants and better than shuffled assignment (0.31). The author notes caveats: subjective resolution, rounding, small n, and an unknown transfer to long horizons. This moved me slightly toward the view that the surveys carry signal on relative pace and near-term events. It did not move me on long-horizon HLMI timing. Since 2023, several milestones the scoring judged unmet have been met, such as competition-level mathematics. Re-scored today, the record would likely look more like underestimation in language and reasoning domains.
Weighing: the instability argument is the strongest consideration and rests on premises that are essentially settled facts about the published surveys. The framing gap is internal to a single survey, so no outside reference is needed. The track-record argument is weak for the claim as stated, because its bridge premise (that short-range skill transfers to multi-decade forecasts) is unsupported. "Reliable guide" is stronger than "informative", so the evidence counts against the claim as stated even though the surveys carry some signal. Status is doubtful rather than contradicted: no long-horizon forecast has resolved, and "reliable" is partly evaluative.
What would change it: future waves showing stable aggregates across framings; a re-scoring of 2016, 2022 and 2023 milestone forecasts showing good calibration; or evidence that short-horizon calibration transfers to long horizons.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because forecasts differ by decades between near-equivalent question framings, because fixed-years elicitation yields systematically later answers than fixed-probability elicitation, because the aggregate moved substantially earlier between consecutive survey waves, and because individual forecasts vary widely, the aggregate reflects elicitation choices and recent salience more than a stable, well-grounded estimate, and so is not a reliable guide to when human-level AI will arrive.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 4 not separately assessed
Granting its premises, an aggregate that shifts by decades with wording and by more than a decade in a single year is very unlikely to be a reliable timing guide, though instability does not rule out the median being roughly right by chance. The argument rests mainly on the framing gap and the shift between waves. Both are documented in the survey reports themselves and are not seriously disputed.
Because AI experts underestimated the pace of capability progress over the past decade, their aggregate forecasts carry a systematic bias toward lateness, and a biased instrument is not a reliable guide to the timing of human-level AI.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that experts underestimated recent progress, that shows a bias in the past decade. A bias need not persist, and a known bias could in principle be corrected for, so it bears only modestly on present reliability. The argument rests on the underestimation record, which is not yet assessed and conflicts in part with the reasonable-calibration finding for the 2016 survey.
Because the 2016 survey's near-term milestone forecasts turned out reasonably calibrated, and because accuracy on short-range forecasts predicts accuracy on long-range ones, the same population's aggregate forecasts of human-level AI can be trusted as a guide to its timing.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 2 not separately assessed
If near-term calibration held and short-range skill really transferred to multi-decade horizons, the conclusion would follow with reasonable strength. The argument depends almost entirely on the transfer premise, for which little evidence exists. The calibration finding is itself small-sample and subjective. As premises currently stand, the argument adds little.
Provenance
Where this claim has been said, linked to its canonical form.
There are signs in this research and past surveys that these experts are not accurate forecasters across the range of questions we ask.
Section discusses framing effects and dispersion as evidence against accuracy, while arguing AI researchers are among the best-positioned to make informative educated guesses.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 1, 2026. Every judgment on this page is accompanied by a reasoning trace.