Growth in AI task-completion time horizons on benchmarks tracks progress toward AI that can automate AI research or most cognitive work.
Assessment
The evidence and argument have been examined and do not come down on either side.
The claim is that the steady growth in the length of tasks AI agents can complete on benchmarks (the "time horizon" metric introduced by METR, with a reported doubling time of roughly seven months over 2019 to 2025, and faster since 2024) is a meaningful indicator of progress toward AI that can automate AI research or most cognitive work.
Two readings separate the question. On a weak reading, horizon growth is positively correlated with progress toward automation. That is probably true, since long-horizon autonomy is a capability that automation needs, and horizons appear to grow at similar rates across several domains beyond software, which suggests the metric is not an artifact of one benchmark. On a strong reading, horizon growth tracks progress closely enough to anchor forecasts of when automation will arrive. That depends on whether long-task autonomy is the main bottleneck, as opposed to reliability, research judgment, compute, or gathering context. It also depends on whether benchmark trends transfer to real work.
The evidence on transfer is mixed. Whether benchmark horizon trends generalize to real-world software work remains unsettled. Maintainer review of agent-written patches finds merge rates well below automated pass rates, with suggestive evidence of a slower trend. METR's 2025 randomized trial found experienced developers slower with early-2025 tools, though METR now regards that result as outdated. The 50%-success threshold behind the headline metric is also far below what deployment requires. A repeated, like-for-like measurement of horizons on real work across several model generations, compared against the benchmark trend, would largely resolve the question.
Full reasoning: the evidence and decisions behind this verdict
This assessment draws on background knowledge of METR's time-horizon work (Kwa et al. 2025), METR's July 2025 developer RCT, and the graph's existing assessment of the claim that benchmark software horizon trends generalize to real-world software work. That claim is rated inconclusive and cites METR's 2026 maintainer-review study, which found merge rates about 24 points below grader pass rates and a possibly slower trend, along with cross-domain evidence of similar doubling times. The weak reading of the claim is likely true. The strong reading, that horizon growth is a forecasting anchor, is doubtful because of the transfer gap, the reliability threshold, and rival bottlenecks. Since the claim's force turns on which reading is meant and on an unsettled transfer question, the verdict is inconclusive with credence 0.45. Three developments would change it. Sustained real-world measurement could show that transfer holds, which would move the verdict toward supported. Persistent divergence between benchmark and real-work trends would move it toward doubtful. Evidence on whether long-task autonomy is the binding bottleneck would also bear on it. No new web search was done in this pass.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Time horizon measures how long an AI system can work autonomously and still succeed. Because sustained autonomy on long tasks is the main capability bottleneck to automating AI research, and because horizons grow at similar exponential rates across many domains beyond software, growth in benchmark horizons tracks progress toward AI that can automate research and most cognitive work.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 2 not separately assessed
Granting that long-task autonomy is the main bottleneck and that horizons grow similarly across domains, horizon growth would be a good tracker of progress toward automation. It would not be a guarantee, because benchmark tasks could still differ from real work in ways both premises leave open. The argument rests mostly on The ability to sustain autonomous work on long tasks is the main capability bottleneck preventing AI from automating AI research., a strong bottleneck claim that is not yet assessed and has rivals such as reliability and research judgment. AI time horizons grow roughly exponentially across many task domains beyond software engineering, with similar doubling times. is also not yet assessed.
Benchmark tasks are self-contained, well specified and scored by algorithm, and benchmarks systematically overestimate real-world capability. On this argument, the claim that time-horizon trends on benchmark software tasks generalize to real-world software work is false. If the trend does not transfer even to real software work, it cannot track progress toward automating research or most cognitive work.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 inconclusive1 supported
Granting that benchmark horizon trends fail to generalize even to real software work, the tracking claim would very likely fail, since AI research is largely software work. The argument depends on the denial of Time-horizon trends measured on benchmark software tasks generalize to real-world software work., which is currently assessed as inconclusive. It is supported by evidence that benchmarks overestimate real-world capability, though overestimating absolute capability is weaker evidence than a slower trend. As things stand, the argument raises a serious doubt but does not establish its conclusion.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 2, 2026. Every judgment on this page is accompanied by a reasoning trace.