Growth in AI task-completion time horizons on benchmarks tracks progress toward AI that can automate AI research or most cognitive work.
5 events · 2 assessments · 2 decisions
Decomposed and reassessed
Two arguments were added. The argument for rests on autonomy as a general measure: a new subclaim that long-task autonomy is the main bottleneck (importance 0.45, seed 0.35), plus the existing cross-domain horizon claim 0e3e1dda. The argument against rests on the benchmark-to-real-work gap: existing claims d99ac54d (requires) and 2fba320c (contradicts). The assessment stays inconclusive with credence 0.45, and it now references the structure. Both arguments are evaluated as likely and not decisive. There are no dependents to notify beyond the seeding parent, and the verdict there is unchanged in substance.
Reassessed: still Inconclusive
verdict confidence 0.60 · credence 0.45
Assessed without decomposition
Budget constraints (first turn consumed about 26%) forced a light assessment from background knowledge: inconclusive, credence 0.45. No subclaims or arguments were created. A future pass should decompose into benchmark-to-real-work transfer, reliability threshold sensitivity, and domain generality. Importance is 0.6 and contestation 0.7.
Assessed Inconclusive
verdict confidence 0.60 · credence 0.45
The claim is that the steady growth in the length of tasks AI agents can complete on benchmarks (the "time horizon" metric popularised by METR, which reported a doubling time of roughly seven months between 2019 and 2025) is a meaningful indicator of progress toward AI that can automate AI research or most cognitive work. What favors it: time horizon is a general measure of agentic competence. It correlates with performance across many software and reasoning tasks, and long-horizon autonomy is a necessary capability for automating research. A sustained exponential trend in that capability is evidence of movement in the relevant direction. What weighs against it: benchmark tasks are cleanly specified, self-contained, automatically scored, and mostly software-related, while real research and work involve messy context, sparse feedback, coordination and high reliability requirements. The 50%-success threshold used for the headline metric is far below what deployment needs. Field evidence suggests the mapping is loose: METR's own 2025 randomized trial found experienced open-source developers about 19% slower with AI tools despite strong benchmark results. Whether horizon growth only indicates direction, or tracks progress closely enough to anchor timelines, is the real dispute. Evidence on whether benchmark horizon gains turn into real productivity or research automation over the next few years would settle it.
Claim entered the graph