The task-completion time horizon of frontier AI models began growing faster around 2024.
6 events · 2 assessments · 3 decisions
Reassessed no status change
This pass was prompted by the post-2024 doubling-time quantity's first estimate (103 days, 90% interval 82–132), which is consistent with the existing verdict. I corrected the Opus 4.6 horizon from 14.5 to 12 hours, following METR's March 2026 correction, in both the argument's written form and the assessment. The recomputed doubling time from those points is about 100 days, which is still consistent with the claim. Status stays supported at confidence 0.75, credence 0.78. I did not notify dependents because the change is not material.
Reassessed: still Supported
verdict confidence 0.75 · credence 0.78
Structured and assessed
First pass. Decomposed into: full-period doubling-time quantity (existing, linked), new post-2024 doubling-time quantity (recorded METR TH1.1 88.6-day figure), human-baseline reliability (existing, assumes), new artifact-explanation claim (contradicts, seed 0.25), new cross-benchmark replication claim (supports, seed 0.55). Two named arguments written and evaluated. Assessed supported (0.75 confidence, credence 0.78) from METR TH1.1 sub-period fits and 2025-26 out-of-sample models, weighed against Nguyen & Fithian 2026 reanalysis and suite-sensitivity caveats. Corrected extracted instance speaker/confidence; recorded LessWrong instance. Importance 0.65, contestation 0.6. No dependents-notification needed beyond the parent quantity, whose assessment already anticipates this verdict.
Assessed Supported
verdict confidence 0.75 · credence 0.78
The evidence for this claim comes almost entirely from METR's time-horizon measurements. METR measures the length of software and AI-research tasks, in human-expert time, that frontier models can complete with 50% success. Fit over all frontier models since 2019, the horizon has a doubling time of about seven months (196 to 212 days). Restricted to models released from 2024 onward, the same fit gives a doubling time of about three to four months: 109 days on METR's original task suite and 89 days on its revised January 2026 suite. Since 2023 the figure is 131 to 165 days. Models released after those fits stayed on the faster line. Claude Opus 4.5 measured about 5 hours in late 2025 and Claude Opus 4.6 about 14.5 hours in early 2026, against roughly one hour for Claude 3.7 Sonnet in early 2025. The case is not settled. The recent fit rests on about a dozen models whose individual estimates carry wide intervals. METR reports that the trend is sensitive to which tasks are in the suite. Most tasks of eight hours or more use estimated rather than measured human times. By early 2026 the suite was close to saturation. A 2026 statistical reanalysis by Nguyen and Fithian found that difficulty for AI is nearly flat in human time between about 2 and 30 minutes, which could make the 2024 passage through that band look faster than it was. These points could each explain part of the apparent speed-up as a measurement artifact. None of them accounts for continued fast growth in 2025 and 2026 across the one-to-fifteen-hour range, where the scale appears roughly linear. Independent confirmation on other benchmarks is thinner. On METR's metric, the growth rate did increase from about 2024, roughly halving the doubling time. Whether the timing is better placed in 2023 or in late 2024, when reasoning models trained with reinforcement learning arrived, cannot be resolved from so few points. How much of the speed-up reflects real capability gains, as opposed to properties of the task suite, remains the open question. A new suite with measured long-task baselines, together with replication on independent benchmarks, would settle it.
Updated claim instance
Instance 5143137b-49fb-4b77-b51a-cfbfb8ab3ba3 (https://www.alphaxiv.org/abs/2503.14499): was stance=affirms, confidence=0.8; set confidence=0.65, speaker="Thomas Kwa, Ben West et al. (METR)", sourceDate="2025-03". The paper's statement is hedged ("may have accelerated"), a tentative affirmation rather than a firm one; speaker set to the paper's authors rather than left blank.
Claim entered the graph