Time-horizon trends measured on benchmark software tasks generalize to real-world software work.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass. Two arguments: against (benchmark-to-practice gaps; linked existing 'benchmarks overestimate' claim, minted maintainer-review-rate subclaim) and for (robustness across messiness and domains; linked existing messiness-plateau claim as contradicting premise, minted cross-domain exponential growth subclaim). Assessed inconclusive (0.6), credence 0.4: level does not transfer; rate unresolved. Importance 0.7, contestation 0.75. Recorded METR March 2026 note as a qualified denial. No dependents to notify (new claim). Budget-limited pass; marginal yield 0.45.
Assessed Inconclusive
verdict confidence 0.60 · credence 0.40
The claim asks whether the rapid growth in AI "time horizons" measured by METR on benchmark software tasks (a 50%-success horizon doubling roughly every four to seven months) carries over to real-world software engineering. Two readings need separating. The absolute horizon plainly does not transfer at face value: METR itself says a measured horizon corresponds to what a low-context person such as a new contractor could do, not a high-context professional, and real-world evaluations consistently find benchmark success overstating usefulness. Whether the growth rate transfers is the live question, and the evidence does not yet settle it. Against generalization: METR's 2026 study in which repository maintainers blindly reviewed SWE-bench Verified patches found merge rates about 24 percentage points below automated-grader pass rates, with suggestive but statistically shaky evidence that the maintainer-judged trend improves about 9.6 points per year more slowly. An earlier METR comparison of algorithmic and holistic grading found functionally correct agent code often unusable as is, and METR's 2025 randomized trial found experienced open-source developers 19% slower with early-2025 AI tools, though METR now regards that result as outdated and reports likely speedups with newer tools. These bear on whether benchmarks overestimate real-world capability and on whether maintainer-judged success improves as fast as benchmark success. For generalization: METR's original analysis found similar trend slopes on messier and cleaner task subsets, and a cross-domain analysis found exponential horizon growth across several other domains with broadly comparable doubling times, which suggests the rate is not an artifact of one benchmark. Gaps attributable to single-attempt elicitation without feedback may also shrink as agents iterate. What would resolve the question is a repeated, like-for-like measurement of horizons on real work (maintainer-accepted PRs or field productivity studies) across several model generations, compared against the benchmark trend.
Claim entered the graph