A single 50% task-completion horizon inadequately summarizes AI-agent capability for forecasting.
Structured and assessed
First pass. Added for/against arguments, linked unreliability-of-long-p50 subclaim as support, related links to doubling-trend and run-variance claims, recorded METR paper as a mention. Assessed supported (0.72), credence omitted as evaluative. Importance 0.4, contestation 0.35.
Assessed Supported
verdict confidence 0.72
The 50% task-completion time horizon, popularized by METR, is the human task length at which an agent is predicted to succeed half the time on a particular suite of mostly software tasks. The case that this single figure is an inadequate summary for forecasting is well supported, and is acknowledged in part by METR itself, which reports 80% horizons alongside it and notes that they are roughly four to six times shorter. Forecasts of when agents can do real work need success rates at the reliability that work demands, the effect of scaffolding, inference budgets and repeated attempts, and evidence that the suite generalizes to other domains; none of these can be read off one p50 number, and the p50 estimate itself becomes poorly constrained at long durations. Independent methodological audits, such as a 2026 preprint on frontier-AI forecasting measurement, reach the same conclusion and propose reliability curves across duration, attempts and budgets in its place. The strongest counterpoint is that METR found the 80% horizon doubling at essentially the same rate as the 50% horizon. If the reliability curve shifts in parallel, the p50 trend is an adequate proxy for the rate of progress, though not for the level of capability at deployment-relevant reliability. The claim therefore holds most clearly for forecasts of absolute capability or deployment readiness, and more weakly for forecasts only of growth rates. Whether the parallel-shift pattern persists as models and task suites change is the main open question.
Claim entered the graph