A single 50% task-completion horizon inadequately summarizes AI-agent capability for forecasting.
Since this assessment: 1 source added.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The 50% task-completion time horizon, popularized by METR, is the human task length at which an agent is predicted to succeed half the time on a particular suite of mostly software tasks. The case that this single figure is an inadequate summary for forecasting is well supported, and is acknowledged in part by METR itself, which reports 80% horizons alongside it and notes that they are roughly four to six times shorter. Forecasts of when agents can do real work need success rates at the reliability that work demands, the effect of scaffolding, inference budgets and repeated attempts, and evidence that the suite generalizes to other domains; none of these can be read off one p50 number, and the p50 estimate itself becomes poorly constrained at long durations. Independent methodological audits, such as a 2026 preprint on frontier-AI forecasting measurement, reach the same conclusion and propose reliability curves across duration, attempts and budgets in its place.
The strongest counterpoint is that METR found the 80% horizon doubling at essentially the same rate as the 50% horizon. If the reliability curve shifts in parallel, the p50 trend is an adequate proxy for the rate of progress, though not for the level of capability at deployment-relevant reliability. The claim therefore holds most clearly for forecasts of absolute capability or deployment readiness, and more weakly for forecasts only of growth rates. Whether the parallel-shift pattern persists as models and task suites change is the main open question.
Full reasoning: the evidence and decisions behind this verdict
Own model: p50 is one quantile of a fitted logistic over task length on one task suite; any forecast whose target is "reliably does X" or "does X in another domain" needs more than that point unless the curve shape and domain mapping are stable and known. Evidence: Kwa et al. 2025 (arXiv 2503.14499, sec 3.2.1) report 80% horizon doubling time 204 days vs 207 for 50%, with 80% horizons 4-6x shorter, and say models "cannot reliably perform tasks of moderate length". This both supports the claim (level at useful reliability differs greatly from p50) and grounds the main counter-argument (rates are parallel). Costa 2026 (arXiv 2608.14903) audits the forecasting record: only seven systems jointly observe compute and a METR p50, TH1.0-to-1.1 linking slope 1.206, 73% of quantitative events from one programme; it lists one-point scores as an audit failure and recommends reliability curves over duration, attempts and budget, while finding no replacement scalar. The linked subclaim that long p50 horizons on TH1.1 are unreliable (unassessed) would further support the claim. Status supported rather than verified because "inadequate" is evaluative and depends on forecasting purpose; for pure rate-of-progress extrapolation the parallel-trend evidence gives p50 a defensible role. Credence omitted as the claim is evaluative.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
A 50% horizon is one point read off a fitted success-versus-task-length curve. Forecasts of when agents can do economically or strategically relevant work need the success rate at the reliability that work demands (METR's own 80% horizons are about 4 to 6 times shorter), its dependence on scaffolding, inference budget and repeated attempts, and how it transfers beyond one software task suite. These vary across models and domains and are not recoverable from the p50 figure alone, and the p50 estimate itself becomes noisy at long durations (long p50 horizons on the current suite are unreliable measurements). So a single p50 number leaves out information a forecast needs.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that forecasts need reliability-, protocol- and domain-specific information that varies independently of the p50 figure, the conclusion that one p50 number is an inadequate summary follows for most forecasting purposes. The premise about 80% horizons being several times shorter is directly reported by METR; the premise that p50 is noisy at long durations rests on METR Time Horizon 1.1's 50% time horizons above roughly 16 human-hours are unreliable measurements., not yet assessed. The inference is weaker for forecasts that target only growth rates, where the parallel-trend counterargument bites.
Provenance
Where this claim has been said, linked to its canonical form.
Neither source has been traced to an origin, or to one another.
Reliability curves replace a single p50 horizon with success probability across task duration, repeated attempts and time budgets.
Section 7 portfolio; Figure 6 lists 'One-point score as intrinsic ability' as an audit failure.
Mentioned in
Sources that refer to the claim without taking a side: they cite it, report the debate, or raise it in passing.
However, models' 80% time horizons are 4-6x shorter, suggesting that even models that sometimes succeed on difficult and diverse tasks cannot reliably perform tasks of moderate length.
Section 3.2.1 compares 50% and 80% horizons; doubling times are similar (207 vs 204 days) but levels differ substantially.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="50758adb-ea92-4663-adb8-70d8985d4ebd"><a href="https://minerval.ai/claims/50758adb-ea92-4663-adb8-70d8985d4ebd">A single 50% task-completion horizon inadequately summarizes AI-agent capability for forecasting.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/50758adb-ea92-4663-adb8-70d8985d4ebd" title="Minerval: A single 50% task-completion horizon inadequately summarizes AI-agent capability for fore…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/50758adb-ea92-4663-adb8-70d8985d4ebd)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.