AI agents’ improvement trends plateau or slow specifically on higher-messiness tasks.
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
The only direct measurement of AI agents' improvement trends split by task messiness, in METR's 2025 study of long-task completion, points against this claim. METR scored its tasks on 16 messiness factors. Agents did worse on messier tasks than task length alone predicts, but success on the messier half improved at about the same rate as on the cleaner half. The authors report no evidence of a plateau, or of much slower trends, specific to the high-messiness subset.
The claim is still repeated in popular commentary on the time-horizon graph. That commentary usually confuses a lower level of performance on messy tasks with a slower rate of improvement on them, and the data distinguish the two. The serious case for the claim rests on the limits of the evidence, not on any contrary measurement. The high-messiness subset is small and its success rates are low, so a modest difference in slope would be hard to detect. And even METR's messiest tasks are much cleaner than real-world work, so the finding says little about genuinely messy tasks. The evidence weighs against a messiness-specific slowdown without ruling one out. A trend analysis on a larger, more realistic task set, with a formal test of whether the slopes differ, would settle the question.
Full reasoning: the evidence and decisions behind this verdict
Prior model: a lower success level on messy tasks is expected. Whether the rate of improvement is lower is a separate question that needs trend data split by messiness. I recalled METR's 2025 paper making exactly this comparison and finding no difference, and reading the arXiv text (arxiv.org/html/2503.14499v1) confirmed it. On HCAST, the length-controlled residual regresses on messiness with b = -0.081 and R² = 0.251. The paper says trends in model performance improvements "are not obviously slower on the high messiness split" and that it finds "no evidence of either much slower performance trends, or a plateau, specific to our higher messiness subset." It also notes that a similar exponential trend appears on the messier subset with a shorter doubling time. If anything, that points the other way, though it is noisy.
The affirming voice found is a popular Medium critique (medium.com/write-your-world/this-is-the-most-misunderstood-graph-in-ai-fb14ad52d743). It cites low absolute success on the messiest half (no model above 30%) as evidence of "far slower progress." That conflates level with rate, and it contradicts the paper's own trend analysis. A LessWrong discussion raises the possibility that real-world improvement could be slower, but only as speculation.
Limits: the subset is small, so a modest difference in slopes would be hard to detect. The comparison is mostly descriptive, with no formal test of a slope difference. The messiest tasks sit low on the messiness scale compared with real work. And the evidence is from early 2025 models; later METR updates were not checked in this pass. Because of these limits the status is doubtful, not contradicted. Credence 0.3 reflects the remaining chance that a messiness-specific slowdown appears on realistically messy tasks.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because agents perform worse on messier tasks than length alone predicts, and because even METR's messiest tasks are much cleaner than real-world work, proponents conclude that the impressive time-horizon trend is driven by clean tasks and that progress on genuinely messy tasks is slower.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 2 not separately assessed
Even granting both premises, a lower level of performance on messy tasks does not imply a slower rate of improvement on them, and that step is where the popular version of the argument fails. The premise that benchmark messiness is far below real-world messiness does real work, but only to weaken the contrary evidence. It does not show a slowdown. The premise that agents do worse on messier tasks is well supported but bears on the level, not the trend.
METR split its task suite by messiness score and found that success on the messier half improves at roughly the same rate as on the cleaner half, with no sign of a plateau; since this is the only direct measurement of improvement trends by messiness, the claimed messiness-specific slowdown is not seen in the available data.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
If the premise holds, the claimed messiness-specific slowdown is absent from the only data that measure it, which makes the claim's denial much more likely. It is not certain, because a small subset can hide a modest difference in slopes and the tasks are not realistically messy. The argument rests on similar improvement rates across the messiness splits, which METR's 2025 paper reports directly.
Provenance
Where this claim has been said, linked to its canonical form.
In particular, we find no evidence of either much slower performance trends, or a plateau, specific to our higher messiness subset.
Section on messiness factors: after controlling for length agents do worse on messier tasks, but the improvement trend on the messier half is similar.
The cleanest tasks—the ones that look least like real work—drive the impressive time horizon numbers. The realistic tasks show far slower progress.
Popular critique of the METR time-horizon graph, arguing messy tasks show slower progress.
In this subset, however, the authors found no evidence that the improvement trend had plateaued or slowed specifically for higher-messiness tasks.
Agents performed worse on messier tasks after controlling for length.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 1, 2026. Every judgment on this page is accompanied by a reasoning trace.