AI benchmarks overestimate models' capabilities on real-world tasks.
4 events · 1 assessment · 2 decisions
Structured and assessed
First pass. Created three arguments: two for (grading versus holistic review, and contamination) and one against (under-elicitation). Minted three subclaims and linked the existing time-horizon generalization claim as contradicting. Added see-also links to the benchmark-impact claim, the reconciliation question, and the maintainer-rate trend claim. Reclassified the METR July 2025 instance as 'poses', since it was one hypothesis among several. Recorded the METR March 2026 note as affirming. Assessed as supported (0.7, credence 0.75). Importance 0.6, contestation 0.55. There are no dependents to notify.
Assessed Supported
verdict confidence 0.70 · credence 0.75
Read at face value, the scores of widely used AI benchmarks tend to overstate how often the same models succeed on the corresponding real-world tasks. The best direct evidence comes from software engineering. Blinded maintainer reviews of agent patches that passed SWE-bench Verified found that, after adjusting for reviewers' own strictness, roughly half would not have been merged. Merge rates ran about 24 percentage points below the automated grader. A second mechanism, training-data contamination and overfitting to public benchmarks, inflates some scores relative to fresh items of the same kind. Both effects follow from what benchmarks have to be: tasks that are well scoped and scored by algorithm, in a form models can also be trained against. The overestimation is not universal or fixed. Benchmarks run models with fixed scaffolds, one attempt and no feedback, and so can understate what models achieve with better elicitation and iteration. The researchers behind the maintainer study say explicitly that their gap is not a fundamental capability limit. Benchmark trends may also carry over to real-world work even where the levels differ. The claim is therefore best read as true of naive readings of benchmark scores, especially in coding, and less certain as a statement about every benchmark or about elicited capability. Systematic comparisons of benchmark and real-world success outside software, and studies that give agents real-world feedback loops, would settle how general the gap is.
Updated claim instance
Instance a7a8d42a-9b0f-46f2-8bb7-d42b3d51b9cb (https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/): was stance=affirms, confidence=0.7; set stance="poses", speaker="METR", publication="METR", sourceDate="2025-07-10". The passage is one of several hypotheses METR lists for reconciling its RCT slowdown with high benchmark scores, alongside the possibility that the RCT underestimates capabilities; it states the proposition as a possibility rather than asserting it.
Claim entered the graph