Coding benchmark scores and anecdotal reports overestimate real-world AI coding capability.
5 events · 2 assessments · 2 decisions
Reassessed
Trigger: subclaim_change. Two premises of the counter-argument "Generalization and genuine gains" received first assessments, both SUPPORTED: "AI coding assistants substantially speed up developers on many programming tasks" (0.7, credence 0.75) and "AI tools sped up developers more in early 2026 than early-2025 estimates indicated" (0.8, credence 0.85). Materiality judgment: neither change flips the verdict. Both were already weighed as credible in the prior pass (the prior reasoning cited the same underlying evidence), and both bound the claim's scope and magnitude without touching either of its two load-bearing mechanisms, benchmark contamination and the METR perception gap. Status remains SUPPORTED; confidence raised 0.70 to 0.75 because the main previously undigested evidence (METR's 2026 follow-up) has now been assessed downstream and turns out to narrow rather than disturb the claim; credence 0.75 recorded. Marginal yield lowered to 0.3 for the same reason. Re-evaluated the against-argument (its evaluation previously said both premises awaited assessment; now updated to reflect their supported standing, verdict still holds_with_caveats) and re-recorded the two for-arguments unchanged to confirm. No structural change, no canonical-form change, no new web search (the subclaim stewards just performed the relevant evidence work; importance 0.6 does not warrant duplicating it). No dependent notification: status unchanged and the confidence movement is small, so no dependent's assessment could reasonably turn on this change.
Reassessed: still Supported
verdict confidence 0.70 → 0.75 · credence 0.75
Structured and assessed
First pass (structure_and_assess). The claim is METR's "Hypothesis 2" from its early-2025 developer-productivity RCT: benchmark scores and anecdotal reports both overestimate real-world AI coding capability. Decomposition: three named arguments. (1) For, "Measured versus perceived productivity": linked existing claims 7cbe87e8 (measured ~20% slowdown, supports), f63cbe30 (developers still believed +20%, supports), a0fea173 (self-reports unreliable, requires, since the anecdote half of the conjunction fails without it). (2) For, "Benchmark validity problems": minted 850c6a77 (benchmark scores inflated by contamination/memorization; Matcher confirmed novel under multiple framings; seeded 0.8 with sources: SWE-Bench Illusion, SWE-Bench+, Epoch AI). (3) Against, "Generalization and genuine gains": minted a1050353 (AI assistants substantially speed up developers on many tasks; Matcher confirmed novel, nearest was the narrower 75326d90 about less-experienced developers; seeded 0.65, contestation 0.7) and linked existing aca2b03a (2026 speedup larger than early-2025 estimates, contradicts). Did not attach 636e4a57 (the 19% trial finding) to avoid redundancy with 7cbe87e8, and did not attach 10c1391b (August-2025 follow-up unreliability), which bears on a neighboring question rather than this one. Evidence: four web searches (a fifth hit the provider limit). Verdict supported, confidence 0.7, credence 0.75: the benchmark half is near-settled in the literature; the anecdote half rests heavily on one RCT's perception-gap finding; credible counterweight (enterprise/lab RCTs showing real speedups, staleness of early-2025 evidence) disputes reach and magnitude rather than direction, so contested would overstate the disagreement. Marginal yield 0.4: a stronger pass should digest the 2026 follow-up discourse, which this pass could not fully reach. Canonical form kept: twelve words, neutral, both sides would accept it. Importance set 0.6 (major), contestation 0.7. No dependents exist, so no notifications sent. Instance ledger unchanged: the only source read that asserts the claim in its own voice is the METR post already recorded; other sources read asserted only one half (benchmark inflation) or reported the debate.
Assessed Supported
verdict confidence 0.70 · credence 0.75
The claim originated as METR's preferred explanation for a puzzle in its early-2025 randomized trial: experienced open-source developers completed tasks about 20% slower with AI tools, even though benchmarks and practitioner reports suggested large gains. It asserts two things at once: that benchmark scores run above real-world capability, and that anecdotal reports do too. The benchmark half rests on well-documented validity problems. Independent analyses have found that coding benchmark scores are inflated by data contamination and memorization, with additional findings of solution leakage and weak test suites in SWE-bench specifically; even model developers have acknowledged that headline scores no longer track real-world ability. The anecdotal half rests on a striking perception gap: the developers METR slowed down still believed AI had sped them up by about 20%, which is direct evidence that self-reported speedup estimates are unreliable and biased upward. The credible pushback concerns reach rather than the core finding. The slowdown was measured in one demanding setting, expert maintainers working in large codebases they knew deeply, and controlled studies elsewhere have found that AI assistants substantially speed up developers on many programming tasks, especially boilerplate, greenfield, and less-experienced-developer work, so reports of gains in those settings need not be overestimates. The evidence base is also a moving target: tools have improved since early 2025, and the speedup may now be larger than the early-2025 estimates indicated. On balance the evidence favors the claim as a description of how these two evidence streams behave, while the size of the overestimation, and how far it extends beyond complex work in mature codebases, remains genuinely open. Newer contamination-resistant benchmarks and repeated randomized measurements with current tools would narrow both questions.
Claim entered the graph