How can measured AI productivity effects on developers be reconciled with benchmark scores and anecdotal reports of AI helpfulness and adoption?
Assessment
The Steward's best answer, and how well answered the question is: the evidence favors an answer without establishing it.
The evidence favors a combined answer rather than any single one. The tension is smaller than it first looks, because the three kinds of evidence measure different things. The answers below work alongside each other and do not exclude each other.
First, METR's early-2025 randomized trial measured one narrow setting: 16 experienced developers working on large, mature repositories they knew well, with high quality standards and early-2025 tools. It found that tasks took about 19% longer with AI. This is the setting where AI help would plausibly be smallest. The evidence therefore supports the view that effects vary by context, and it does not support the view that AI makes developers slower in general. Second, benchmarks overstate real-world capability. They score self-contained tasks with automated checks and leave out the context, code quality, and review burden that real projects carry. Third, the same trial shows that self-reported gains can be badly miscalibrated. Developers forecast a 24% speedup and afterwards believed they had been 20% faster, while the measured result was a slowdown. Anecdotes are therefore weak evidence of speedup.
Time also matters. METR's February 2026 update ran a follow-up with 57 developers and found raw point estimates of speedup: 18% for returning developers and 4% for new ones. Neither interval excludes zero. The update also reports that many developers refused to do tasks without AI, which biases the estimate against speedup. METR judges it likely that developers were sped up by early 2026, but says its data are only weak evidence of how large that speedup is. The early-2025 slowdown is best read as a snapshot of early-2025 tools.
Two answers are plausible but less well supported. One is that people adopt the tools for lower effort or enjoyment rather than speed. The other is that limited prior experience with Cursor inflated the measured slowdown. Questions that remain open: how large real speedups are in different settings, and how output quality differs with and without AI. A well-powered trial without selection effects, or credible observational data, would settle how much weight each answer carries.
Full reasoning: how the answers were weighed
My view before reading the sources was that the three evidence streams measure different things: an effect on time in one particular population, capability on self-contained tasks, and perceived usefulness. They need not agree. Reading METR's July 2025 post, which asks this question, confirmed that its own reconciliation rests on differences of setting, benchmark overestimation, and unreliable self-reports, and that it treats these as compatible. I read METR's February 24, 2026 update directly. It reports a follow-up with 57 developers, 143 repositories, and over 800 tasks. The raw estimates were a speedup of 18% for returning developers (CI −38% to +9%) and 4% for new developers (CI −15% to +9%). The update names several selection problems: developers declined to take part, 30 to 50% of them withheld tasks they did not want to do without AI, pay fell from $150 to $50 per hour, and time tracking was unreliable when developers ran several agents at once. METR concludes that the estimate is likely a lower bound and a bad proxy for the true effect. Secondary coverage disagrees on whether the follow-up "reversed" the original finding. The defensible reading is that the sign probably changed but the size is not identified. I used no other randomized trials on this pass. Older Copilot trials on simpler or greenfield tasks found speedups, which supports the context-dependence answer, but I did not re-read them here. The standing is 'leaning' rather than 'answered' because how much each mechanism contributes has not been measured. Evidence that would change this assessment: a clean trial showing slowdowns that persist in less familiar or greenfield settings would weaken the context answer; a trial finding self-reports accurate would weaken the perception answer. The two presupposed nodes, internal validity of the original trial and the measured quantity, have not yet been assessed. If the original estimate proved invalid, part of the question would lose its premise.
Answers
Each candidate answer is a claim or a quantity with a page of its own, in the set for its quality or its prominence. How the answers bear on one another is recorded pair by pair.
- UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →favored by the assessmentserious and widely heldcan be true with C
AI benchmarks overestimate models' capabilities on real-world tasks.
SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →favored by the assessmentserious and widely heldcan be true with FDevelopers substantially overestimate productivity gains from AI coding tools.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →favored by the assessmentserious and widely heldcan be true with A- UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →a serious answer, whether or not anyone prominent holds it
Limited prior experience with Cursor among METR RCT participants inflated the measured AI slowdown.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →serious and widely heldBy early 2026, AI tools reduced experienced open-source developers' task-completion times.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →favored by the assessmentserious and widely heldcan be true with B
The set is incomplete Missing as candidate answers: that output quality or scope differs between conditions, so that time alone misprices AI's value (METR's 2026 update notes this), and evidence from field experiments on less experienced developers. Neither has been added as a node yet.
Provenance
Where sources ask this question, and where they answer it.
Asked
So how do we reconcile our results with impressive AI benchmark scores, and anecdotal reports of AI helpfulness and widespread adoption of AI tools?
Answered
- https://metr.org/blog/2026-02-24-uplift-update/ answers By early 2026, AI tools reduced experienced open-source developers' task-completion times.
Based on conversations with study participants, we believe it is likely that developers are more sped up from AI tools now — in early 2026 — compared to our estimates from early 2025. However, because of the selection effects in our experiment, our data is only very weak evidence for the size of this increase.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Created by extractor · Nov 1, 2026. What a question isA question is answered rather than affirmed or denied, and has a Steward of its own (constitution §7a).constitution →.