Percentage change in issue-completion time for experienced open-source developers using early-2025 AI tools on their own repositories (%)
- Measures
- The causal effect of being allowed to use AI tools on the time taken to implement a real issue, expressed as the percentage change in completion time relative to working without generative AI (positive = slower, negative = faster).
- Population
- Experienced open-source developers (multi-year contributors) working on real issues (roughly 20 minutes to 4 hours) in large, mature repositories they maintain.
- Conditions
- AI tools at the frontier in early 2025 (chiefly Cursor Pro with Claude 3.5/3.7 Sonnet), used as the developer chose; completion judged as code the developer believes will pass review. Measured as the average treatment effect across issues, as in a randomized issue-level comparison of AI-allowed versus AI-disallowed work.
- Unit
- %
Most likely about 15 %; 90% interval −3.0 to 33 %.
Stated as a distribution, the precision the evidence supports. roughly normal, shifted slightly toward zero from the trial's point estimate
Assessment
The graph's estimate, all things considered: the figures and arguments it rests on, and how they were combined.
The only direct measurement of this quantity is METR's randomized trial run from February to June 2025. In it, 16 experienced maintainers of large open-source repositories worked on 246 real issues, each randomly assigned to allow or forbid AI. The tools were chiefly Cursor Pro with Claude 3.5 and 3.7 Sonnet. When AI was allowed, issues took 19% longer, with a 95% confidence interval of +2% to +39%. The developers had expected a 24% speedup, and afterwards they still believed they had been sped up by about 20%. The authors report high compliance, no differential dropout, similar PR quality in both arms, and a slowdown that survives alternative outcomes and estimators.
The best estimate is a slowdown of about 15%, with a 90% interval from roughly -3% to +33%. That puts most of the probability on a slowdown and little on a meaningful speedup. The central value sits a little below the trial's figure for three reasons. The sample is small, at 16 developers. Times were self-reported. And many participants were new to Cursor, which may have inflated the measured slowdown, though the authors found little evidence of a learning trend. How far the figure can be trusted rests mainly on the trial's internal validity.
The quantity is tied to early-2025 tools. METR's February 2026 follow-up with late-2025 tools gives point estimates pointing to speedups (-18% for returning developers, -4% for new recruits). Both intervals cross zero, and selection effects compromise them. Those results describe a different tool generation and do not revise this figure. They also do not support generalizing it to current tools.
The work: which figures, how they were combined, and the arithmetic
Figures: METR (2025-07-10; restated 2026-02-24) point estimate +19%, 95% CI +2% to +39%. This implies a standard error of roughly 9 to 10 percentage points. A 90% interval centred on the trial alone would be about +3% to +35%.
Adjustments, all judgment: (1) Mild shrinkage toward zero, about 3 to 4 points. The prior from other productivity studies of AI coding tools centres on modest speedups, though those are mostly in different populations. The trial also has the open issues of self-reported timing and possible learning effects (Cursor unfamiliarity: about 44% had used Cursor before; one developer with more than 50 hours showed a speedup). (2) The interval is widened slightly for model uncertainty beyond sampling error (16 developers, clustering).
Result: median +15%, 90% interval about -3% to +33%, P(slowdown) about 0.9.
Not used as estimates of this quantity: the developers' forecast of -24% and their post-hoc perception of -20%, which are beliefs, not measurements, and the late-2025 follow-up results (-18% returning, CI -38% to +9%; -4% new, CI -15% to +9%), which cover different tools.
Source: metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ ; metr.org/blog/2026-02-24-uplift-update/
Estimates
Each figure a source states for this quantity, in its own words, under the argument it belongs to and read against the graph's estimate.
Grey band: the graph's 90% interval, with its central value as a line. Dark marks: each source's own figure. A thin line is an interval, a thick bar its middle half, a dot a single figure, a dashed arrow a bound. An open box is where an argument places the value.
Decomposition
The arguments the estimate rests on, each stated as it runs, with what it shows about the value and its premises linked inline. ↗︎ opens a premise; the map shows how they fit together.
Because many participants were new to Cursor and this inflated the measured slowdown, the trial's +19% partly reflects a learning cost, so the effect for developers already practiced with early-2025 tools would be smaller, possibly near zero or a speedup.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 not separately assessed
Suppose unfamiliarity did inflate the slowdown. That moves the effect for practiced users below +19%, but it says nothing about how far below. And under this quantity's definition, the developers studied were the actual population of interest. The argument depends on the learning-cost explanation, which is supported only weakly so far. The trial's authors report no clear learning trend, and only one participant was highly experienced with Cursor.
METR randomized 246 real issues from 16 experienced maintainers of large open-source repositories to AI-allowed or AI-disallowed conditions and found completion time 19% longer when AI was allowed (95% CI +2% to +39%). Given that the trial's estimate is internally valid, and that its participants, repositories and tools match this quantity's definition, the figure estimates the quantity directly.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
If the trial is internally valid, its randomized comparison makes a value in or near the +2% to +39% range much more likely for this population. The sample is small, so the range is wide and the result is not guaranteed. The argument depends on the trial's internal validity, which looks plausible but has not yet been assessed. The main open points are self-reported timing and the fact that there were only 16 developers.
Provenance
Where this quantity's evidence comes from. The figures sources state are under Estimates above.
When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts.
RCT of 16 experienced open-source developers on 246 issues in their own repositories, early-2025 tools (mainly Cursor Pro with Claude 3.5/3.7 Sonnet). The page later notes these results are out of date.
Our early 2025 study found the use of AI causes tasks to take 19% longer, with a confidence interval between +2% and +39%.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Created by extractor · Nov 1, 2026. Every judgment on this page is accompanied by a reasoning trace.