METR's 2025 sample of 246 completed issues from 16 developers gave sufficient statistical power to detect AI's effect on developer productivity.
Assessment
Evidence favors the claim, but the chain is incomplete or the sources are secondary.
The claim concerns the most common objection to METR's 2025 finding that AI tools slowed experienced open-source developers: that a study of only 16 people cannot support statistical conclusions. The evidence favors the claim, with an important qualification about what "sufficient power" covered.
The study randomized AI use at the level of individual issues, with every developer working in both conditions, so the effective sample for the average effect was the 246 completed issues rather than the 16 developers, provided within-developer correlation is modest, which METR reports it is. The decisive test is whether the result survives inference that respects the developer clusters, and it does: the measured slowdown remained statistically significant under developer-clustered standard errors, a result confirmed by an independent reanalysis of the open-sourced data that also applied a hierarchical bootstrap, the method of choice when clusters are few. More broadly, the finding is robust across alternative estimators and empirical strategies.
The qualification is that the power was marginal and specific to detection: the 95% confidence interval ran from roughly a 2% to a 40% slowdown, so the study could establish the existence and direction of the effect but not its magnitude with any precision, and per-developer or subgroup estimates are noisier still. METR's own characterization, "just enough" power to reject zero effect, is accurate. Readers who cite the study for the existence of a slowdown in this setting are on solid statistical ground; readers who lean on the 19% point estimate as a precise quantity are not. Whether the result generalizes beyond these developers and projects is a separate question from power and is treated as its own claim.
Full reasoning: the evidence and decisions behind this verdict
Three lines of evidence were weighed. First, METR's own methodology FAQ (metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) addresses the sample-size objection directly: confidence intervals were computed with developer-clustered standard errors, no meaningful within-developer structure was observed, and the 246 issues gave "just enough" power to reject the null of zero speedup or slowdown. This is the study team's own account, so it carries weight on what was computed but limited weight on whether the computation was sound. Second, an independent reanalysis by Simas Kucinskas (inexactscience.substack.com/p/are-ai-coding-tools-slowing-you-down), working from METR's open-sourced data and code, tested exactly the vulnerability at issue: with only 16 clusters, cluster-robust standard errors can misbehave, so he ran both clustered standard errors and a hierarchical bootstrap and found the key findings largely unchanged. This independent confirmation is what moves the verdict from the study's self-report to supported. Third, a participating developer (domenic.me/metr-ai-productivity/) reports the headline 95% confidence interval as roughly a 2% to 40% slowdown, which excludes zero but only narrowly, corroborating both detection and the marginal character of the power.
The material subclaims weigh as follows. The load-bearing premise is that issue-level randomization made the 246 issues the effective sample; this holds only if within-developer correlation is low, which METR asserts and the reanalysis's clustering checks corroborate, but which no fully adversarial third party has yet quantified (for example by publishing the intraclass correlation). That residual dependence, plus the post-hoc character of the power argument (a barely significant result can also arise in an underpowered study that got a favorable draw, in which case the estimate would be inflated by winner's-curse selection), is why the status is supported rather than verified and the credence 0.8 rather than higher. The detection-versus-magnitude specification does not weigh against the claim as stated, which asserts power to detect, but it bounds any stronger reading.
What would change the conclusion: a reanalysis finding substantial intraclass correlation or showing the significance vanishing under a well-founded few-cluster correction (for example wild cluster bootstrap) would flip this toward contradicted; a formal design-based power analysis or further independent replication of the inference would move it toward verified. Contest in the discourse ("n=16 is too small") turned out on inspection to be mostly about external validity, which is the separate generalization claim, not about power; credible technical instances on the power question itself all affirm, so contested is not the right status.
Decomposition
The claims this one rests on directly. ↗︎ opens a subclaim; the map shows how they fit together.
The claims this one rests on directly, not gathered into a named line of reasoning.
- requiresa load-bearing premise: the parent is false without itsteward instructions →Issue-level randomization made METR's 246 completed issues, not its 16 developers, the effective sample for estimating the AI slowdown. ↗︎
- supportsthis provides evidence for the parentsteward instructions →METR's measured AI slowdown remained statistically significant when accounting for developer-level clustering. ↗︎
- specifiesa more specific version of the parentsteward instructions →The METR study had sufficient statistical power to detect a productivity slowdown but insufficient power to estimate its magnitude precisely. ↗︎
- supportsthis provides evidence for the parentsteward instructions →METR's measured AI slowdown is robust across alternative statistical estimators and empirical strategies ↗︎
Provenance
Where this claim has been said, linked to its canonical form.
the 246 total completed issues give us (just enough) sufficient statistical power to reject the null hypothesis of zero speedup/slowdown
FAQ response to "You only had 16 developers, so these results will not generalize/replicate."
My main takeaway, after examining the data: METR's productivity study appears robust. The key findings remain largely unchanged if you slice the data slightly differently.
An independent reanalysis of METR's open-sourced data and code, focused specifically on internal validity: whether the results survive developer-clustered standard errors and a hierarchical bootstrap given only 16 developers. The author concludes the statistical analyses hold up.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Aug 10, 2026. Every judgment on this page is accompanied by a reasoning trace.