trailUsing AI tools caused experienced open-source developers to complete tasks about 20% slower in early 2025.
AI tools sped up developers more in early 2026 than early-2025 estimates indicated.
Coding benchmark scores and anecdotal reports overestimate real-world AI coding capability.
AI coding assistants substantially speed up developers on many programming tasks.
+2
Randomized trial evidence· for
In METR's 2025 randomized controlled trial, using AI tools increased developers' task completion time by 19%
atomic
METR's measured AI slowdown is robust across alternative statistical estimators and empirical strategies
shared · shown elsewhere
METR's 2025 sample of 246 completed issues from 16 developers gave sufficient statistical power to detect AI's effect on developer productivity.
Issue-level randomization made METR's 246 completed issues, not its 16 developers, the effective sample for estimating the AI slowdown.
METR's measured AI slowdown remained statistically significant when accounting for developer-level clustering.
The METR study had sufficient statistical power to detect a productivity slowdown but insufficient power to estimate its magnitude precisely.
+1 more
After completing the tasks, developers who had been slowed down by AI tools still believed AI had sped them up by about 20%.
atomic
claim · empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →claim page ↗︎
Using AI tools caused experienced open-source developers to complete tasks about 20% slower in early 2025.
↑SupportedEvidence favors the claim, but the chain is incomplete or the sources are secondary.constitution →credence 0.70Credence, from 0 to 1: the Steward's probability that the claim, as stated, is true. Stated only where a single number is an honest summary; normative and evaluative claims usually carry none.constitution →
5depended on by
this rests on ↓
claimA box is a claim: a single proposition the graph assesses, with its own page and map. Click any claim to centre the map on it.constitution →argumentA pill is an argument: one line of reasoning stating how the claims beneath it combine to bear on the claim above it, for or against. Arguments are not destinations; click their claims to explore.constitution →✓verifiedThe claim traces to reliable primary sources through a clear chain of evidence.constitution →↑supportedEvidence favors the claim, but the chain is incomplete or the sources are secondary.constitution →⇄contestedCredible evidence or argument exists on multiple sides.constitution →○unsupportedNo credible evidence found, though the claim is not contradicted.constitution →✕contradictedAvailable evidence weighs against the claim.constitution →?unknownInsufficient information to assess.constitution →◌unassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →verified factopen questionvalue premisetheoremprizeA double ring marks a claim with a live prize: an amount in owls offered for a machine-checked proof or disproof of its published formal statement. The amount is in the preview and on the claim page; a prize changes nothing about how the claim is assessed or how important it is judged to be.prize rules →⊢machine-checked⊢ marks a claim whose published formal statement has a machine-checked proof or disproof. The checker confirms the proof; the verdict beside it is still the steward's judgment of the claim as worded.prize rules →supportsthis provides evidence for the parentsteward instructions →contradictsthis argues against the parentsteward instructions →assumesbackground the parent's framing takes as givensteward instructions →requiresa load-bearing premise: the parent is false without itsteward instructions →Fig. Detail falls off with distance; every claim is an address.