Selection effects from AI-reliant developers opting out bias measured AI productivity speedup estimates downward.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass (structure_and_assess). Decomposition: separated the claim into its observed premise and its counterfactual premise, both novel per match_claim: "Developers who rely on AI increasingly decline to participate in productivity experiments that require working without AI" (requires, seeded 0.9) and "Developers who opt out of AI productivity experiments would experience larger AI speedups than those who participate" (requires, seeded 0.65, the crux). Attached existing claims 843dc97c (selection driven by expected uplift, supports) and a0fea173 (self-reported speedup estimates unreliable, contradicts). Grouped under two named arguments, "Differential attrition" (for, holds) and "Miscalibrated self-selection" (against, holds_with_caveats), with written forms and evaluations recorded. Evidence: read METR's 2026-02-24 uplift update (primary, originator; both recorded instances affirm) plus secondary commentary via two web searches; recorded one new affirming instance (philippdubach.com); found no published outright denial, only a reframing of opt-out as dependency (paddo.dev), reflected in the against argument. Verdict: supported, confidence 0.75, credence 0.8: the behavioral premise is established, the direction-of-bias inference is consensus and plausible but rests on an unobservable counterfactual that METR itself hedges as "likely". Marginal yield 0.25: little to gain until METR's redesigned experiment yields data. Importance set 0.5 (from 0.55), contestation 0.4. Canonical form kept: fifteen words, neutral, accepted by both the source and commentators as what is in dispute. Notified all three dependent parents, since this is the claim's first assessment and each attached it pre-assessment.
Assessed Supported
verdict confidence 0.75 · credence 0.80
The claim originates with METR, whose February 2026 update to its developer productivity experiment reported that the primary problem with its new data was a significant rise in developers declining to participate because they did not wish to work without AI, with commentators citing refusal rates of 30 to 50 percent. That behavioral fact is directly observed and essentially undisputed. Whether the opt-out biases measured speedups downward turns on a premise that cannot be observed directly: that the developers who opt out would show larger AI speedups than those who participate. The case for it is that the selection appears driven by expected AI benefit rather than by the reduced pay rate, and that developers who refuse to work without AI have typically built their workflows around it, which tends to raise their measured speedup relative to unassisted work. The main reservation is that developers' self-assessments of AI-driven speedup are unreliable: in METR's own earlier study, participants believed AI had sped them up while it had in fact slowed them down, so a strong preference for working with AI does not by itself establish a larger true benefit. On balance the direction of the bias is probable but not established. METR itself states the effect only as likely, and the surrounding commentary broadly accepts it, with some observers reframing the refusal to work without AI as dependency rather than evidence of higher productivity. Data from METR's redesigned experiment, or any measurement of opt-out developers under a design they will accept, would sharpen the picture.
Claim entered the graph