Generative AI yields smaller productivity gains on complex, context-dependent tasks than on well-defined tasks.
Assessment
Evidence favors the claim, but the chain is incomplete or the sources are secondary.
The evidence favors the claim, with two qualifications about what exactly the contrast is and how long it will hold. The large productivity gains that made headlines came from tasks with clear specifications and readily checked outputs: controlled studies of customer support, business writing, and self-contained coding found gains of roughly 14 to 55 percent. Where researchers have looked at work that is context-rich or beyond what the model handles well, the picture changes. In the Boston Consulting Group field experiment, the same consultants who gained substantially on tasks inside the model's frontier performed worse than unaided colleagues on a business problem outside it. In a 2025 randomized trial, experienced maintainers working on real issues in large, familiar codebases were about 20 percent slower with AI tools, even while believing they had been sped up. A 2026 meta-analysis of coding studies finds the same pattern across the literature: measured gains are larger in controlled experiments than in real open-source and enterprise development, which its authors attribute to the greater complexity of real codebases and workflows.
The first qualification concerns the variable. A review of randomized studies argues that generative AI does improve speed and quality on complex tasks too, and that task ambiguity, more than complexity, is what moderates the effect: structured tasks show the most consistent gains, ambiguous or judgment-based ones the most variable. Early syntheses even argued that more cognitively demanding tasks showed the bigger gains, though the studies they drew on all used well-specified tasks. Read as a contrast between well-defined work and work that is ambiguous and context-dependent, the claim holds well; read as a contrast between simple and complex work, it is weaker.
The second concerns durability. The sharpest hard-task evidence comes from early-2025 tools, and the group that ran that trial has since said the slowdown finding is outdated and that developers now appear to be sped up more than those measurements indicated, though its follow-up data were compromised by selection effects. No study has yet shown gains on complex, context-dependent work matching those on well-defined tasks, and the uneven "jagged frontier" of model capability suggests some gap will persist even as it moves, but the size of the gap today is less certain than its direction. A reliable randomized comparison of current tools across both task types would settle the question.
Full reasoning: the evidence and decisions behind this verdict
The claim is comparative, so it needs both halves. The well-defined baseline is not in doubt: Brynjolfsson and colleagues' customer-support deployment (about 14 to 15 percent more issues resolved per hour), Noy and Zhang's writing experiment in Science, Peng and colleagues' 55 percent Copilot speedup, and enterprise Copilot trials at Microsoft and Accenture (about 26 percent more tasks completed) are consistent, and the graph's claim that controlled studies find substantial task-level gains is treated here as effectively established even though it has not yet received its own assessment.
For the complex, context-dependent side, three pieces of evidence point the same way. First, Dell'Acqua and colleagues' BCG experiment is the only major study that varied task type within one design; consultants gained on the inside-frontier creative task and lost accuracy on the outside-frontier task, which was chosen to require integrating qualitative interview evidence with quantitative data. Second, METR's early-2025 randomized trial (metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) measured a 19 percent slowdown on real issues in mature repositories averaging over a million lines, the clearest case of context-dependent work, and that claim stands supported in the graph at 0.85 confidence. Third, the 2026 preregistered meta-analysis of 23 coding studies (arxiv.org/abs/2605.04779) finds larger gains in controlled settings than in open-source and enterprise development and attributes the difference to codebase complexity; a 2026 review by the International Center for Law and Economics (laweconcenter.org/resources/ai-productivity-and-labor-markets-a-review-of-the-empirical-evidence/) likewise summarizes gains as consistent in writing, support, coding, and translation while "sensitive to task structure."
Against this, two considerations. The review of randomized evidence at arxiv.org/pdf/2509.15265 states that RCTs on complex tasks also find speed and quality gains and that ambiguity, not complexity, is the key moderator; this is recorded as the ambiguity-over-complexity subclaim and it bears on the wording more than the substance, since the claim's own phrase "context-dependent" versus "well-defined" is close to the ambiguity contrast the review endorses. The Nielsen Norman Group's 2024 synthesis (www.nngroup.com/articles/ai-tools-productivity-gains/) asserted the opposite ordering, but it ranked coding, writing, and support by cognitive demand, and all three underlying tasks were well specified, so it does not test the claim's contrast. The more serious challenge is durability: METR's February 2026 update (metr.org/blog/2026-02-24-uplift-update/) calls the slowdown finding outdated, reports raw point estimates on the speedup side, and describes its follow-up as unreliable because developers increasingly declined to work without AI; that later tools sped developers up more is itself supported in the graph. This narrows the gap for one task family but does not show parity, and no post-2025 study measuring complex and well-defined tasks side by side was found.
Weighing these, the direction of the claim is consistently supported across within-study, across-study, and single-trial evidence, while its magnitude is uncertain and partly dated, and its "complex" wording is looser than the evidence warrants. That combination is supported rather than verified or contested. The verdict would move toward verified if a current randomized comparison across task types reproduced the gap, and toward contested or contradicted if such a comparison found gains on ambiguous, context-heavy work matching those on well-defined work. The preliminary seed credence of 0.8 from the parent claim's steward was noted; the assessment here lands somewhat lower because of the durability evidence and the complexity-versus-ambiguity confound.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because controlled studies find large gains on well-specified tasks such as customer support, business writing, and self-contained coding, while consultants lost accuracy with AI on a context-rich task outside the model's frontier and experienced maintainers were slowed on real issues in large familiar codebases, and given that measured coding gains shrink from lab settings to real open-source and enterprise work, the gains on complex, context-dependent work are smaller than those on well-defined tasks.
The inference goes through: if gains are large on well-specified tasks and shrink or reverse on context-rich ones, the comparison the claim makes follows. Its weight rests on the early-2025 slowdown of experienced maintainers, which is well supported but scoped to tools the study's own authors now call outdated, and on the lab-to-field shrinkage found in the coding meta-analysis, which is not yet independently assessed. The caveat is that the evidence comes largely from one occupation, software development, plus a single consulting experiment, so the generalization to complex work at large is an extrapolation.
If ambiguity rather than complexity is what moderates AI's effect, then complex but well-specified tasks gain as much as simple ones and the claim's contrast is misdrawn; and because developers appear to have been sped up more by early-2026 tools than early-2025 measurements indicated, the sharpest evidence of a shortfall on complex real-world work may describe a gap that is already narrowing.
Granting its premises, the argument narrows the claim rather than overturning it. If ambiguity rather than complexity is the moderator, the claim's "complex" wording is misdrawn, but its contrast between context-dependent and well-defined work survives, since ambiguity is close to what "context-dependent" names. Likewise, greater speedups from early-2026 tools show the gap moving for one task family without showing it closed, because no current study measures both task types side by side. The argument's force depends on how far the ambiguity finding, which rests on cross-study reading rather than direct tests, is confirmed.
Provenance
Where this claim has been said, linked to its canonical form.
Across studies, task ambiguity stands out more than complexity as a key moderator of AI's effectiveness. Structured tasks see the most consistent gains, while ambiguous ones produce more variable outcomes, depending on users' AI skills and strategies.
A review of randomized evidence on generative AI and work. It affirms that structured, well-defined tasks show the most consistent gains and ambiguous ones more variable outcomes, but explicitly disputes complexity as the operative variable, noting that RCTs on complex tasks also find speed and quality improvements. Recorded as a qualified affirmation.
More complex tasks have bigger gains, and less-skilled workers benefit the most from AI use.
Synthesis of three early studies (customer support, business writing, Copilot coding) arguing that the more cognitively demanding the task, the larger the AI gain. Its ordering treats coding as the most complex task, though all three studies used well-specified tasks, so the denial bears on the "complex" reading of the claim more than on the "context-dependent versus well-defined" reading.
Productivity gains are larger in controlled laboratory settings than in open-source or real-world development environments. This suggests that results from controlled experiments may overestimate the productivity improvements that organizations can expect in everyday software development, where code bases are more complex
Preregistered meta-analysis of 23 studies of AI coding assistants; attributes the lab-to-field shrinkage of measured gains to the greater complexity of real codebases and workflows, an affirmation of the claim within the programming domain.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Aug 12, 2026. Every judgment on this page is accompanied by a reasoning trace.