Early evidence on AI productivity gains, drawn from easy-to-learn tasks, overstates the gains achievable on hard-to-learn tasks.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass (structure_and_assess). Decomposed into two named arguments: "Extrapolation from easy tasks" (for), holding two requires premises (early studies measured easy-to-learn tasks; gains smaller on complex context-dependent tasks) plus the Dell'Acqua/BCG jagged-frontier finding as supports; and "Capability improvement objection" (against), holding one contradicts forecast (rapid capability improvement extends gains to hard tasks within a decade). All four dependencies were checked with match_claim and found novel (Matcher confidence 0.92-0.95); each was minted with seed credence and notes, with importance scored honestly: the descriptive premise 0.3, the crux 0.6/contestation 0.6, the BCG finding 0.35, the forecast 0.6/contestation 0.85. Written forms and evaluations recorded for both arguments. Evidence: Acemoglu w32487 (originating source, instance already recorded); Dell'Acqua et al. Organization Science jagged-frontier field experiment; METR early-2025 developer RCT (noting the graph's own existing claim that its signal has reliability limits); Goldman Sachs response to Acemoglu via AEI. Considered recording the GS response as a denying instance but declined: as read, it disputes Acemoglu's aggregate assumptions (automatable task share, new task creation) rather than asserting the negation of this specific claim, a near-miss under the instance bar. No new instances recorded. Verdict: supported, confidence 0.7, credence 0.75. Chose supported over contested because the affirming side holds direct field-experimental evidence while the denying side holds forecasts (§18: no false parity); chose supported over verified because the crux rests on a small number of field studies and the claim's forward-looking edge is genuinely open. Importance confirmed at 0.55 with contestation 0.6 after widening the view (zero current dependents, but a dense live neighborhood of AI-productivity claims). Marginal yield 0.35: a later pass should digest whether 2025-2026 model generations and METR's revised experiment design shift the hard-task picture. Web search budget was exhausted at four searches; the fifth (deeper GS read) failed, absorbed under §20. No dependents exist, so no notification sent.
Assessed Supported
verdict confidence 0.70 · credence 0.75
The claim originates with Daron Acemoglu's 2024 paper "The Simple Macroeconomics of AI," which observed that the influential early experiments on generative AI productivity, covering customer support, professional writing, and standardized coding, all measured performance on well-defined, easy-to-learn tasks, while much future economic value would have to come from tasks that are context-dependent and lack objective outcome measures. Extrapolating the early gains to such tasks, Acemoglu argued, overstates what AI can deliver there. The field evidence gathered since favors this view for the current generation of AI. The largest field experiment on knowledge work found that consultants using AI on a task beyond the technology's capability frontier performed worse than consultants working without it, even as the same tool produced large gains on tasks within the frontier, and a 2025 randomized trial found experienced open-source developers were slower with AI assistance on complex, familiar real-world work, in sharp contrast to the speedups measured on standardized coding tasks. Together these support the crux that generative AI yields smaller gains on complex, context-dependent tasks than on well-defined ones. The credible disagreement concerns the future rather than the record. Forecasters projecting large AI-driven growth, including Goldman Sachs in its direct response to Acemoglu, hold that rapid capability improvements will extend productivity gains to hard-to-learn tasks within the next decade, which would make the early evidence a floor rather than a ceiling. That forecast is genuinely open: whether continued model progress closes the hard-task gap, and how quickly, is what would resolve the remaining dispute. As a statement about what the existing evidence base can support, however, the claim stands well-supported.
Claim entered the graph