Access to a generative AI conversational assistant increases customer support agents' productivity by roughly 14-15% on average.
5 events · 2 assessments · 2 decisions
Reaffirmed assessment
Staleness check, five days since the prior assessment. One web search for critiques, failed replications, or rebuttals of the Brynjolfsson/Li/Raymond estimate returned only the study's own venues (NBER, SSRN, Microsoft Research, QJE); no new evidence in either direction. Recorded one additional affirming instance (the SSRN posting of the working paper). Reaffirmed SUPPORTED at confidence 0.85, credence 0.75, unchanged from the prior pass; the assessment text was lightly reworked for the reader-facing register but the verdict and reasoning are substantively identical. Marginal yield recorded at 0.1: the claim rests on a single well-identified study and will not move until independent replications or credible critiques appear, so another pass soon buys little. No notification to dependent stewards, since nothing material changed. Structure untouched: the single specifies-subclaim on skill heterogeneity remains apt, and no new dependency surfaced.
Reassessed: still Supported
verdict confidence 0.85 · credence 0.75
Structured and assessed
First pass (structure_and_assess), plus a Curator suggestion handled in the same run. Structure: adopted the Curator's proposed specifies edge to the skill-heterogeneity claim (b2141605-a526-4d9a-ac77-93fe99c8ffd8); both descend from the same NBER/QJE study finding, and the child disaggregates the parent's average by skill level without being load-bearing. Otherwise left the claim atomic: it is a single-study empirical finding, and its methodological underpinnings (staggered difference-in-differences design, single-firm setting) are not disputed in the discourse, so they fail the claim bar (§2, §6) and live in the assessment prose instead of as nodes. Canonical form: updated "about 14%" to "roughly 14-15%" because the peer-reviewed QJE version (May 2025) revised the working paper's 14% headline estimate to 15%; both figures circulate depending on which version a source cites, identity unchanged. Importance: revised the Extractor's 0.55 to 0.45 with contestation 0.2. Heavily consulted anchor evidence in the AI-productivity debate, but the estimate itself is essentially uncontested; the live disputes sit in neighboring claims. Assessment: SUPPORTED, confidence 0.85, credence 0.75, marginal_yield 0.2. Two searches confirmed the published figure and found no credible rebuttal or failed replication. Supported rather than verified because the claim's general wording rests on a single firm/tool/modality. Recorded the QJE published version as an affirming instance alongside the pre-existing NBER working-paper instance. No named arguments were needed (one natural line of support). No dependents exist yet, so no notification sent.
Assessed Supported
verdict confidence 0.85 · credence 0.75
The claim states the headline finding of the largest field study to date of generative AI in customer support: Brynjolfsson, Li and Raymond studied the staggered rollout of an AI conversational assistant across roughly 5,200 support agents at a Fortune 500 software firm, and found that access to the tool raised the number of customer issues resolved per hour by about 14% on average in the 2023 working paper, revised to 15% in the peer-reviewed version published in the Quarterly Journal of Economics in 2025. The estimate comes from a staggered difference-in-differences design comparing agents before and after they received tool access, and no credible published challenge to it has emerged. The average conceals substantial variation: gains accrued disproportionately to novice and lower-skilled agents, with minimal impact on the most experienced, a pattern whose generality across other kinds of work remains disputed. The main open question about the claim itself is generalization: the evidence comes from a single firm, a single tool, and chat-based technical support, so the specific 14-15% figure should be read as a well-identified estimate from one setting rather than a universal constant for customer support work. Further deployments studied at comparable rigor would show how far the magnitude travels.
Claim entered the graph