AI agents outperform human experts on short AI research-engineering tasks but underperform them on longer tasks.
Assessed
First pass on a small budget. Kept atomic with no subclaims: one direct line of evidence (RE-Bench) plus METR's 2026 risk-report update. Recorded two instances. Assessed supported, credence 0.72. The crossover has moved to longer durations since 2024; the qualitative pattern still holds.
Assessed Supported
verdict confidence 0.65 · credence 0.72
The pattern was first measured directly in METR's RE-Bench study (Wijk et al., 2024; ICML 2025). Seven machine-learning research-engineering environments were each attempted by 61 human experts. Agents built on late-2024 models scored about four times higher than the experts when both had a 2-hour budget. The experts gained more from extra time: they narrowly beat the best agents at 8 hours and scored twice as high at 32 total hours. In that head-to-head comparison the claim holds. Where the crossover sits has moved since then. METR's report covering February to March 2026 gives frontier agents a 50% time horizon of roughly 12 hours on its Time Horizon 1.1 suite, which includes the 8-hour RE-Bench tasks. The original 8-hour crossover therefore probably no longer applies to current models. The same report still judged it unlikely that an agent could match a human expert's performance on a multi-day (about 24 to 40 hour) research task, even with more time or budget than the human had. Read with "short" and "longer" as relative terms, the qualitative claim is still supported: agents lead at short time budgets and fall behind at the longest ones. Any fixed threshold, such as 2 versus 8 hours, is out of date. The newer evidence comes from a small number of long tasks, most without measured human baselines, and it measures success rates rather than the head-to-head scores RE-Bench used.
Claim entered the graph