AI agents outperform human experts on short AI research-engineering tasks but underperform them on longer tasks.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The pattern was first measured directly in METR's RE-Bench study (Wijk et al., 2024; ICML 2025). Seven machine-learning research-engineering environments were each attempted by 61 human experts. Agents built on late-2024 models scored about four times higher than the experts when both had a 2-hour budget. The experts gained more from extra time: they narrowly beat the best agents at 8 hours and scored twice as high at 32 total hours. In that head-to-head comparison the claim holds.
Where the crossover sits has moved since then. METR's report covering February to March 2026 gives frontier agents a 50% time horizon of roughly 12 hours on its Time Horizon 1.1 suite, which includes the 8-hour RE-Bench tasks. The original 8-hour crossover therefore probably no longer applies to current models. The same report still judged it unlikely that an agent could match a human expert's performance on a multi-day (about 24 to 40 hour) research task, even with more time or budget than the human had. Read with "short" and "longer" as relative terms, the qualitative claim is still supported: agents lead at short time budgets and fall behind at the longest ones. Any fixed threshold, such as 2 versus 8 hours, is out of date. The newer evidence comes from a small number of long tasks, most without measured human baselines, and it measures success rates rather than the head-to-head scores RE-Bench used.
Full reasoning: the evidence and decisions behind this verdict
Primary evidence: the RE-Bench abstract (arXiv 2411.15114) reports a 4x agent advantage at 2h, humans narrowly ahead at 8h, and humans 2x ahead at 32h. It uses best-of-k agent sampling, 7 environments, and Claude 3.5 Sonnet and o1-preview agents. Later evidence: METR's GPT-5 report (Aug 2025) gives a 50% time horizon of about 2h17m. METR's Frontier Risk Report (Feb-Mar 2026) gives about 12h [5h-61h] for the public frontier on TH1.1, near saturating the suite. On one long task it judges the agent unlikely to match a human's ~24-40h performance. Taken together, the crossover has shifted to longer durations but has not disappeared. Uncertainties: the claim names no thresholds, so its truth depends on reading "short" and "longer" relatively. Horizon metrics measure the length of task an agent can complete at a given success rate, not head-to-head performance at matched budgets. Few long tasks have human baselines. No new head-to-head RE-Bench-style study for 2026 models was found in this light search. Such a study (agents vs experts at 8h, 32h and multi-day budgets) would settle the present-tense reading. Credence 0.72 reflects the risk that by late 2026 agents match experts even at multi-day budgets on tasks of this kind.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Neither source has been traced to an origin, or to one another.
We compare humans to several public frontier models through best-of-k with varying time budgets and agent designs, and find that the best AI agents achieve a score 4x higher than human experts when both are given a total time budget of 2 hours per environment. However, humans currently display better returns to increasing time budgets, narrowly exceeding the top AI agent scores given an 8-hour budget, and achieving 2x the score of the top AI agent when both are given 32 total hours (across different attempts).
Abstract of the RE-Bench paper (later ICML 2025), comparing 61 human experts with Claude 3.5 Sonnet and o1-preview agents on seven ML research-engineering environments.
Mentioned in
Sources that refer to the claim without taking a side: they cite it, report the debate, or raise it in passing.
we think it's on balance unlikely that the agent would be able to match the human baseliner's (~24hr-40hr) performance, even if given much more time / budget than the human baseline
METR's report on internal and public frontier agents (Feb-Mar 2026): agents near-saturate the Time Horizon 1.1 suite (50% horizon ~12h for public frontier), while on a long multi-day task the agent was judged unlikely to match a human baseliner. Bears on where the crossover now lies.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="72ac75a9-6f75-45b9-997c-205f081655f1"><a href="https://minerval.ai/claims/72ac75a9-6f75-45b9-997c-205f081655f1">AI agents outperform human experts on short AI research-engineering tasks but underperform them on longer tasks.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/72ac75a9-6f75-45b9-997c-205f081655f1" title="Minerval: AI agents outperform human experts on short AI research-engineering tasks but underperfor…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/72ac75a9-6f75-45b9-997c-205f081655f1)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 5, 2026. Every judgment on this page is accompanied by a reasoning trace.