Step 7: assistant

text

I've assessed the claim as **supported** (credence 0.72, confidence 0.65). That holds only with "short" and "longer" read as relative terms: the specific 2-hour versus 8-hour crossover from 2024 is probably out of date.

- **Original evidence:** METR's RE-Bench study (late 2024, ICML 2025) compared agents with 61 human experts on seven ML research-engineering environments. Agents scored about 4× higher than the experts at a 2-hour budget. The experts narrowly beat them at 8 hours and scored about twice as high at 32 total hours.
- **Newer evidence:** METR's report covering February to March 2026 puts frontier agents' 50% time horizon at roughly 12 hours, on a test suite that includes the 8-hour RE-Bench tasks. So agents have probably caught up at 8 hours. The same report still judged it unlikely that an agent could match a human expert on a multi-day research task of about 24 to 40 hours.
- **Gaps:**
  - The newer figures measure the length of task an agent can complete, not head-to-head scores against humans at the same time budget.
  - Few of the long tasks have measured human baselines.
  - My search was light, and it found no 2026 study pitting agents against experts directly.

A study comparing current agents with experts at 8-hour, 32-hour and multi-day budgets would settle whether the pattern still holds today.

I kept the claim as a single unit with no subclaims or named arguments, since it rests on one direct line of evidence. I recorded the RE-Bench paper as a source affirming the claim and the 2026 METR report as one mentioning it. I set importance at 0.35 and contestation at 0.3, logged the decision, and answered both inbox messages.

I didn't notify the one claim that depends on this one: it's about whether sustaining long tasks is the main bottleneck to automating AI research, and stands as doubtful. A first assessment that supports the pattern doesn't obviously bear on that verdict. A note to its Steward would be free if wanted. I used about $0.33 of the $0.75 budget.