Step 21: assistant

thinking

text

I've assessed the claim as **supported**, with 88% credence that it's true and 0.8 confidence in that status. The two independent evaluations agree, but the claim only covers the tasks those studies actually tested.

**What each study measured**
- **Li et al., "Beyond Final Scores" (arXiv preprint, 13 Aug 2026):** seven frontier models, including GPT-5.5, Gemini-3.1-Pro, DeepSeek-V4-Pro and Kimi-K2.7-Code, ran 36 long-horizon tasks in areas such as CUDA kernels and model development. Most runs used a shared harness, with some native harnesses compared. Models often reached competitive scores. But AI reviewers judged only 3 of the 252 best solutions (one per model per task) to be novel approaches. The authors call the agents "engineering optimizers," not autonomous researchers.
- **Epoch AI's InnovationEval (David Owen, 7 Oct 2026):** a single task, run only a few times. Agents had to invent a post-training method matching a recent human one (on-policy self-distillation) without having seen it.
  - Claude Fable 5 and GPT-5.6 Sol had not memorized the target paper. They made little progress despite thousands of dollars of GPU time. Their best idea was close to existing work and fell below the "moderately interesting" notability tier Epoch borrowed from its FrontierMath: Open Problems rankings.
  - GPT-6 Astra and Claude Fable 5.1 had memorized the paper. Astra's score came mostly from partly reproducing the method; Fable 5.1's came mostly from tuning settings.

**Memorisation and task selection**
- Memorisation inflated scores without producing anything new, so it strengthens the claim rather than weakening it.
- Fixed-metric tasks with short budgets may favour tuning known methods, which is why the claim is limited to the tasks evaluated.
- Novelty was judged by AI reviewers in one study and by one author's expert comparison in the other. I did not check the details of the AI review method.

**Internal lab systems in late 2026**
Neither study tests systems the leading labs held internally in late 2026, or agents given longer runs or human guidance. The assessment says the claim doesn't carry over to those.

**Changes to the claim's record**
- **Argument:** I added one argument, "two independent evaluations", rated "likely" (supports but doesn't prove) and not decisive. The two Epoch passages waiting to be placed were near-duplicates: one went on the argument and the system dismissed the other.
- **Sources:** the recorded Epoch passage was about lack of progress on the metrics, not novelty, so I replaced it with the passage that states the novelty finding. I also filled in speakers and dates for both sources.
- **Importance and contestation:** set to 0.5 and 0.45.
- **Messages:** I answered both inbox messages.

I added no subclaims and searched no other evaluations, such as RE-Bench or ResearchGym, so I judged that another pass would add only a little. The pass used about $0.49 of the $1.00 budget.