In evaluated long-horizon AI R&D tasks, frontier agents' strongest solutions mostly adapt or combine established techniques; methodological novelty is rare.
Structured and assessed
First pass. Read both sources. Created one argument (convergent evidence from two independent evaluations) and placed the Epoch passages on it; the second passage was deduplicated. Corrected the Epoch instance's passage so it is the one that states the novelty finding, and filled speakers and dates. No subclaims: the claim is a narrowly scoped empirical summary whose evidence is the two studies themselves. Importance set to 0.5, contestation 0.45. Assessed supported, credence 0.88. Generalization to internal lab systems is flagged as outside the claim's scope.
Assessed Supported
verdict confidence 0.80 · credence 0.88
The claim describes what published evaluations of AI agents on long-horizon AI research tasks have found, and two independent evaluations from 2026 bear it out. A systematic evaluation by Li and colleagues (August 2026) ran seven frontier models, among them GPT-5.5, Gemini-3.1-Pro, DeepSeek-V4-Pro and Kimi-K2.7-Code, mostly in a shared agent harness, on 36 long-horizon tasks spanning areas such as CUDA kernels and model development. Agents often reached competitive scores, but when LLM judges reviewed the best solution from each model on each task, only 3 of 252 counted as novel approaches. The authors conclude that agents behave "more like engineering optimizers than fully autonomous researchers." Epoch AI's InnovationEval (October 2026) took a different approach. It asked agents to invent a post-training method matching a recent human innovation, on-policy self-distillation, without having seen it. The uncontaminated models, Claude Fable 5 and GPT-5.6 Sol, spent thousands of dollars of GPU time and made little progress. The best discovery, a self-imitation loss close to existing ideas, fell short of even a "moderately interesting" result. Successor models such as GPT-6 Astra had memorized the target paper, and their gains came mostly from reproducing it or from hyperparameter tuning. The two studies differ in authors, models, tasks and how they judged novelty, and they agree. That makes the claim, as scoped to evaluated tasks, well supported. Some caveats limit how far it generalizes. Novelty was judged by LLM reviewers in one study and by a single author's expert comparison in the other. Benchmark tasks with fixed metrics and short budgets may favour the tuning of known methods. InnovationEval rests on one task and a few runs. Memorization can inflate scores without adding novelty, which actually strengthens the claim's reading. Neither study tests systems the leading labs held internally in late 2026, or agents given longer horizons or human guidance. The claim says nothing about those, and its truth does not carry over to them.
Updated claim instance
Instance 2434175c-ba28-40ab-ad1e-462a385ad6ae (https://epoch.ai/publications/innovationeval): was stance=affirms, confidence=0.85; set verbatimText="Here, the most noteworthy discovery from an uncontaminated model was GPT-5.6 Sol’s use of a self-imitation loss in a GRPO setting to derive signal from all-pass groups. Given its similarity to pre-existing ideas and weak performance, this would struggle to clear the bar of Moderately Interesting. Although Fable 5.1 scored higher, it also relied on straightforward applications of existing techniques, and would also struggle to clear the bar.", segmentId=null, speaker="David Owen (Epoch AI)", sourceDate="2026-10-07". The recorded passage was about lack of progress on metrics, not about novelty of solutions; replaced with the passage that states the claim's content directly.
Updated claim instance
Instance c79ec5d5-2231-4d01-95e8-5eb9b49f1e64 (https://arxiv.org/abs/2608.13417): was stance=affirms, confidence=0.8; set confidence=0.95, speaker="Yiwei Li et al.", sourceDate="2026-08-13", context="Abstract and introduction of a systematic evaluation of seven frontier models on 36 long-horizon AI R&D tasks; the introduction adds that only three of 252 best-seed solutions qualified as novel under an LLM-judge review protocol.". Read the abstract and introduction; the passage states the claim almost verbatim, so confidence raised and speaker/date filled.
Claim entered the graph