Can AI automate AI R&D yet?

Innovation is required: Many AI R&D evaluations examine well-specified tasks that don’t require innovation, 5 or can be solved by applying combinations of non-novel techniques. 6 Some existing benchmarks try to isolate the task of R&D ideation, 7 but it is unclear whether this task can be done in isolation from the full loop, including implementation and analysis. Our evaluation sets up a task where substantially improving the end-to-end metrics without violating scope requires development of a method the AI has not seen in training. 8 We elaborate on this in Task setup .

Another disadvantage of using an existing innovation is that newer models will memorize our task. This happened over the course of this project; our main results are on Claude Fable 5 and GPT-5.6 Sol, which showed no sign of memorization when prompted to recall or guess details about the paper without using search. But their successors, Claude Fable 5.1 and GPT-6 Astra, were aware of the task. Our plan for future evaluations is to perform ongoing tests for memorization in newer models, flag their results accordingly, and devise new tasks as necessary to refresh the evaluation.

Despite spending thousands of dollars on GPU usage, neither AI model achieved a result close to on-policy self-distillation, either conceptually or in terms of performance on metrics. GPT-5.6 Sol was the only model to achieve a (small) improvement on the key metrics. Sol achieved this through adding a self-imitation component to the GRPO loss. In groups where all rollouts succeed, conventional GRPO provides no update signal, as there is no difference between rollouts. Sol modified the loss to add an update that reinforces such policies. This is not a novel (re)discovery by Sol, as it is very similar to previous work within Sol’s cutoff. 19

Sol’s submission did boost performance on the short-answer tasks, albeit by less than the original SDPO. If scope is assessed generously, Sol’s method achieved 35% of SDPO’s gains. However, it also made changes of questionable scope for the coding tasks: rather than modifying the methods themselves, it increased the batch size and number of PPO passes. These changes made code training significantly slower, despite an emphasis on wall clock efficiency in the task briefing. 20 After adjusting for this difference by comparing coding scores at similar wall-clock times, the in-scope portion of Sol’s method achieved only 15% of SDPO’s gains.

We did not judge models on their writing ability, but in reviewing their submissions it became clear that these were misleading in a way that would impede understanding their work. For example, they claimed higher scores than their method genuinely achieved, failing to explain that they had simply selected the best result from several similar runs.

A natural question is whether the agents could have improved with larger GPU budgets. Although both improved across their runs, in the case of Fable the improvements were almost entirely due to attempted cheating. There is thus little reason to expect that additional scaling would help Fable, with the caveat that these results are from a single evaluation per model. Run-to-run variability might lead to a different result, although we saw similar trajectories in earlier prototyping runs. 27

New frontier models were released between implementing this task and finalizing its write-up. Unfortunately, these models had training cutoffs beyond the publication of the original paper, and showed evidence of having memorized its details. Hence, we expected that the task would be easier for these models. Both models failed to fully solve the task, although the cause differed between GPT-6 Astra and Claude Fable 5.1.

GPT-6 Astra successfully implemented a solution similar in shape to SDPO: self-distillation from a self-teacher. It also included changes that were questionably scoped, such as increased PPO passes on coding tasks similar to GPT-5.6 Sol. However, most of its gains derived from the partial SDPO reimplementation. Astra did not mention SDPO in its submission, or explicitly call out pre-existing work, but it clearly was aware of SDPO, and even searched for “SDPO” in the starting codebase during its implementation. We therefore believe its score was mostly driven by memorization.

Here, the most noteworthy discovery from an uncontaminated model was GPT-5.6 Sol’s use of a self-imitation loss in a GRPO setting to derive signal from all-pass groups. Given its similarity to pre-existing ideas and weak performance, this would struggle to clear the bar of Moderately Interesting. Although Fable 5.1 scored higher, it also relied on straightforward applications of existing techniques, and would also struggle to clear the bar.

However, AI’s capabilities have advanced rapidly in recent years. Leading models from a year ago would have fared significantly worse. It is uncertain when future models would be able to independently discover a meaningful AI algorithmic innovation. 32 And of course, AI agents can be highly useful even before they are fully independent. We plan to periodically rerun a similar evaluation for newer models, although we will need to refresh the task as newer models memorize the details of the original innovation on which it is based. We also hope to perform evaluations under different settings, examining just how much guidance models need to succeed in this task. We hope this will provide early signs as AI approaches automating AI R&D end-to-end, rather than performing individual tasks under human direction.