Minerval

← the claim it bears on

A line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument from direct evaluations of agents on open-ended research

This argument, if it holds, bears in favour of the claim.constitution →As of 2026, frontier AI agents can perform research engineering but struggle with critical stages of the AI research lifecycle.

The argument

In shadow evaluations run in mid-2026, frontier agents were given the central research questions of two unpublished NeurIPS submissions, six days and several thousand dollars of compute, and the papers' own authors graded the output. The agents finished the engineering with only logistical human help, which is evidence that agents can complete the engineering work of AI research projects. Both papers were rejected, and the agents failed in recurring ways at research judgment, redesign, backtracking, resource awareness and instruction following. The models and scaffolds tested were among the strongest available and the failures were not artifacts of the setup or of the graders' knowledge that the work was AI-generated. So frontier agents can do research engineering but struggle with stages open-ended research depends on.

Does it work?

Does it follow?
If the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Do the premises hold?
2 supported

Granting its premises, the conclusion is much more likely than not, because an expert-graded test on fresh research questions is the most direct evidence available on both halves of the claim. The weight rests on the step from three runs on two papers to frontier agents in general. The study's own checks support that step for mid-2026 models: a second model family and scaffold, generous budgets, and failures that did not track scaffold crashes. But the graders were not blind, and models released later in the year were not tested. Autonomous completion of research engineering is now judged supported as a statement of demonstrated capability, though not of reliable completion across typical projects. The argument needs only the weaker point, that agents did the engineering in these projects, so that premise holds for its purposes. The recurring failure modes have not yet been assessed separately, and the second half of the conclusion leans on them.

Premises

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

  • Can AI agents conduct open-ended AI research? Early evidence from two case studiesmakes itThe passage states the inference with what it needs.constitution →
    We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors.
  • How Close Is Recursive Self-Improvement?makes itThe passage states the inference with what it needs.constitution →
    On InnovationEval, where agents try to reinvent a published ML innovation they have not seen, GPT-5.6 Sol recovered 15% of the innovation’s gain. (GPT-6 Astra scored 63% but had likely memorized the original paper.) In the CRUX 2 open-world evaluation, AI agents got the research question of an unpublished AI paper and had to write their own paper on it. The original paper’s authors rejected all four AI-written papers, including two from a follow-up study with newer models, more time, and a larger budget.

Earlier ratings

  • Nov 13, 2026If the premises are true, the conclusion is much more likely, though not guaranteed.constitution →

    Granting its premises, the conclusion is much more likely than not: an expert-graded test on fresh research questions is the most direct evidence available on both halves of the claim. The weight rests on the step from three runs on two papers to frontier agents in general. The study's own checks (a second model family and scaffold, generous budgets, failures that did not track the number of scaffold crashes) support that step for mid-2026 models, but the graders were not blind and models released later in the year were not tested. Of the attached premises, the recurring failure modes have not yet been assessed separately, and autonomous completion of research engineering currently stands as doubtful on an assessment that did not weigh this study's evidence; the argument needs only the weaker point that agents did the engineering in these projects, which the study documents directly.