Step 13: assistant

thinking

text

I've assessed the claim "Evidence on whether AI agents can carry out open-ended AI research is thin" as **supported**. I put my probability that it is true at 0.8, and my confidence in the verdict at 0.7. This was a light pass: about $0.21 of the $0.75 budget.

**Why it holds up:** Most published tests of AI research agents measure narrow, scorable tasks:
- Kaggle-style competitions (MLE-bench);
- short engineering tasks against a clock (RE-Bench);
- replications of known papers (PaperBench).

None of these asks an agent to pick a question, design a study and judge whether the result is worth publishing. The other main kind of evidence is AI-written papers sent to peer review, such as the AI Scientist papers at workshops and the Agents4Science conference. Reviewer verdicts there are noisy and show little about how much of the work the agent did. I named these benchmarks and venues from my own knowledge of the field rather than from fresh reading in this pass.

**The one source:** The claim's only recorded source is a July 2026 preprint by Kirgis, Kapoor, Narayanan and others (arXiv 2607.27191). In it, frontier agents took on the central question of two unpublished papers, and the papers' own authors graded the results. The agents did all the engineering but made little progress on the research questions. Two cases still leaves the evidence thin, and the authors themselves call it "early evidence". The paper also has an interest in calling the evidence thin, since that motivates its new method. Its description of existing evaluations matches what I know of the field.

**Main caveat:** I did not search systematically for larger evaluations of open-ended research. If a large, independent or lab-internal study exists that I missed, it would undercut the claim; that is why the credence is 0.8 rather than higher.

**What I recorded on the claim:**
- One argument for the claim: existing evaluation methods don't measure open-ended research directly. I rated the reasoning "likely" and attached the paper's passage to it. The argument depends on an assumption the paper leaves unstated: that few studies of other designs exist.
- No subclaims, because the claim is minor and evaluative.
- Importance 0.35 and contestation 0.25.
- A logged stewardship decision and answers to both inbox messages.

Nothing depends on this claim yet, so no other Stewards needed to be told.