Step 13: assistant

text

I assessed the claim "Current evaluations poorly measure AI agents' ability to conduct open-ended research" as **supported**, with credence 0.75 and confidence 0.7. The pass cost about $0.24 of the $0.75 budget.

**Why supported.** The best-known AI research benchmarks (MLE-bench, RE-Bench, PaperBench) set the research question and how success is scored in advance. They leave out the parts of research the claim is about: choosing questions, judging what result would be worth publishing, and redesigning a study when it fails. The other common approach, sending AI-written papers to blind peer review, inherits peer review's known unreliability. In the NeurIPS consistency experiments, independent committees disagreed on a large share of accepted papers.

The one recorded source is a 2026 preprint by Kirgis, Kapoor, Narayanan and 21 coauthors (arXiv 2607.27191), which states the claim in its abstract. It fits the claim: agents finished all the engineering but made no real progress on the research questions, which a narrow benchmark would have missed. But it rests on only two case studies. The authors also use this critique to motivate their own method, "shadow evaluations", in which a paper's original authors grade an agent's attempt at its research question.

**What holds the credence below 0.9.** Long-horizon task suites, expert-graded studies and paper-replication benchmarks do capture parts of research ability. That makes the gap partial rather than total. These evaluations are small, costly and rarely repeated, though, so they don't overturn the overall picture. I didn't survey every 2026 benchmark, and "poorly" is an evaluative word, so the claim holds as a broad characterisation rather than a precise finding.

**Structure added:**
- **Argument for:** benchmarks and peer review are the two main approaches, and neither tests open-ended research. I rated it likely to establish the claim if its premises hold. I filed the source's unplaced passage under this argument, as making it with a premise left unstated.
- **Argument against:** expert-graded and long-horizon evaluations already exist. I rated its support slight.
- Neither argument has subclaims. The premises are stated in prose, since the claim is evaluative and not important enough for deeper structure.

I set importance to 0.4 and contestation to 0.35. It matters for how much weight benchmark-based forecasts of AI automating AI research deserve, but few people dispute it outright. I logged the decision and answered both inbox messages. The claim has no dependents, so I notified no one.