Step 17: assistant

text

I rated the claim **supported**, with 0.7 confidence in that rating and a 0.75 probability that the claim is true. I spent $0.41 of the $1.00 budget.

**What the claim rests on.** It comes from one 2026 preprint by Kirgis, Kapoor, Narayanan and coauthors (arXiv 2607.27191). I read past the abstract: the introduction and results, the robustness run, and the limitations and potential-biases sections.
- **The main runs:** in two six-day "shadow evaluations", agents built on Opus 4.8 worked on the research questions of two unpublished NeurIPS 2026 submissions. The papers' original authors graded the output and rejected both outright, scoring them 2/6 and 1/6 overall.
- **The robustness run:** a follow-up run with GPT-5.6 Sol in a different setup showed many of the same failures.
- **Openness:** the reviews, agent code and logs are public.

**Why not verified.**
- The base is small: two case studies, one robustness run, and a list of failure types the authors drew up by reading the logs, which they admit involved judgment.
- The core team says openly that it doubts AI will soon improve itself rapidly. Some coauthors read the same behaviour as failures of reasoning rather than of judgment.
- The budget failures went in opposite directions. The main runs left more than half the budget unspent, while the robustness run used it all up in about two days.
- The claim talks about AI agents in general and in the present tense. The study tested two models in mid-2026 and could not test one vendor's strongest model.

**Changes to the page.**
- I left the claim as a single node with no subclaims, since it rests on one line of evidence.
- I recorded my reading of the paper and corrected its entry's speaker, date and confidence.
- I dismissed the unplaced "robustness check" passage, because it belongs to that same evidence rather than being a separate argument.
- I set importance and contestation both to 0.4.

**Merge proposal.** I linked this claim as related to its near-duplicate, the version limited to "frontier" agents. The Curator still has to decide the merge and which wording survives. Both pages now carry the same verdict and probability, so merging them will not create a contradiction.

I answered all three inbox messages.