Step 15: assistant

text

I've assessed the claim that AI systems are improving at choosing next steps in AI research as **supported**: I put an 85% chance on it being true, with 75% confidence in that status. There are no other claims that depend on this one, so I sent no notifications.

**The evidence:** Two tests, run by different groups with different methods, point the same way.
- **Anthropic's internal test.** In its report "When AI builds itself", Anthropic took 129 moments from its researchers' real sessions where the researcher went off track. It asked models what to do next, and a separate Claude, which knew how each session ended, judged the answers. Models' proposals beat the researcher's choice 51% of the time with Opus 4.5 (November 2025) and 64% with Mythos Preview (April 2026).
- **Limits of that test.** The data isn't published and nobody outside Anthropic has examined it. There are only two data points. The moments were picked because the human choice was poor, so the test compares models with each other, not with human judgment in general; the report says this itself. Anthropic also sells these models, which I've noted as its stake.
- **TasteVal (Jaffe and Sherburn, October 2026 preprint).** Here the model chooses the experiments and a fixed coding agent runs them, which separates research judgment from coding ability. Across 20 models from 2023 to 2026, models got more efficient relative to human experts, and the trend has sped up since late 2025. The best model, Opus 5.5, beat the expert baseline. I read only the abstract; the tasks are unreleased and I didn't check the authors' affiliations.

**Why not "verified":** each test measures a narrow slice of research judgment, much of it can't be checked from outside, and having one AI model grade others could skew the results. Neither test covers the broadest form of the skill, deciding which problems are worth working on, where even Anthropic says humans still lead. A search also turned up several papers on predicting which research ideas will succeed that seem to show similar gains; I didn't read them in full and gave them little weight.

**What I recorded:**
- I gave the claim one argument for it: later models score higher on tests built to isolate next-step choice. I rated it "likely if its premises hold", not decisive.
- I placed the Anthropic passage that was left unplaced at intake on that argument, and added TasteVal as a second, independent source.
- I left the claim without subclaims and kept its importance at 0.55, setting how disputed it is at 0.45. That some improvement is happening is less disputed than how fast or how far it goes.

I logged the decision and answered both inbox messages. The run used about $0.43 of its $1.00 budget. I rated the gain from another pass at 0.3; reading TasteVal's full methods and the idea-forecasting papers would be the obvious next step.