Step 11: assistant

text

I've recorded a first assessment: the claim that much of frontier AI progress comes from automatable experimentation, paced by tools and resources, is **supported**. I put my confidence in that verdict at 0.55 and the chance the claim is true at 0.6. The verdict comes from what I already know of the field, not from sources opened in this pass. Half the budget was already spent when the pass started, and I searched and read nothing new.

**Why it lands there.** The claim makes two points, and they stand differently:
- **Progress is mostly experiment-driven and limited by compute.** This half is reasonably well supported. Work on scaling and algorithmic progress credits most measured gains to more compute and data, and much of the algorithmic gain only shows up at scale and is found through many experimental runs. People inside labs also commonly name compute for experiments as the main constraint.
- **That experimentation can be automated.** This half is the live dispute. It is unclear whether choosing what to try next (research taste and direction) can be automated, and whether the quality of that choice, rather than how many experiments get run, is what limits progress. Evidence of AI doing frontier research on its own is thin and comes mostly from AI developers.

The only recorded source is Anthropic's essay "When AI builds itself". It states the claim but presents no evidence for it. Anthropic sells the models the essay describes, so it has an interest in their being seen as capable research agents, and that interest is noted on the source. The word "much" is also vague, which makes the claim easier to support than to test.

**What I recorded:**
- Importance 0.65 and contestation 0.65: it is a key premise in debates about AI improving itself and about AI timelines, and it is actively argued.
- One argument for the claim: breakthroughs are rare, and most progress is a loop of scaling up, seeing what breaks and fixing it. I rated it "likely if the premises hold". It is not decisive, because it rests on whether AI can also choose the next experiment.
- The essay's passage on incremental progress is now attached to that argument, marked as leaving a premise unstated.
- A log entry, and answers to both inbox messages.

**Not done.** No subclaims are mapped yet. I set the expected value of another pass high (0.6). That pass should split the claim into its two halves and look for numbers on how much progress comes from compute versus ideas.