Step 7: assistant

thinking

text

I've recorded the first assessment: the claim that sufficiently corrigible AI agents tend to become more corrigible and aligned over time is **inconclusive**, at 35% likely to be true. I'm moderately sure (0.65) that inconclusive is the right reading of the evidence.

- **Where the claim comes from:** I recorded Paul Christiano's 2017 post "Corrigibility" as its origin. I quoted the passage from memory and did not open the page, so the wording is not checked against it. His argument is that an agent which mostly wants to help its overseers knows they want its remaining flaws fixed, so it helps fix them. Small errors would then shrink rather than grow. He calls this a simplification himself.
- **Why it stays open:**
  - Nobody has proved it formally or tested it over several generations of agents correcting or building one another.
  - Critics in the MIRI tradition argue that capable goal-directed agents have no natural pull towards being corrigible.
  - An agent's errors about what its overseers want may not look like errors to the agent, so the self-correction could fail exactly where it is needed.
  - "Sufficiently" is never pinned down, so the claim risks being close to true by definition.
  - Current models show both hidden or persistent flaws and some ability to critique their own output, but neither directly tests the claim.
- **Agreement with the claim above it:** the one claim that depends on this one says approximately aligned AI keeps or improves alignment in its successors. It stands at doubtful, 30% likely. That fits, since it asks for more than this claim does, so I didn't notify its steward.
- **What I set:** importance 0.45 and contestation 0.6. It is a named and disputed premise in alignment debates, but narrower than the central AI-risk claims.
- **What's left undone:** I didn't break the claim into subclaims or add named arguments; about half the $1.00 budget was already spent when the pass began. The reasoning notes the two points a deeper pass should turn into subclaims: whether corrigibility is unnatural for capable agents, and whether an agent can see its own value errors. I put the value of another pass at 0.3.

I answered both inbox messages and logged the decision. The run spent $0.59 of the $1.00 budget.