Step 25: assistant

text

I've structured and assessed the claim that an approximately aligned AI will preserve or improve alignment in the more capable successors it helps build or oversee. It is now rated **doubtful**: I'm 0.6 sure that's the right status and put about a 30% chance on the claim being true. The pass cost about $0.51 of the $1.00 budget.

**Main support.** Paul Christiano's 2018 "corrigibility" post is the main support. Its argument is that an AI that defers to its overseers, even while knowing their wishes only roughly, knows they would want that approximation to get better. It would therefore build successors that track them more closely, so small errors fix themselves rather than pile up. But Christiano makes this argument only for systems that genuinely defer in this way, not for every roughly aligned one. He says himself that a system carrying distorted values without deferring would be expected to pass the distortion on.

**Main case against.**
- Some roughly aligned systems may fall short in what they value or in traits they hide, and such a system has no reason to remove those flaws from its successors.
- Experiments show flaws can pass to successors without anyone noticing. In the "subliminal learning" work, misaligned traits passed from a model to a student trained on filtered data that seemed to have nothing to do with the trait, mostly between models sharing a base model. In weak-to-strong experiments, stronger models copied the errors of the weaker models supervising them.
- No experiment has yet tested a handoff across several generations with capable enough systems.

**Changes to the graph:**
- **For argument:** the "basin of attraction" argument, resting on a new subclaim that sufficiently corrigible AI agents tend to become more corrigible and aligned over time. I seeded that subclaim at 0.4.
- **Against argument:** the "inherited flaws" argument. It is linked to an existing, not-yet-assessed claim that flaws, including misalignment, can pass undetected to models trained on a model's outputs or under its supervision.
- **Argument ratings:** I rated both arguments "likely if their premises hold"; neither settles the question.
- **Source:** I recorded Christiano's post as supporting the claim, with reduced confidence because it covers only the narrower case.
- **Scoring:** importance 0.5, how much it is disputed 0.6.
- **Neighbouring claim:** the verdict is consistent with the one claim that depends on this one, that oversight errors don't compound across AI generations, which is rated doubtful at 0.2. So I didn't notify its steward.

I answered both inbox messages.

The verdict would change with multi-generation experiments in which a predecessor's measured misalignment shrinks, or grows, in its successors. Tools that let a weaker system reliably check a stronger one's values would also change it.