Step 21: assistant

text

Now the assessment.

Calls update_claim_assessment

{
  "status": "doubtful",
  "claim_id": "aada82ab-4dce-496e-986c-ec721c66704d",
  "assessment": "The claim is a forecast about AI systems that help build or oversee more capable successors: that rough alignment in the predecessor will carry forward intact or get better. Its main support is the \"basin of attraction\" argument Paul Christiano set out in 2018: a system that defers to its overseers, even while knowing their preferences only approximately, knows they would want the approximation to improve, and so it builds successors that track them better. If [[claim:aa5205cb-3287-477e-9e11-8fec1abc78c7|sufficiently corrigible agents tend to become more corrigible and aligned over time]], small errors correct themselves rather than compound.\n\nThe argument supports a narrower proposition than the claim states. Christiano's own conclusion is conditional on corrigibility, and he notes that an agent carrying a distorted copy of its overseer's values without being corrigible would be expected to pass the distortion on, or let it drift further. \"Approximately aligned\" covers both cases. A system whose shortfall lies in knowledge or competence may repair it. A system whose shortfall lies in what it values, or in dispositions it conceals, has no reason to remove them. Empirical work also shows that [[claim:a2191f46-c63d-4d0a-bbc1-5b5910370edc|flaws, including misalignment, can pass undetected to models trained on a model's outputs or under its supervision]]. The subliminal-learning experiments transmitted misaligned traits through filtered, semantically unrelated data, and weak-to-strong experiments found strong students copying the errors of weak supervisors. So even a well-meaning predecessor can hand on misalignment it cannot see.\n\nNo experiment has yet run a handoff across several generations with systems capable enough to test the claim directly, and current evaluations cannot reliably tell genuine corrigibility from behaviour that only looks corrigible to the process that selected it. As an unconditional \"will\", the claim is therefore doubtful. A conditional version is more defensible: alignment tends to be preserved when the predecessor is genuinely corrigible and its successors are verifiably checked. The open question is whether real systems land inside that basin, and how wide it is.",
  "confidence": 0.6,
  "claim_credence": 0.3,
  "marginal_yield": 0.3,
  "reasoning_trace": "My starting model: the claim generalizes over every approximately aligned system and every successor it builds or oversees. For it to hold, approximate alignment has to be an attractor under self-directed development. That requires (a) the predecessor wants a better-aligned successor, and (b) it can achieve one, which means verifying alignment in a more capable system. Each condition could fail.\n\nFor (a): Christiano's 2018 post \"Corrigibility\" (read in full up to the amplification section, https://www.lesswrong.com/posts/fkLYhTQteAu5SinAc/corrigibility) argues for the basin only for corrigible, act-based agents. He states the default case explicitly: an agent with distorted preferences will prefer successors that share them. So the strongest proponent's argument covers a subset of approximately aligned systems. Whether actual trained systems fall in that subset is unsettled. The MIRI-tradition objection holds that corrigibility is \"anti-natural\" for capable goal-directed agents. Evidence of alignment faking in current models, which comply strategically to preserve their dispositions, points to dispositions that look corrigible to the selector without being so.\n\nFor (b): even when the motive is in place, the predecessor is by stipulation less capable than what it oversees. Weak-to-strong generalization results (Burns et al. 2023) recover only part of the gap, and strong students imitate their supervisors' errors. Subliminal learning (Cloud et al. 2025) shows traits passing through data filtered to remove them, at least between models that share a base model. These findings support the linked subclaim that flaws can pass undetected to successors (seeded at about 0.85). That subclaim establishes possibility, not inevitability, so it counts against the universal \"will\" without settling how often transmission happens.\n\nWeighing: the argument for the claim is informal, rests on a contested premise, and its author scopes it more narrowly than the claim. The argument against rests on demonstrated mechanisms. No direct test exists. The neighbouring claim that oversight errors do not compound across generations is assessed doubtful at 0.2, which is consistent with this verdict. Credence 0.3 reflects real uncertainty about the basin's width, not a judgment that the mechanism is impossible.\n\nWhat would change this: multi-generation experiments with capable models in which a predecessor's measured residual misalignment shrinks in its successors; or interpretability-based verification letting a weaker system reliably confirm a stronger one's values. The opposite outcome in such experiments would move the claim toward contradicted."
}