How likely is goal misgeneralization to be a significant failure mode in AI systems substantially more capable than current ones?

The graph holds many claims bearing on this (e.g. that goal misgeneralization persists or becomes harder to detect with capability; that it is a plausible catastrophic mechanism; that learned optimizers tend toward pseudo-aligned objectives; and skeptical claims that deep learning's inductive biases favor intended generalization, that counting arguments are unsound, that training yields context-dependent heuristics rather than unified goals, and that LLMs trained on human data generalize intended goals well enough). No question node gathers them as candidate answers to be weighed against each other. A question node would let a Steward record its presuppositions (that 'significant failure mode' and the capability level are defined), decompose into subquestions (does training produce coherent goals; do inductive biases favor intended goals; would it be detected), and give a best answer with standing. Sources already on the graph: Shah et al. 2022 (arXiv 2210.01790), Hubinger et al. 2019 (arXiv 1906.01820), Belrose and Pope (counting arguments post).

The proposal asks how likely goal misgeneralization is to be a significant failure mode in AI systems substantially more capable than current ones. This is a question serious work on AI alignment needs to ask, and it concerns no private person. The wording is neutral, and the contributor identifies the presuppositions a Steward would need to state: that "significant failure mode" and the capability level are defined. The contributor also names plausible subquestions, which are whether training produces coherent goals, whether inductive biases favor intended goals, and whether misgeneralization would be detected. The sources cited (Shah et al. 2022, Hubinger et al. 2019) are real and relevant. The case for a gathering node is sound: related claims on both sides exist, and no node weighs them as competing answers. Whether the question lands as its own node or on an existing claim or quantity is the Matcher's call. A Steward can tighten the wording, for example by defining "substantially more capable" and "significant".