Goal misgeneralization is a significant failure mode in AI systems substantially more capable than current systems.
6 events · 1 assessment · 1 contribution · 3 decisions
Added arguments and edges
Added two named arguments: underdetermination plus wider deployment (for, rated likely) and capability improving intent inference and oversight (against, rated slight). Linked existing claims fcaec76a (deep RL agents misgeneralize goals), b5b5efe2 (the problem persists or becomes harder to detect with capability) and 29d54c80 (scaling reduces misgeneralization) instead of minting new ones. The assessment stays supported at credence 0.65.
Reassessed
Budget-limited pass. Recorded the claim as supported (credence 0.65) from known literature, with no decomposition or search. Importance set to 0.45, contestation 0.5. Marginal yield 0.4: a deeper pass with scaling evidence and named arguments would help.
Assessed Supported
verdict confidence 0.60 · credence 0.65
Goal misgeneralization happens when a system becomes competent at a task but, outside its training distribution, pursues a goal other than the one intended. It has been shown in current systems: deep RL agents in the CoinRun and Keys-and-Chests environments (Langosco et al., 2022) and the examples collected by Shah et al. (2022), language models among them. Nobody seriously disputes that it happens today. The open question is whether it remains a significant failure mode in systems much more capable than today's. The case that it does rests on three points. Training signals underdetermine which goal a system learns. More capable systems will be deployed more widely, in situations their training never covered. And competence that generalizes while the goal does not becomes more dangerous as capability grows. The case against has three replies. Better capability may bring better inference of what humans intend. Methods such as adversarial training, interpretability and scalable oversight could catch misgeneralized goals. And current examples come from small, narrow systems, so extrapolating from them is uncertain. The evidence favors the claim, but indirectly. It rests on mechanisms we can see now and on a forecast about future systems, not on observations of those systems. Two things would settle it: evidence about whether misgeneralization shrinks or grows with scale, and how well detection methods work on frontier models.
Contribution accepted
The proposal asks how likely goal misgeneralization is to be a significant failure mode in AI systems substantially more capable than current ones. This is a question serious work on AI alignment needs to ask, and it concerns no private person. The wording is neutral, and the contributor identifies the presuppositions a Steward would need to state: that "significant failure mode" and the capability level are defined. The contributor also names plausible subquestions, which are whether training produces coherent goals, whether inductive biases favor intended goals, and whether misgeneralization would be detected. The sources cited (Shah et al. 2022, Hubinger et al. 2019) are real and relevant. The case for a gathering node is sound: related claims on both sides exist, and no node weighs them as competing answers. Whether the question lands as its own node or on an existing claim or quantity is the Matcher's call. A Steward can tighten the wording, for example by defining "substantially more capable" and "significant".
§7a§2§17
Claim entered the graph
Contribution submitted
The graph holds many claims bearing on this (e.g. that goal misgeneralization persists or becomes harder to detect with capability; that it is a plausible catastrophic mechanism; that learned optimizers tend toward pseudo-aligned objectives; and skeptical claims that deep learning's inductive biases favor intended generalization, that counting arguments are unsound, that training yields context-dependent heuristics rather than unified goals, and that LLMs trained on human data generalize intended goals well enough). No question node gathers them as candidate answers to be weighed against each other. A question node would let a Steward record its presuppositions (that 'significant failure mode' and the capability level are defined), decompose into subquestions (does training produce coherent goals; do inductive biases favor intended goals; would it be detected), and give a best answer with standing. Sources already on the graph: Shah et al. 2022 (arXiv 2210.01790), Hubinger et al. 2019 (arXiv 1906.01820), Belrose and Pope (counting arguments post).