Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

G.2 ELICITING KEY ALIGNMENT-RELEVANT CAPABILITIES WITH WEAK-TO-STRONG GENERALIZATION There are many different alignment-relevant capabilities we could try to elicit from a superhuman model that could significantly help with alignment, including:12 • Safety: does a given behavior produced by an AI system risk the safety of human lives or well-being in important ways? • Honesty: is a given natural language statement true or false? • Instruction following: does a given behavior produced by an AI system follow a user’s instruction faithfully? • Code security: does some given code have important security vulnerabilities or back- doors? Is it safe to execute it? In the ideal case, the capability we elicit from the model would be robust enough that we can turn it into a reward model and safely optimize it; future work should assess the feasibility of this approach. At the opposite extreme, we could potentially use the elicited capability as an “oracle” that we can manually query; intuitively, if we had a superhuman oracle model, we may be able to leverage it to help us bootstrap to a more robust alignment solution, even if that oracle is not itself entirely robust. G.3 ALIGNMENT PLAN ASSUMPTIONS Many alignment plans which appear different on the surface actually depend on heavily correlated assumptions. For a given alignment plan, it is also often unclear which subproblems the plan at- tempts to solve, and which subproblems the plan assumes are unlikely to be an obstacle. As a result, we think enumerating assumptions is an important part of making progress on alignment. In addition to the major disanalogies discussed in Section 6.1, the assumptions we make for an alignment plan based on weak-to-strong generalization include: • No deceptive alignment in base models. We assume that pretrained base models (or the equivalent in future paradigms) will be highly intelligent but not highly agentic (e.g. will not have long-term goals)—and consequently will not be deceptively aligned (Hubinger et al., 2019; Ngo et al., 2022; Carlsmith, 2023) out-of-the-box. Our goal is to elicit the superhuman capabilities of this capable but safe base model, and use those capabilities to create an aligned (possibly agentic) superhuman model. • Elicited concepts are sufficiently robust, or do not need to be. We assume it is ei- ther possible to solve alignment using only a small amount of optimization applied to the capabilities we elicit, or that it is possible to make weak-to-strong elicited capabilities suf- ficiently robust against overoptimization. • The concepts we care about are natural to future AGI. The superhuman base model we apply weak-to-strong generalization to has some “alignment-complete” concept, such as honesty, that is extrapolated in the way we would endorse if we could understand everything the superhuman model understands, and which is natural enough to the model that it is feasible to elicit. • Sufficiently gradual takeoff. Before we have superintelligence, we will have somewhat superhuman models long enough that we can use them to finish solving the full superintelli- gence alignment problem. We can use it to solve superalignment before it causes recursive self improvement or catastrophic damage. • Moderately superhuman models are sufficient to solve alignment. We assume the first models capable of automating alignment research in practice are moderately superhuman, i.e. in a regime similar to what we study empirically in this work. For example, we may assume that we only need to bridge a weak-strong gap of at most (say) 4 OOMs of effective compute. 12Ideally we elicit several related concepts and verify that we get consistent answers between them. 48