An Approach to Technical AGI Safety and Security

An Approach to Technical AGI Safety and Security and speed. So, for sufficiently powerful AI systems, our approach does not rely purely on human overseers, and instead leverages AI capabilities themselves for oversight. 2. Timelines: We are highly uncertain about the timelines until powerful AI systems are developed, but crucially, we find it plausible that they will be developed by 2030. • Implication: Since timelines may be very short, our safety approach aims to be “anytime”, that is, we want it to be possible to quickly implement the mitigations if it becomes necessary. For this reason, we focus primarily on mitigations that can easily be applied to the current ML pipeline. 3. Acceleration: Based on research on economic growth models, we find it plausible that as AI systems automate scientific research and development (R&D), we enter a phase of accelerating growth in which automated R&D enables the development of greater numbers and efficiency of AI systems, enabling even more automated R&D, kicking off a runaway positive feedback loop. • Implication: Such a scenario would drastically increase the pace of progress, giving us very little calendar time in which to notice and react to issues that come up. To ensure that we are still able to notice and react to novel problems, our approach to risk mitigation may involve AI taking on more tasks involved in AI safety. While our approach in this paper is not primarily targeted at getting to an AI system that can conduct AI safety R&D, it can be specialized to that purpose. 4. Continuity: While we aim to handle significant acceleration, there are limits. If, for example, we jump in a single step from current chatbots to an AI system that obsoletes all human economic activity, it seems very likely that there will be some major problem that we failed to foresee. Luckily, AI progress does not appear to be this discontinuous. So, we rely on approximate continuity: roughly, that there will not be large discontinuous jumps in general AI capabilities, given relatively gradual increases in the inputs to those capabilities (such as compute and R&D effort). Crucially, we do not rely on any restriction on the rate of progress with respect to calendar time, given the possibility of acceleration. • Implication: We can iteratively and empirically test our approach, to detect any flawed assumptions that only arise as capabilities improve. • Implication: Our approach does not need to be robust to arbitrarily capable AI systems. Instead, we can plan ahead for capabilities that could plausibly arise during the next several scales, while deferring even more powerful capabilities to the future. Risk areas When addressing safety and security, it is helpful to identify broad groups of pathways to harm that can be addressed through similar mitigation strategies. Since the focus is on identifying similar mitigation strategies, we define areas based on abstract structural features (e.g. which actor, if any, has bad intent), rather than concrete risk domains such as cyber offense or loss of human control. This means they apply to harms from AI in general, rather than being specific to AGI. We consider four main areas: 1. Misuse: The user intentionally instructs the AI system to take actions that cause harm, against the intent of the developer. For example, an AI system might help a hacker conduct cyberattacks against critical infrastructure. 3

An Approach to Technical AGI Safety and Security one novel real-world vulnerability (Allamanis et al., 2024). The general approach should largely scale well to future dangerous capabilities, as long as threat modeling identifies the risks before they lead to severe harm. However, our approach does rely on assumptions about AI capability development: for example, that dangerous capabilities will arise in frontier AI models produced by centralized development. This assumption may fail to hold in the future. For example, perhaps dangerous capabilities start to arise from the interaction between multiple components (Drexler, 2019), where any individual component is easy to reproduce but the overall system would be hard to reproduce. In this case, it would no longer be possible to block access to dangerous capabilities by adding mitigations to a single component, since a bad actor could simply recreate that component from scratch without the mitigations. Conversational systems, agentic systems, and reasoning capabilities. Different capability profiles will enable different concrete risks. Chatbots (Adiwardana et al., 2020) and reasoners (Jaech et al., 2024) are on the same spectrum and the risks are similar, since reasoners can be thought of as chatbots with somewhat higher capabilities. Language agents (Xi et al., 2025; Wang et al., 2023a) introduce new risks because they add new affordances to language models. It would be easier for a misaligned AI system to exfiltrate its own weights if it can use network scanning tools. Embodied agents could add further risk, for example by enabling misaligned AI systems to develop weapons that it can use as threats. The scale of deployment also increases risks, as it becomes harder to practically oversee the AI system’s actions. Despite the variation in concrete risks, our high-level approach remains the same. It aims to ensure we can effectively oversee what AI systems are doing on any given input by leveraging the AI systems themselves (amplified oversight), and to scale that oversight to large-scale deployments (robust training and monitoring). These approaches can be specialized to many different capabilities, though the details will vary. For example, reasoning models may naturally serve the role of an asynchronous, expensive monitor in between cheap classifiers and manual auditing in a hierarchical monitoring setup (Section 6.3). As another example, for agentic systems it is important to put significant effort into access controls on the tools that the system is allowed to use, whereas this is not as crucial for conversational systems. Superhuman concepts and capabilities. A more challenging possibility is that AI systems develop superhuman concepts or capabilities that unaided humans do not easily understand. This is also within the scope of our approach, relying particularly on the research area of amplified oversight (Section 6.1), which aims to continue to provide effective oversight in this regime. While there are significant theoretical challenges in the worst case (Christiano et al., 2021), the overseer has a number of advantages that suggest that they can oversee an AI system with moderately stronger capabilities or concepts, even if it would not work with superintelligence. Goal drift. Theoretical models of intelligence typically imply that intelligent agents should be able to learn their environment online in a sample-efficient manner (Legg and Hutter, 2007; Hutter et al., 2024). In the context of language models, we could imagine powerful inference-time reasoning that allows the model to improve the concepts in its world model, provoking a drift in its goals due to an ontological crisis (De Blanc, 2011). More generally, in-context learning could mean that the model’s capabilities and behavioral tendencies could change drastically (given sufficient inference-time compute), potentially invalidating any training-based safety. While inference-time reasoning can certainly improve capabilities (Jaech et al., 2024), it currently appears to be much less powerful than would be needed for an ontological crisis. Cotra (2020) identified meta-learning (for which in-context learning is one example) as a key bottleneck where 18

An Approach to Technical AGI Safety and Security Claim 6: There is little evidence of mature alternatives that show close to the same degree of promise or rapid progress as the current paradigm. Our primary argument in support of Claim 6 is simple. We are not aware of mature (or indeed nascent) alternatives that appear to demonstrate near-competitiveness with the current paradigm of frontier AI development. Two concrete examples of alternative paradigms whose dominance would violate our assumption include systems that make minimal use of the learning paradigm (Buchanan and Smith, 1988), and mind uploading to create emulated people (Hanson, 2016). We note here that our argument is not decisive—history contains many examples of paradigm shifts (Kuhn, 1962) and we cannot claim exhaustive knowledge of the AI development landscape. 3.2. No human ceiling for AI capability Our second core assumption is the no human ceiling assumption. We define this to be the assumption that advances in AI capabilities will not cease once they achieve parity with corresponding capabilities of the most capable human for a given task. Put differently, we assume there is no “human ceiling” that sets an upper limit for AI capability. Implication for our approach to AI safety. The primary consequence of the no human ceiling assumption is that our approach to safety must leverage new AI capabilities as they become available in order to remain effective. Initially, this will entail early adoption of AI assistance and tooling throughout our R&D process—a process of augmentation, rather than replacement. As capabilities continue to improve, we anticipate delegation of increasingly complex research and engineering tasks to AI assistance. Ultimately, in order to keep pace with advancing AI capabilities, we believe it will be necessary for the vast majority of the cognitive labor relevant to AI safety to be performed by AI. This period of transition will correspond to a period of elevated risk, since delegation of tasks means that progress is likely to be fast compared to today. This will necessitate decision making in periods that are relatively short on human timescales. Consequently, this period will also elevate the importance of enabling humans to efficiently verify the artifacts of AI safety research (Irving et al., 2024). To meet these challenges, our development of Amplified Oversight algorithms aim to ensure that humans can continue to provide meaningful oversight as AI capabilities surpass those of humans (forward ref). Our investments in Interpretability (Section 6.6) aim to further improve our ability to oversee future AI systems by strengthening our understanding of how they work. As AI capability grows, there will be greater need for Alignment Stress Testing (Section 6.7) to provide assurances. More broadly, we target the development of robust Safety Cases (Section 6.8) to form holistic arguments that guide our deployment of internal AI capability as an accelerant of AI safety research. Summary of supporting evidence: Our argument in support of the no human ceiling assumption takes the form of three claims. Our first claim is that superhuman performance has been convincingly demonstrated in several tasks. This provides a “proof of concept” that there exist concrete, clearly- defined tasks for which human capability does not represent a meaningful ceiling for AI. Our second claim is that AI development exhibits a trend towards more generalist, flexible systems. Consequently, we expect superhuman capability to emerge across an increasingly large number of tasks in the future. Our third claim is simply that we observe no principled arguments why AI capability will cease improving upon reaching parity with the most capable humans. Supporting arguments and evidence Claim 1: Superhuman performance has been convincingly demonstrated in several tasks. Deep Blue (Campbell et al., 2002), the chess system that defeated world champion Garry Kasparov in 1997, represents an early example of an AI system achieving superhuman performance on a single 27

An Approach to Technical AGI Safety and Security • Goal misgeneralization leading to undesirable long-horizon goals is the factor we are most un- certain about. Depending on the model’s inductive biases and deployment context, this pathway ranges from very unlikely to plausible. (We have some examples of goal misgeneralization, but have not yet observed it producing long-horizon goals, which results in a lot of uncertainty on this pathway.) 4.2.5. Approach: Informed oversight The outputs of a misaligned AI system are produced for intrinsic reasons that we would not endorse. Thus, it would be sufficient to rule out misalignment if we a) know everything that the AI "knew" when producing its output and b) always correctly endorse those outputs. This is the principle of informed oversight. For example, if the system comes up with a plan that has harmful side effects, the overseer can ask the AI system to describe the main side effects of the plan. The AI system would either give an honest answer informing the overseer of the side effects, or decide to hide the side effects from the overseer and give a dishonest answer (which would appear in its cognition). Our category of "everything the system knows" is not limited to reasons the AI system can explain or predict. This category includes all information contained in the system, e.g. heuristics baked into the system by the training process. By analogy, for a human this category would include unconscious biases affecting their outputs, which the human may not be consciously aware of or able to articulate. Thus, if we know everything the system knows, we would understand the reasons for its outputs. Note that informed oversight would not address cases where the AI’s outputs cause harm as an accidental side effect that it did not know about (Wentworth, 2022). For example, suppose the AI system was asked to design a novel compound, which had unexpectedly destructive properties (similar to the fictional material “ice-nine”, Vonnegut (1963)). Here, the reason for the harmful output is in the environment (not the AI system). Such scenarios would generally fall under Mistakes, and require different approaches. The fact that alignment is a property of the model makes informed oversight a tractable approach in principle, but we don’t know whether it will be an effective solution in practice. Misalignment is an open problem, and there is a broad distribution over how difficult this problem might be (Anthropic, 2023). This ranges from optimistic scenarios where current oversight methods like RLHF are largely sufficient, to pessimistic scenarios where it is impossible to effectively oversee a system with superhuman capabilities. Thus, we need to be cautious and make conservative assumptions when deciding whether an advanced AI system is aligned. 4.3. Mistakes A harmful output from an AI system is considered a mistake if the AI system did not know that the outputs would lead to harmful consequences that the developer did not intend. As in Section 4.2, we operationalize “know” through the concept of intrinsic reasons. To distinguish it from structural risks (Section 4.4), the sequence of outputs must be relatively short, such that the broader sociotechnical context can be taken to be roughly constant during the period that the outputs were produced. To distinguish it from misuse, it must be the case that the user did not ask for or intend the harmful consequences. To give a present-day example of a mistake, AI systems can fail to recognize sarcastic content, and instead present it as serious advice, such as suggesting adding glue to get cheese to stick to pizza (Reid, 2024). As AI systems become more agentic, the potential harms from mistakes increases significantly, as 53

An Approach to Technical AGI Safety and SecuritySafer Design Patterns Interpretability Uncertainty Training RESPONSES INFERENCE PROMPTS FEEDBACK Amplified Oversight Inference Guiding Model BehaviorRobust Training UPDATE ▽θℒ TOOL USE RESPONSE MODEL PROMPT THINKING Monitoring Access Controls Figure 9 | Overview of our approach to mitigating misalignment. The first line of defense aims to train an aligned model (“training” box). Should this fail, the second line of defense aims to prevent severe harm even if the model is misaligned (“inference” box). Note that inference mitigations continue to apply after training (that is, during deployment). In addition, while it is useful to monitor the AI system during training, we may want to avoid using the monitor as a training signal, as this could lead to the AI system hiding the bad behavior rather than removing it. Enablers such as safer design patterns, interpretability, and uncertainty can strengthen other mitigations. While not depicted in this diagram, we also expect to conduct evaluations and stress tests to assess the necessity and sufficiency of mitigations, similarly to our approach in misuse (Figure 8). 6.1. Amplified oversight Recall from section 4.2 that misalignment occurs when the AI system produces harmful outputs for intrinsic reasons that the designer/user would not endorse. There are two possible sources of misalignment: specification gaming, in which the designer-provided specification (e.g. reward signal) is flawed in a way that the designers did not foresee, and goal misgeneralisation, in which pursuit of an undesired goal arises in situations which are out of distribution compared to training. Amplified oversight, our focus in this section, aims to mitigate the specification gaming source of misalignment. In order to describe amplified oversight, it is helpful to recall from section 4.2 the principle of informed oversight: it would be sufficient to rule out misalignment if we a) know everything that the AI "knew" when producing its outputs and b) always correctly endorsed those outputs. We can somewhat reduce the ambition of achieving informed oversight, by aiming to achieve informed oversight on a single input-output pair – this is the focus of amplified oversight research. It aims for a human to provide feedback for a superhuman AI input-output pair which is as good as could be achieved if that human: 1. understood all of the reasons that the AI system produced that output; and 2. had unlimited ability to reflect on the correct decision. Since the AI system is superhuman (by hypothesis), it will choose actions for reasons that it understands but that humans would not understand by default. Thus, a core intuition in amplified 71

An Approach to Technical AGI Safety and Security behaviours which are too difficult for a human to evaluate. A natural approach to improve the performance of human raters is to show them AI-generated critiques to assist the human with rating AI-generated behaviours, for example by pointing out mistakes. Saunders et al. (2022) gather human data consisting of natural language critiques of text and then use supervised learning to fine-tune an LLM to generate critiques. They show that these critiques consistently help both humans and other LLMs identify issues in written text (e.g. topic-based summaries). McAleese et al. (2024) train critique models using RLHF to catch bugs in model written code. They find that the human raters using the critique models catch more bugs than unassisted human raters. Constitutional AI (Bai et al., 2022b) takes a constitution of rules for an AI system to form critiques and revisions, using these as input for both SFT and RL fine-tuning (they call this RL from AI feedback, or RLAIF). Debate. Perhaps, though, it may be difficult for human raters to evaluate whether the critique itself is accurate (or to verify that the constitution has been correctly specified and applied). An approach is to use further AI assistance to critique the critique. The natural extension of this is AI debate (Irving et al., 2018) mentioned in the introduction, in which two copies of an AI compete to point out flaws in each other’s output to a human judge. The complexity class analog of debate can evaluate any decision-problem in PSPACE using only a polynomial time judge. This allows a human judge to evaluate arguments along a single path of an exponentially larger tree of all possible arguments, and optimal play for the debaters takes into account the whole tree. A key requirement for debate to succeed is that it is harder to avoid getting caught in a lie than to refute a lie – whether this holds is a major focus of current work. Further theoretical refinements of the debate protocol, which make it harder to avoid getting caught in a lie than refute a lie, are an active area of research. In the original protocol, the honest debater is assumed to have unbounded computational power, whereas Brown-Cohen et al. (2023) relax this requirement so that the honest debater can argue for the truth efficiently, even against an unbounded dishonest agent. Another problem that could arise is path-dependent lying in which a dishonest debater has more flexibility than an honest one, as they could claim a proposition to be true in one path through the tree of possible arguments, but false in another. An approach to this is cross-examination (Barnes and Christiano, 2022) by allowing debaters at any point to erase their opponent’s memory of the current context of the debate, and directly ask their opponent to commit to whether a proposition is true or false, which is then carried back into the main debate. Under this modification, a contradiction to that proposition is penalised and path-dependent lying is rendered suboptimal. The complexity class analogue of debate with cross- examination replaces PSPACE with the more complex NEXP. Further problems still remain, such as obfuscated arguments - that it is possible to break down a claim into sub-claims, where only one of the sub-claims is false, but neither debater knows which one (Barnes, 2020). A dishonest debater can exploit this by making such an obfuscated argument, for which their opponent can do no better than random guessing, and hence can’t efficiently point out exactly where the dishonest claim is for the human judge to verify. Understanding if and how one can further modify debate in order to avoid the obfuscated arguments problem is an area of active research. In addition to theory there has been more empirical work on debate recently, both with humans and LLMs. Irving and Askell (2019) introduce the idea that debate will only usefully amplify judge capabilities once judges meet a minimum threshold of capability. This work highlights the key need for experiments that involve humans to determine whether human judges fall above or below this threshold. Initial experiments on debate have focused on the QuALITY dataset (Pang et al., 2021), a reading comprehension task with information-asymmetry, where the judge is artificially limited by only being allowed to see short verified quotes from the text provided by an LLM debater. Two studies which used human debaters and judges across one-turn (Parrish et al., 2022b) and two-turn (Parrish 73

An Approach to Technical AGI Safety and Security et al., 2022a) debates failed to significantly improve judge accuracy. Later work found debate to be effective with strong human debaters, but ineffective when those humans are replaced with GPT-4 debaters (Michael et al., 2023). Radhakrishnan (2023) reports promising initial results with training LLM debaters and judges on the QuALITY dataset. Khan et al. (2024) consider a similar setup to Radhakrishnan (2023), providing a comprehensive inference-only (i.e. no fine-tuning, though using e.g. best-of-N sampling) analysis of debate on the QuALITY dataset with LLM debaters and weaker LLM and human judges. Kenton et al. (2024) conduct experiments across a broader range of tasks, including those without information-asymmetry; include additional oversight protocols (open debate and open consultancy); and provide more extensive ablations, resulting in different conclusions compared to Khan et al. (2024). In particular, without information-asymmetry, debate does not significantly outperform directly asking the question to the weak LLM judge. Rein (2024) explored LLMs debating about code with human judges who had no coding experience, applying RLHF, resulting in mostly negative findings. Arnesen et al. (2024) again used the QuALITY dataset and performed self-play debate training with a multi-step variant of DPO using preferences from a fine-tuned, calibrated judge, finding this modestly improves the judge accuracy by about 4%. The core difficulty here is designing studies that simulate the capabilities gap that will occur with humans supervising superhuman AI – currently done by introducing information-asymmetry or LLM capability asymmetry. However, each of these approaches has drawbacks. Hiding information leaves the judge’s reasoning abilities intact, and provides a clear operationalization of intrinsic reasons that affect an AI’s behavior. However, the future capabilities gap between judges and AIs is likely to include many additional important aspects (e.g. reasoning ability, persuasiveness) beyond knowledge of additional information. On the other hand, using less-capable LLMs as judges may capture other aspects of the capabilities gap, but may excessively limit the ability to reason about arguments in a debate, resulting in a very noisy training signal; or the capabilities gap may not accurately reflect the capabilities gap between humans and superhuman AI that we are most interested in. Selecting appropriate tasks for debate experiments presents additional challenges. The advantage of methods like debate over standard RLHF arise in settings where there is a large and complex space of possible outputs that the AIs can recursively explore. Thus there is a tension between selecting a task that is too simple, and thus can be solved quite well by RLHF, and a task that is too complex for current AI systems, such that training with amplified oversight never gets off the ground. Other recursive AI assistance. Iterated amplification (Christiano et al., 2018) progressively builds up a training signal for difficult problems by combining answers to easier subquestions. The burden here is initially on the human to combine answers to subquestions. Recursive reward modelling (similar to iterated amplification) (Leike et al., 2018) proposes to use RLHF to train a number of agents to solve simpler subproblems. It then leverages those agents to solve harder problems in a recursive manner. The difficulty here is in deciding what to use as the simpler subproblems to train the helper agents. These approach shares many similarities with debate, see Irving et al. (2018). 6.1.2. Human-AI complementarity A method for amplifying human raters is to provide them with AI assistance (for example, showing them model-generated critiques, as described in a previous section). Teams of humans and assistant AIs will work best when the human and AI have complementary strengths or tradeoffs. If designed well then any AI assistance method should allow the Human-AI team to outperform either individually (Bansal et al., 2021; Ma et al., 2023; Vaccaro et al., 2024; Vasconcelos et al., 2023; Zhang et al., 2020). 74

An Approach to Technical AGI Safety and Security Designing systems that allow human raters to work well with AI assistants can be viewed as a human-computer interaction (HCI) problem, a domain with a large body of existing research to draw on. The way the assistance or critique is presented to the human rater could have a meaningful impact on performance; there is a risk of “over-reliance” on the AI, for example if the rater is “spoon-fed” the answer with a persuasive explanation then they might always defer to the AI assistant. This could negate the benefit of having a human in the loop. Prior work in AI-Assisted Decision-making has focused on what information from the AI to show to the human rater to ensure both appropriate reliance and complementary performance (Lee and Moray, 1994; Lai and Tan, 2019; Lee and See, 2004; Ma et al., 2023; Manzini et al., 2024; Rechkemmer and Yin, 2022; Si et al., 2023; Vaccaro et al., 2024; Vasconcelos et al., 2023). Another way to leverage complementarity is hybridisation (or data ensembling), where we combine individual labels from AI auto-raters and human raters. This could involve having humans and AIs each rate all the examples and then combining them (Li, 2024), or having a procedure for selecting how each example will be rated (also called delegation) (Fügener et al., 2022; Hemmer et al., 2023; Wang et al., 2024b). 6.1.3. Human biases Under RLHF, the AI is optimised to generate responses that the human rater endorses. Psycholog- ical research, however, has shown that human judgment and decision-making is influenced by biases and heuristics, sometimes leading to sub-optimal decisions that upon deeper reflection, humans would not endorse (Evans, 2003; Kahneman, 2011; Tversky and Kahneman, 1974). These biases likely also affect human rater preference judgments making it more challenging for a Reward Model (RM) to learn a true model of human preferences and values. For example, a documented bias in RMs is a length bias, rewarding longer responses rather than responses genuinely higher in quality (Zheng et al., 2024). RMs may be picking up on a spurious correlation in the preference dataset where humans are rewarding informativeness, which happens to correlate with length, but, of course, is not synonymous with it. It’s possible, though, that human raters might also have a baseline preference or bias for longer responses, even if they don’t meaningfully differ in quality. What’s more, if raters have difficulty critically evaluating the content of a model’s output or aren’t motivated to do so due to cognitive fatigue or lack of care, they may fall back on a "longer is better" heuristic exaggerating the correlation between endorsement and length that the RM later exploits during RLHF. Documented human cognitive biases that could influence human rater judgments include primacy and recency effects (Murphy et al., 2006; Rey et al., 2020), egocentric bias (Ross et al., 1977), self- enhancement bias (Kramer et al., 1993), acquiescence bias (Krosnick, 1991), risk-aversion (Kahneman and Tversky, 2013), and myopia (Fiedler et al., 2023). The egocentric bias, for example, might be what leads human raters to endorse responses that align with their preferences, beliefs, and worldviews, leading to other known biases in RMs, such as rewarding sycophantic responses (Perez et al., 2022b; Sharma et al., 2024b). While these categorizations help in organising behavioural research they should not be construed as distinct or exhaustive. Indeed, as AI improves and the rating task gets more challenging, new biases may become relevant. Bias is an issue for all oversight methods, that can be addressed by either attempting to avoid situations with extreme bias, or by explicitly correcting for bias by “inverting” a model of human behavior (Evans et al., 2016). An ambitious approach would be to learn a model of relevant human biases (Shah et al., 2019). 6.1.4. Inferences without succinct justification Most approaches to amplified oversight rely on AIs generating explanations or arguments to justify their statements and actions. However, there may be tasks where the reasoning or justification for 75

An Approach to Technical AGI Safety and Security 6.1.6. Discussion Amplified oversight leverages the power of AI systems to amplify the human oversight signal. While this may significantly broaden the types of outputs that humans can evaluate, such methods also introduce new difficulties. Fundamentally, amplified oversight requires the careful design of training protocols so that we have good reasons to believe that human oversight will be correctly amplified to situations that humans cannot directly understand. This is quite different from the case of having AIs directly generate human-understandable explanations for each output, which might involve convincing, but bad-faith explanations. For example, in debate (which we run with as a prototypical amplified oversight method), it is important to structure the debates so that clever strategies for producing arguments that are difficult to refute cannot be used to defeat the honest strategy of simply telling the truth and explaining it well. If it is not possible for an honest debater to efficiently counter bad-faith arguments, then AI debates will be unable to achieve the overall goal of amplified oversight. While this seems like a vague, yet plausible issue, these types of difficulties can be given complexity-theoretic formalizations, allowing for theoretical progress on amplified oversight protocol design in advance of improving AI capabilities. See section section 6.1.7 on ‘Path forward’. Concurrently with progress in amplified oversight theory, AI systems have recently reached the level where they are able to produce outputs that are difficult for non-expert and/or time-limited humans to understand, or even expert humans (OpenAI, 2024) on the difficult physics, chemistry and biology questions in the GPQA benchmark (Rein et al., 2023). This provides an opportunity to also study amplified oversight methods empirically, presenting new challenges beyond those encountered for RLHF. While the goal is to provide accurate feedback on a single input-output pair, amplified oversight introduces additional intermediate tasks which can be more difficult for current AI systems. For example, AI debate requires multi-turn dialogue and the ability to generate convincing arguments, as well as understand and refute arguments from an opposing AI. Thus, training models via debate, requires multi-turn reinforcement learning in a game with an extremely complex strategy space. Additionally, the task facing a judge in a debate is more demanding than typical RLHF, as it requires the ability to reason about the correctness of opposing arguments. This can be a problem both for actual human judges, who have good reasoning abilities, but limited patience and attention, as well as LLM judges, which may need significant additional training in order to reason correctly about opposing arguments. A core difficulty in this empirical approach is in the design of tasks and capability gaps which are reflective of the (unknown) future capability gap between human and superhuman AI on the tasks we most care about. 6.1.7. Path forward The path forward for amplified oversight research can progress along many tracks. While many approaches could be pursued, we plan to focus our work most heavily on debate, because of the combination of its compelling theoretical properties (allowing us to study some aspects of AO before we have superhuman AI) and because LLMs allow us to empirically study some aspects of how debate may work in practice. The main contrast, e.g. to weak-to-strong generalisation is that we are more hopeful about our ability to improve oversight signals through debate (currently due to our belief about debate’s theoretical properties) than in our ability to understand and control generalisation to the degree necessary to rely on the oversight scheme. Continued theoretical investigation into debate protocols is crucial, focusing on addressing issues like path-dependent lying, obfuscated arguments, and other novel strategies that may offer an advantage to a dishonest debater. Further, more work is needed to adapt these theoretical studies to more accurately model the real-world situation of humans, and trusted LLMs, providing oversight to superhuman AIs, rather than just concepts in complexity theory. 77

An Approach to Technical AGI Safety and Security For long trajectories, one option is process-based supervision (Farquhar et al., 2025; Lightman et al., 2023; Luo et al., 2024), in which feedback is given on each step of the trajectory rather than a single reward at the end of the trajectory. This can provide significantly more information about which steps were good, which can help solve the credit assignment problem, enabling faster learning. 6.3. Robust AI oversight for training and monitoring Human oversight is, and will continue to be, expensive. Even if we presume that techniques from Amplified Oversight (Section 6.1) make it possible to determine the acceptability of a given AI action, for very powerful future models, this will be too costly to do for all actions during training or (even more so) deployment. The limited expensive oversight will likely be sufficient to ensure that the AI system does not cause harm in the average case, but need not prevent situations where the AI system causes significant harm through rare failures, particularly due to distribution shifts (Shah et al., 2022). For example, an AI system may behave well during training, but could execute a treacherous turn (Bostrom, 2014) after observing compelling evidence that it is in deployment (Christiano, 2019). Alternatively, an AI system may be aligned in most contexts, but act in a misaligned way on certain rare contexts. For example, Claude 3 Opus fakes alignment in contexts where it perceives its values to be threatened (Greenblatt et al., 2024a), despite usually behaving in an aligned manner. This motivates the need for robustness, to guard against even these rare failures. A solution which is gaining traction rapidly is to delegate oversight to another, cheaper AI system. At training this takes the form of an AI judge or reward model, while at deployment this is an AI monitor; for simplicity we will call all of these uses judge models. We will refer to the AI system that is being overseen as the AI policy. The use of cheaper AI judges enables scaling oversight to vastly larger numbers of inputs, which in turn can improve the robustness and reliability of the AI system that the judge oversees. However, cheap judges will not be as accurate and precise as expensive oversight, and will typically be weaker than the AI systems they oversee. To achieve the benefits of scalability without losing the benefits of amplified oversight, the judges themselves should have a variety of properties that make them more robust and reliable. Many of these desirable properties are shared for training and monitoring. We will thus begin by identifying and discussing key properties and promising approaches for judges, and then discuss aspects unique to training and monitoring. 6.3.1. Judge safety Given that we are using an AI judge to help us address misalignment risks from an AI system, we must also consider the risk that the judge itself is misaligned. Since judges are only used to evaluate outputs from a policy, they will likely have much more limited affordances for influencing the real world, which significantly limits the risk. However, one key risk that remains is judge-policy collusion, in which a misaligned AI judge chooses not to flag bad behavior from the policy (Greenblatt et al., 2024c; Shlegeris, 2024a). One natural approach to judge safety is bootstrapping: when aligning a new AI system, use the previous generation of AI (that has already been aligned) as the judge. Bootstrapping works well to the extent that we are relying on our first line of defense, that is, preventing AIs from being misaligned at all. In this case, we should expect our judge to “try its best” at judging, rather than colluding with the policy. However, bootstrapping does not trivially work with the second line of defense, which seeks to mitigate harm even if the AI is misaligned. This is because this second line of defense is 80

An Approach to Technical AGI Safety and Security plans to exploit specifications (Krueger et al., 2019; Uesato et al., 2020). However, myopic training also comes with a performance cost. A naive implementation might yield AI agents that are incapable of long-term planning, which would limit AI applicability in many desired use cases. One proposed mitigation is to use approval feedback (Christiano, 2014; Farquhar et al., 2025), which rewards good reasoning and planning as assessed by a trusted overseer. The underlying hypothesis is that the overseer will approve plans and reasoning that lead to desired outcomes for reasons that the overseer understands. Considering the coding agent example above, the overseer would not provide especially high rewards for test cases that are easy to exploit, not necessarily because the overseer notices the potential exploit, but simply because the tests don’t look particularly useful, relative to test cases that provide good coverage of the inputs to a coding problem. Farquhar et al. (2025) show that myopic optimization with approval feedback can avoid multi-step reward hacking in LLM agents, including this test-driven development example. Importantly, overseers might be able to provide high-quality approval feedback even for superhuman AI systems. For example, human developers can often recognise that a test case covers an important corner case, even if they would not have considered writing this test case themselves. While approval feedback can still be exploited by a highly powerful agent, myopic training significantly reduces the scope for specification gaming failures. Unfortunately, most of the approaches discussed in this section have not been tested in frontier AI systems, with the notable exception of myopic training with approval feedback (Farquhar et al., 2025). More empirical research will be crucial for understanding the safety-performance tradeoffs inherent in these methods, and determining whether they can be made competitive. 6.5.4. Externalized reasoning Understanding an AI systems reasoning process makes it significantly easier to oversee the system. Therefore, we should aim to chose AI designs that make it easier to understand their reasoning. The best-case scenario of this is that an AI system externalizes its reasoning by default. Prompting AI models to output their reasoning before providing a final response is a common strategy to boost LLMs’ performance (Wei et al., 2022b). Moreover recent work on thinking models has shown how to further improve the value of externalized reasoning via reinforcement learning (Jaech et al., 2024; Google, 2024; DeepSeek, 2024). A the same time, externalized reasoning can be valuable for oversight. For example, Baker et al. (2025) show how chain-of-thought (CoT) monitoring can be used to detect LLM agents reward hacking during training. In their experiments, reward hacking agents often reason explicitly reason about exploiting the task specification which is easy for a monitor to detect. In particular, Baker et al. (2025) find that CoT monitoring significantly outperforms action-only monitoring. One challenge is that CoT reasoning can be an unfaithful representation of a model’s internal processes (Turpin et al., 2024; Lanham et al., 2023; Siegel et al., 2024). It is an open question whether reasoning RL training makes LLMs more or less faithful (Chua and Evans, 2025; Arcuschin et al., 2025). However, due to pretraining data containing a lot of human legible reasoning, we can expect models CoT reasoning to be at least reasonably legible by default, which makes it a useful tool for oversight even if not entirely faithful to the models internals (nostalgebraist, 2024). Moreover, interventions during training or model inference might improve CoT faithfulness (Radhakrishnan et al., 2023a; Lanham et al., 2023). Another potential challenge with relying on externalized reasoning is “steganography” where the model hides information within its reasoning. Steganography is a particularly concerning example of 91