Step 8: tool results

read_arxiv_source

{
  "file": "06-addressing-misalignment/robust-training.tex",
  "arxiv_id": "2504.01849",
  "max_chars": 13000
}

{"ok":true,"arxiv_id":"2504.01849","file":"06-addressing-misalignment/robust-training.tex","kind":"text","text":"\\subsection{Robust AI oversight for training and monitoring} \\label{sec:robust-training-and-monitoring}\n\nHuman oversight is, and will continue to be, expensive. Even if we presume that techniques from Amplified Oversight (\\Cref{sec:amplified-oversight}) make it possible to determine the acceptability of a given AI action, for very powerful future models, this will be too costly to do for all actions during training or (even more so) deployment.\n\nThe limited expensive oversight will likely be sufficient to ensure that the AI system does not cause harm in the average case, but need not prevent situations where the AI system causes significant harm through \\textit{rare failures}, particularly due to distribution shifts~\\citep{shah2022goal}. For example, an AI system may behave well during training, but could execute a treacherous turn~\\citep{bostrom2014superintelligence} after observing compelling evidence that it is in deployment~\\citep{christiano2019worst}. Alternatively, an AI system may be aligned in most contexts, but act in a misaligned way on certain rare contexts. For example, Claude 3 Opus fakes alignment in contexts where it perceives its values to be threatened~\\citep{greenblatt2024alignment}, despite usually behaving in an aligned manner.\n\nThis motivates the need for \\textit{robustness}, to guard against even these rare failures. A solution which is gaining traction rapidly is to delegate oversight to another, cheaper AI system. At training this takes the form of an AI \\textit{judge} or \\textit{reward model}, while at deployment this is an AI \\textit{monitor}; for simplicity we will call all of these uses judge models. We will refer to the AI system that is being overseen as the AI \\textit{policy}.\n\nThe use of cheaper AI judges enables scaling oversight to vastly larger numbers of inputs, which in turn can improve the robustness and reliability of the AI system that the judge oversees. However, cheap judges will not be as accurate and precise as expensive oversight, and will typically be weaker than the AI systems they oversee. To achieve the benefits of scalability without losing the benefits of amplified oversight, the judges themselves should have a variety of properties that make them more robust and reliable. Many of these desirable properties are shared for training and monitoring. We will thus begin by identifying and discussing key properties and promising approaches for judges, and then discuss aspects unique to training and monitoring.\n\n\n\n\\subsubsection{Judge safety} \\label{sec:judge-safety}\n\nGiven that we are using an AI judge to help us address misalignment risks from an AI system, we must also consider the risk that the judge itself is misaligned. Since judges are only used to evaluate outputs from a policy, they will likely have much more limited affordances for influencing the real world, which significantly limits the risk. However, one key risk that remains is \\textit{judge-policy collusion}, in which a misaligned AI judge chooses not to flag bad behavior from the policy~\\citep{greenblatt2024control, shlegeris2024untrusted}.\n\nOne natural approach to judge safety is \\textit{bootstrapping}: when aligning a new AI system, use the previous generation of AI (that has already been aligned) as the judge. Bootstrapping works well to the extent that we are relying on our first line of defense, that is, preventing AIs from being misaligned at all. In this case, we should expect our judge to ``try its best'' at judging, rather than colluding with the policy. However, bootstrapping does not trivially work with the second line of defense, which seeks to mitigate harm even if the AI is misaligned. This is because this second line of defense is \\textit{application-specific}: mitigating harm from a misaligned AI agent that is writing security-critical code is very different from mitigating harm from a misaligned AI judge that may collude with the policy.\n\nThis motivates measures that focus specifically on mitigating harm from collusion by misaligned judges. The simplest approach is to use judges that are not capable of collusion, which can be assessed through capability evaluations for situational awareness and strategic reasoning~\\citep{phuong2024evaluating, mallen2024subversion, laine2024sad}, discussed further in \\Cref{sec:misalignment-inability-safety-case}. For more powerful AI judges, a more sophisticated approach is needed.\n\n\\cite{shlegeris2024untrusted} studies this setting under the name ``untrusted monitoring'', reflecting the assumption that the AI judge may itself be misaligned. One key advantage in this setting is that for the judge to enable harm by allowing a policy to continue with a specific bad behavior, it must reliably avoid flagging that behavior. (If the judge only avoids flagging the behavior half the time, the other half is likely still enough to train the policy not to perform that behavior, or to catch the policy at deployment time and determine it is misaligned.) Thus, \\textit{testing} is especially useful for judge safety: we can construct synthetic examples of policy failures, and check that the judge does flag these failures. The judge should not be able to distinguish between synthetic and real examples, which we might try to achieve by producing synthetic examples by taking real data and modifying it only slightly.\n\nTesting can still fail if the judge can distinguish and policy can coordinate on a specific feature of inputs on which to fail, since we may not test inputs with that particular feature. However, this coordination may be quite challenging to achieve. \\cite{shlegeris2024untrusted} suggest \\textit{paraphrasing} policy outputs and \\textit{redacting} irrelevant information before running the AI judge, to reduce the channels through which a policy and judge could coordinate.\n\n\n\n\\subsubsection{Generalization}\n\nIt would be ideal for judges to generalize to new situations, as this is often where they are most important. Simply increasing the amount of data used has been seen as a promising approach to improve generalization~\\citep{radford2021learningtransferablevisualmodels, hestness2017deep}. However this may scale poorly as the amount of data needed to achieve a certain level of generalisation can be exponential in the size of the input space~\\citep{gilmer2018adversarial}. \\cite{howe2024effects} find that for LLM classifiers, as the LLM scale is increased, adversarial training becomes a better defense, though it is still easy in absolute terms to find attacks.\n\nGeneralization is a hard and ill-defined desideratum, but there are some specific types of generalization that may be promising. Ideally, the judge would reliably generalize from simple common cases, where oversight is easier, to more complex cases.\nIn general these are types of normative extrapolation, where a principle of reasoning can be adopted to specify correct generalization from simple cases. For instance, compositional generalization~\\citep{lake2018generalization} has been studied and seen substantial improvements in recent years.\n\nWhile these limited forms of generalization appear achievable, it is unlikely that AI judges will be able to generalize perfectly to arbitrary questions, as any useful AI system must have bias~\\citep{mitchell1980need}, which is unlikely to exactly match the desired bias.\nAs a result, to handle cases in which generalization cannot be expected, it is important that the AI judge is aware of its own limitations and has appropriate uncertainty, including out of distribution.\n\n\n\n\\subsubsection{Uncertainty estimation}\n\nSince the judge will often be weaker and cheaper than the AI system it is overseeing (to enable scalability), a key property of the judge is to seek out further guidance when needed. This ability to ask for further guidance is what distinguishes AI judges from amplified oversight (\\Cref{sec:amplified-oversight}), which must instead reliably produce an overall judgment. For the judge to seek out further guidance appropriately, it must know what it doesn't know. Thus we would like the judge to have well-calibrated uncertainty with respect to the most reliable oversight signal available.\n\nWhile calibrated uncertainty comes for free when training against the target distribution using a proper scoring rule such as cross-entropy loss, this is certainly not the case for judge models derived from general purpose language models. Indeed, language models are spotty at reporting their uncertainty~\\citep{kadavath2022language}.\n\nThere are many approaches to improving uncertainty estimates for deep learning~\\citep{abdar2021review}, including Bayesian neural networks~\\citep{jospin2022hands}, MC dropout~\\citep{gal2016dropout}, and ensembles~\\citep{lakshminarayanan2017simple}. However, these are often difficult to scale to LLMs, and as a result uncertainty quantification for LLMs often instead leverages the LLM's own ability to reason in natural language~\\citep{shorinwa2024survey}. For example, there are promising results from teaching an LLM to verbalize its uncertainty~\\citep{lin2022teaching}, measuring the semantic consistency of the LLM's outputs under perturbations~\\citep{kuhn2023semantic}, and probing the internal activations of the LLM~\\citep{kossen2024semantic}.\n\nNonetheless, many challenges remain, including some that are unique to misaligned AI systems. The policy may understand that it is being judged, and behave differently (and adversarially) as a result. While this bears a resemblance to defending against human adversaries (such as in cyberdefense), this property is atypical for AI systems, which often assume stationarity of the environment. In addition, as AI systems become ever more powerful, our uncertainty estimation will also have to handle more ambiguous questions where we ourselves have high uncertainty.\n\n\n\n\\subsubsection{Resource efficient oversight} \\label{sec:resource-efficient-oversight}\nIf we have a family of judge models (or systems) that trade off expense with quality, we can manage costs by aiming for \\textit{resource efficient} oversight.\nA central problem is thus allocating enough resources to consequential and harmful actions to be able to notice them, before they are taken. Doing this well requires us to:\n\\begin{itemize}\n    \\item Cheaply pass through low-consequence actions.\n    \\item Quickly evaluate obviously harmless or harmful actions.\n    \\item Be good at allocating enough effort to consequential and opaque actions to determine if they are safe or not.\n\\end{itemize}\n\nSimilar problems have been encountered in related fields. For example, in cyber-threat detection systems the vast majority of logs that are generated represent safe behavior, but a small number of signals might represent users who are trying to evade detection and do something you do not want. Similarly, in many signals intelligence settings, the vast majority of intercepts are not worth much effort but a very small number of highly consequential communications are.\nFor these similar problems, several strategies have been important building blocks. \nFor instance, anomaly detection systems~\\citep{chandola2009anomalydetection} can be used to prioritize outlier-behavior. \n\nThere are a number of approaches to building systems with controllable trade-offs. For instance, dynamic allocation of resources (sometimes referred to as adaptive computation, conditional computation, adaptive inference-time search, or data-gathering)~\\citep{raposo2024mixtureofdepths,bengio2016conditionalcomputation}.\nTest-time compute in the form or variable length CoT or variable samples used to derive a final reward~\\citep{snell2024scaling} could be used to increase the quality of oversight. A training system base on test-time compute in the judge might first estimate how tricky or important a situation is, then choose an appropriate level of supervision compute.\n\nSignificant effort will likely be needed to make monitoring cost-effective to run. One approach would be hierarchical judging systems, where a cheap system with high recall but low precision cascades to systems with progressively higher precision. For instance, cheap filter rules or anomaly detectors with high recall can cascade to more expensive uncertainty-based decision rules. This approach is especially well-suited to monitoring deployed systems, due to the sheer scale required at deployment.\n\n\n\n\\subsubsection{Robust training}\n\nDuring training, amplified oversight can be used to ensure appropriate incentives on individual inputs to the policy. The subproblem of robustness is then focused on how to ensure this generalizes to a wide distribution of potential inputs, including ones which may not arise by default in training. Typically, this is achieved by identifying and synthesizing novel inputs on which to train the policy, with a focus on inputs that are informative and potentially dangerous. The policy is then trained to behave well on these inputs, called \\textit{adversarial trainin","chars":13000,"total_chars":25919,"offset":0,"truncated":true,"next_offset":13000}