Step 10: tool results

read_arxiv_source

{
  "file": "06-addressing-misalignment/amplified-oversight.tex",
  "arxiv_id": "2504.01849",
  "max_chars": 6500
}

{"ok":true,"arxiv_id":"2504.01849","file":"06-addressing-misalignment/amplified-oversight.tex","kind":"text","text":"\\subsection{Amplified oversight}\\label{sec:amplified-oversight}\n\nRecall from section \\ref{sec:misalignment-risks} that misalignment occurs when the AI system produces harmful outputs for intrinsic reasons that the designer/user would not endorse. There are two possible sources of misalignment: specification gaming, in which the designer-provided specification (e.g. reward signal) is flawed in a way that the designers did not foresee, and goal misgeneralisation, in which pursuit of an undesired goal arises in situations which are out of distribution compared to training. Amplified oversight, our focus in this section, aims to mitigate the specification gaming source of misalignment.\n\nIn order to describe amplified oversight, it is helpful to recall from section \\ref{sec:misalignment-risks} the principle of informed oversight: it would be sufficient to rule out misalignment if we a) know everything that the AI  \"knew\" when producing its outputs and b) always correctly endorsed those outputs. We can somewhat reduce the ambition of achieving informed oversight, by aiming to achieve informed oversight on a single input-output pair – this is the focus of amplified oversight research. It aims for a human to provide feedback for a superhuman AI input-output pair which is as good as could be achieved if that human:\n\\begin{enumerate}\n    \\item understood all of the reasons that the AI system produced that output; and \n    \\item had unlimited ability to reflect on the correct decision.\n\\end{enumerate}\n\nSince the AI system is superhuman (by hypothesis), it will choose actions for reasons that it understands but that humans would not understand by default. Thus, a core intuition in amplified oversight is to use the AI system itself to make these reasons legible to oversight mechanisms. A prototypical example that we will return to below is debate, in which two copies of an AI compete to point out flaws in each other’s output to a human judge. It is hoped that exposing flaws will help to surface some of the knowledge used by the AI to produce its output, because if the AI provides an answer that isn't to the best of its knowledge, its opponent could point that out and win the debate. The difficulty of identifying and explaining a subtle flaw is thus offloaded to the superhuman AIs, with the human doing the relatively easier task of evaluating the highlighted flaws. \n\nFor example, consider the case of an AI trained to write long and complex legal contracts. Given such a contract of several thousand pages, it would take even expert human lawyers a very long time to verify that the legal text achieved the intended goals of the humans who requested the contract. Thus, labeling even a single input-output pair would be prohibitively expensive. On the other hand, given a second AI trained to point out flaws in legal contracts, it would be sufficient to identify a flaw in a few lines of the legal text. A human presented with these lines could then quickly judge if they do indeed constitute a flaw relative to the original human intent for the contract.\n\nImportantly, amplified oversight techniques generally maintain humans in the loop—for training, evaluation and monitoring—to ensure that their values are upheld. Some aspects of human judgments can be imitated or delegated, according to the human’s evolving trust in the AI. The form of human involvement in the amplified oversight protocol is an open question.\n\n\\subsubsection{Methods}\\label{sec:ao-methods}\nWe’ll now describe some amplified oversight methods, with a focus on how these appear in the modern era of large language models (LLMs). Our descriptions are framed as outlining  particular problems that occur, and approaches to overcome them. \n\nWe begin with the case in which amplified oversight is actually unnecessary, when the desired behaviour is simple enough that either the data demonstrating it already exists, or it is sufficiently cheap/easy for a human to demonstrate it. \n\nIf the desired behaviour can be algorithmically verified, then this verification signal can be used to generate desired behaviour, e.g. using it as a reward in reinforcement learning, or in a scheme such as Self-Taught Reasoner \\citep{zelikman2022star}.  \n\n\\noindent \\textbf{RLHF.}\nOften an LLM’s behaviour after pretraining is often not precisely as desired for applications (e.g. a chatbot/assistant that is helpful, harmless and honest) and there is no algorithmic verification signal to easily improve it. One approach can be to gather human demonstrations of desirable behaviour and do supervised fine tuning~\\citep{ouyang2022training}. However, there are difficulties with this: generating such human demonstrations can be somewhat costly, and the demonstrations don’t convey comparative information between different responses. An approach to this is to have humans act as raters of AI-generated outputs – it is somewhat easier, faster and cheaper for the humans to evaluate whether the generated output is desirable, and to provide extra comparative information by contrasting different model responses. So further fine-tuning is done using reinforcement learning from human feedback (RLHF) \\citep{christiano2017deepreinforcement}. The LLM is fine-tuned via RL using a reward from a learnt reward model, which is trained on human preference data, over AI-generated outputs \\citep{bai2022constitutional,glaese2022sparrow,ouyang2022training,stiennon2022learningsummarize}, see also DPO \\citep{rafailov2023DPO} which uses preferences but avoids the use of RL. \n\n\\noindent \\textbf{Critique.}\nThe shift from human demonstration to human evaluation has an analogy to the complexity classes P and NP: decision-problems that can be solved easily and those which can be verified easily. But just as there are problems harder than P or NP in complexity theory, there may be AI-generated behaviours which are too difficult for a human to evaluate. A natural approach to improve the performance of human raters is to show them AI-generated critiques to assist the human with rating AI-generated behaviours, for example by pointing out mistakes. \\cite{saunders2022critique} gather human data consisting of natural language critiques of text and then use supervised learning to fine-tune an LLM to generate critiques. They show that these critiques consistently help both humans and other LLMs identify issues in written text (e.g. topic-based summaries). \\cite{mcaleese2024llmcritics} train critique models using RLHF to c","chars":6500,"total_chars":30896,"offset":0,"truncated":true,"next_offset":6500}