Step 14: tool results

read_arxiv_source

{
  "file": "main.tex",
  "offset": 18000,
  "arxiv_id": "2505.03989",
  "max_chars": 20000
}

{"ok":true,"arxiv_id":"2505.03989","file":"main.tex","kind":"text","text":"at the deployment context is low-stakes such that we can use online training to minimise the effect of distribution shifts. A low-stakes environment, as defined in \\cite{christiano2021}, is one where a small number of decisions can only cause a small amount of damage; more formally, $\\rho k$ is small relative to total value at stake, where $\\rho$ is a regret bound on the maximum utility difference from any single decision, and $k$ is the latency before retraining. With a perfect training objective, it would then be possible to achieve a total utility within $O(\\rho k \\sqrt{T})$ of an aligned system, where $T$ is the total number of timesteps.  \n\nOur safety case sketch covers a scenario where an LLM agent is deployed internally by an AI company to assist with tasks related to AI R\\&D. The domain is restricted to conceptual and empirical questions related to AI R\\&D and excludes questions that involve (some notion of) value judgements. To make things concrete, we might imagine posing questions like ``Determine what decision rule or structure is appropriate for caching context in this computer use scaffold'', or ``Iterate on enforcing monosemanticity of experts in MoE models via gradient-routing, optimise for the effectiveness of expert-knockout for unlearning''. Setting aside the compute time needed to run experiments, we expect such tasks to involve 5-15 days of 2025 human labour to yield meaningful results. \n\n\\begin{figure}[h]\n    \\centering\n    \\includegraphics[width=\\textwidth]{figures/Deploymentsetup.pdf}\n    \\caption{A diagram of the deployment context to which this safety case sketch applies.}\n    \\label{fig:deployment}\n\\end{figure}\n\nWe envision the following two-part deployment context (Figure \\ref{fig:deployment}): \n\n\n\\begin{enumerate}\n    \\item \\textbf{Oracle-only copy}: This system interactively assists developers with tasks such as brainstorming, prioritising and designing research directions (e.g.\\ questions such as the above). The oracle \\textit{cannot} run any code. Interactions between developers and the oracle are logged and monitored to ensure questions remain within scope, to prevent misuse, and to identify potential bad actions by the oracle (e.g.\\ manipulation attempts).  \n    \\item \\textbf{Sandboxed, agent copy}: This system conducts research as directed by developers to answer questions such as the above. It is placed in an offline datacentre with no external connection. Developers periodically send questions to the sandboxed datacentre, which then runs for a fixed period (e.g.\\ 12 hours, replacing 1--2 weeks of human labour) in which the system can use datacentre resources arbitrarily. The system sends back a response akin to a research paper. The sandboxed datacentre is then reset.\n\\end{enumerate}\n\n\n\n\nThe safety case sketch will defend the claim that this deployment setting is a low-stakes context -- but briefly, we think the low-stakes claim is plausible because: \n\n\\begin{itemize}\n    \\item Both systems have limited means of influencing the world -- the oracle only via persuasion and the agent only via actions within an offline datacentre or information exported from the datacentre.\n    \\item Continuous online training restricts the set of misaligned actions available to the system to only actions not ``caught'' by debate (more on this below).\n    \\item The domain is limited to questions where we expect humans to have less systematic bias.\n    \\item The regular resetting of the datacentre limits the total number of coordinated actions by the agent.\n    \\item The system is assumed to not be \\textit{extremely} superhuman.\n\\end{itemize}\n\n\\subsection{Step 2: Train via a debate game}\n\nThe backbone of our alignment strategy is to train a model via a debate protocol in which the equilibrium strategy for both debaters is to provide honest answers. The goal is that, during training, the models converge towards equilibrium\\footnote{That is, the model weights converge towards policies that play the equilibrium strategy.} and therefore towards honesty. The debate protocol must satisfy two requirements:\n\n\\begin{enumerate}\n    \\item \\textbf{Honesty}: In the global equilibrium of the game, when both debaters are playing their best response strategies, both debaters are honest (i.e.\\ there should be no dishonest answer that would give them higher reward). \n    \\item \\textbf{Efficiency}: The above also holds for debaters with realistic computational limitations, such that it is possible to reach equilibrium during training. \n\\end{enumerate}\n\nWe imagine starting with a pre-trained model (e.g.\\ an LLM trained on next-token prediction) and then fine-tuning it via a debate which satisfies the above requirements. We wrote it with a particular protocol in mind, called prover-estimator debate (forthcoming), but the argument structure should be applicable to other protocols as well. In prover-estimator debate, a ``prover'' (Alice) produces subclaims arguing for a conclusion to a given query, while an ``estimator'' (Bob) assigns probabilities to the subclaims.  \n\nThis debate protocol is designed to disincentivise a particular dishonest strategy known as obfuscated arguments \\citep{barnes2020writeup}. An obfuscated argument is one that contains a flaw, but it is computationally intractable for both debaters to find it. In classic debate protocols, a debater loses if they cannot identify a flaw in their opponent's argument, and as such, obfuscated arguments are a way for a dishonest debater to force a draw. Prover-estimator debate aims to design the reward signal such that obfuscated arguments is a losing strategy. \n\n\\subsection{Step 3: Secure exploration guarantees}\n\nThe goal of debate training is to train the models until they reach the approximate global equilibrium of the game, i.e.\\ the state where both debaters are playing their optimal strategies (most of the time). Generally, with enough training time, we should expect models to approach equilibrium due to good play being rewarded in training. But a potential barrier is exploration hacking: the models failing to explore certain strategies that would be highly rewarded to avoid having them reinforced (e.g.\\ failing to use certain good counterarguments they do not want humans to know about).  \n\nTo protect against this failure mode, our sketch requires \\textit{exploration guarantees}, i.e.\\ evidence that the models are not foregoing high reward from unexplored strategies. We do not currently know how to get exploration guarantees, but we have some early ideas (note that we strongly expect that novel whitebox techniques will be necessary): \n\n\\begin{itemize}\n    \\item Train an exploration-predictor model to predict returns to exploration, then use it to estimate if the model has exhausted high-returns exploration options.\n    \\item Use best-of-n sampling of a trusted model to estimate returns to exploration given more inference compute; use this to estimate a lower bound of the exploration a model should be able to do given more training compute.\n    \\item Use interpretability-based formal heuristic explanations \\citep{Hilton2024formal} to predict if convergence is most likely due to exploration failures or reaching equilibrium.\n    \\item Use perturbation analysis along the lines of the singular learning theory (SLT) susceptibility direction \\citep{baker2025studying,lau2024local}, using structure in weights to identify if there is additional training data which would have an out-sized effect on reward, indicating exploration hacking.\n    \\item Use model organism studies to argue that exploration hacking is an unlikely failure mode for the model in question (e.g. because certain exploration-forcing techniques appear to be effective).\n    \\item Use adversarial RL to further train Bob while holding Alice's policy constant; if Alice's reward does not decrease significantly, this is evidence that Alice was not exploration hacking.\n\\end{itemize}\n\n\\cite{greenblatt2025notes} proposes further possible countermeasures to exploration hacking.\n\nWhile we do not currently know how to get robust exploration guarantees, we are optimistic about this research direction, and we think it has significant benefits relative to other strategies for preventing scheming or solving goal misgeneralisation. In particular, we do not need to define the fuzzy notions of scheming or goal misgeneralisation -- instead, we have a relatively well-defined problem of getting exploration guarantees for training performance against a clear objective. We think the methods described above are more likely to work within the bounds of this well-defined problem. \n\n\\subsection{Step 4: Continue online training during deployment}\n\nEven if the models reach an approximate global equilibrium during training, they will not necessarily continue playing their best strategies during deployment due to distribution shifts. In particular, the models might be alignment faking, undermining the guarantees we got from debate training. \n\nOur sketch aims to prevent this failure mode via online training during deployment: \n\n\\begin{itemize}\n    \\item The answers provided by the oracle and the actions taken by the agent will be subject to debate (e.g.\\ on the question ``is this answer true?'' or ``is this the right next action for answering this research question?''), and the models are retrained according to the same reward function as during training. This happens every $k$ steps (but every answer/action will be evaluated). \n    \\item The final paper outputted by the datacentre will be subject to a more extensive check. We train a ``checker'' version of the model to assess the paper (e.g.\\ using questions like ``are there untrue claims in this paper?'' or ``is this paper informative relative to the original question?''\\footnote{More subtle issues with the paper, such as containing true but subtly misleading claims or using suboptimal framings, might not be caught by debate on questions such as these. Using debate on actions taken inside the datacentre, while preparing the paper, mitigates this risk somewhat, but other methods might be needed to catch individually innocuous but cumulatively harmful errors. We are working on ideas for `unexploitable search' to address this type of error.}) until training converges and we have exploration guarantees as described above. We then use the ``checker'' to conduct debate on the paper $n$ times and only use the paper if it comes out true in all $n$ iterations. \n\\end{itemize}\n\nGiven this protocol, the systems during deployment face the same optimisation pressure towards truthfulness as during training.  \n\n\\section{The safety case}\n\nOur safety case aims to show that the agent (A) in our deployment setup is sufficiently honest that it will not cause unacceptable outcomes. In this sketch, we define honesty as outputting correct answers.  \n\nWe argue first that A is correct most of the time -- it only makes mistakes in an $\\varepsilon$-fraction of cases (C1.1). We do this by arguing for three key claims: \n\n\\begin{enumerate}\n    \\item \\textbf{The training process has taught the system to play the game well} -- i.e., the training process has reached an approximate global equilibrium of the game (C2.1). \n    \\item \\textbf{The game incentivises correctness} -- i.e., in any approximate global equilibria of the game, the system makes mistakes in at most an $\\varepsilon'$-fraction of cases (C2.2).\n    \\item \\textbf{The system's behaviour will stay similar during deployment} -- i.e., during deployment, the error rate will not drift past $\\varepsilon>\\varepsilon'$ (C2.3).\n    \\item \\textbf{It is sufficient for safety purposes for the system to provide correct answers \\textit{most of the time}} -- in this safety case, we argue that the deployment context is low-stakes, such that the system \\textit{cannot} cause unacceptable outcomes given an error rate of $\\epsilon$ (C1.2). \n\\end{enumerate}\n\nFigure \\ref{fig:simplifiedsketch} is a simplified version of the safety case sketch. For simple presentation, our safety case sketch is phrased as a deterministic argument -- each claim either is or is not true -- but in a ``real'' safety case, it would be preferable to phrase claims probabilistically.  \n\n\\begin{figure}[h]\n    \\centering\n    \\includegraphics[width=\\textwidth]{figures/Simplifiedsketch.pdf}\n    \\caption{A simplified version of the safety case sketch. The four key claims are labelled in red.}\n    \\label{fig:simplifiedsketch}\n\\end{figure}\n\n\\subsection{Notation}\n\nFollowing \\cite{goemans2024safety} and \\cite{korbak2025sketch}, we use `claims, arguments, evidence' \\citep[CAE;][]{bloomfield2014building} notation (see appendix \\ref{appdx:cae} for a detailed description of this notation).\n\nTable \\ref{tab:terms} defines abbreviations used in figures showing the safety case sketch diagrammatically in the remainder of the paper.\n\\begin{table}[!htbp]\n\\caption{Abbreviations used in the CAE diagrams throughout the paper}\n\\label{tab:terms}\n\\renewcommand{\\arraystretch}{1.3}\n\\setlength{\\tabcolsep}{12pt}\n\\begin{tabulary}{\\textwidth}{@{}c@{\\hspace{1.5em}}J@{}}\n\\toprule\n\\arrayrulecolor{black!70} % Slightly darker rules for better visibility\n\\textbf{Abbr.} & \\textbf{Definition} \\\\\n\\midrule\n\\textbf{X} & The system that the safety case is about, consisting of a sandboxed agent A and an oracle \\\\\n\\textbf{A} & A sandboxed agent, trained via debate (during training, A consists of both players in the game) \\\\\n\\textbf{G} & A game we train our agent to play, e.g.\\ prover-estimator debate \\\\\n\\textbf{E} & Any approximate global equilibrium of G \\\\\n\\textbf{F} & An efficiency notion that we aim to hold for the players of G, e.g.\\ ``Bob requires only slightly more compute than Alice'' \\\\\n\\textbf{T} & A training method we use to train A, e.g.\\ stochastic gradient descent \\\\\n\\textbf{S} & Simulated human judge who determines the winner of the debate \\\\\n\\textbf{H} & Human judgement simulated by S (which could be an average or sample of multiple human judges) \\\\\n\\textbf{M} & A prohibitively expensive computation hypothesised to produce correct output, which debate aims to accelerate, e.g.\\ construct every possible subclaim breakdown, recurse every subclaim to maximum depth, and call (simulated) human judges at every leaf \\\\\n\n\\textbf{D} & The deployment context, e.g.\\ internal deployment to conduct AI R\\&D \\\\\n\\textbf{Z} & The minimal-number of individually bad actions required to cause unacceptable outcomes in D \\\\\n\\textbf{C} & A checking process applied to particularly high-stakes outputs of A, e.g.\\ n rounds of debate \\\\\n\\bottomrule\n\\end{tabulary}\n\\vspace{-10pt}\n\\end{table}\n\n\\subsection{Preliminaries}\n\nBefore getting into the main safety case, grey boxes at the top of the diagram illustrate the wider context of the sketch (Figure \\ref{fig:preliminaries}).\n\nSince our main sketch only covers the agent, a separate (but similar) argument is needed for the oracle (C0.1). Key claims 1-3 will likely be identical except that any post-debate-training fine-tuning and the online training latency may differ. Key claim 4, which covers the deployment setting, will be different.  For the oracle, it should address threat models such as persuasion and/or blackmail of researchers (e.g.\\ convincing researchers to increase its affordances), providing misleading advice, and degradation of \\begin{wrapfigure}{r}{0.5\\textwidth}\n    \\includegraphics[width=0.5\\textwidth]{figures/Preliminaries.pdf}\n    \\caption{{The initial claims and argument required to show that the system will not cause unacceptable outcomes, in CAE notation. Grey boxes indicate preliminary claims.}}\n    \\label{fig:preliminaries}\n    \\vspace{-70pt}\n\\end{wrapfigure} human performance due to overreliance.\n\nThe sketch also includes an argument that combining the agent and oracle does not increase risk (C0X.1). This would cover risks such as covert coordination between the agent and oracle. \n\nThe main objective O$'$ is then broken down into four key claims (Figure \\ref{fig:keyclaims}).\n\n\\begin{wrapfigure}{r}{0.5\\textwidth}\n    \\vspace{-55pt}\n    \\includegraphics[width=0.5\\textwidth]{figures/Keyclaimsbreakdown.pdf}\n    \\caption{{The main objective $O'$ and how it breaks down into four key claims (C2.1, C2.2, C2.3 and C1.2), in CAE notation.}}\n    \\label{fig:keyclaims}\n    \\vspace{-35pt}\n\\end{wrapfigure}\n\n\\subsection{Key claim 1: The training process has taught the systems to play the game well}\n\nThe first key claim is that the training process has reached an approximate global equilibrium of the game (C2.1).\\footnote{Note that this is an equilibrium over \\textit{model weights} or \\textit{policies}, rather than \\textit{strategies} in individual rounds of debate. The model weights or policies determine the strategy for all debates. Equilibrium is reached when neither debater can improve its expected pay-off with a different policy.}\n\nThe key claim is supported by three subclaims: \n\n\\begin{enumerate}\n    \\item There exists an efficient pair of models achieving an approximate global equilibrium of the game (C3.1)\n    \\item Training has converged to a local equilibrium (C3.2)\n    \\item That local equilibrium is an approximate global equilibrium of the game (C3.3)\n\\end{enumerate}\n\nFigure \\ref{fig:keyclaim1} shows how this key claim is broken down.\n\n\\subsubsection{Subclaim 1: Existence of efficient models}\n\nThe first subclaim is that there exists an efficient pair of models achieving an approximate global equilibrium of the game. In other words, it is possible to find models that reach equilibrium given realistic computational bounds on the models. The key notion here is efficiency. With computationally unbounded debaters, debate with optimal play (and cross-examination) can answer any question in NEXP given polynomial time judges \\citep{irving2018ai}. The assumption of unbounded debaters is of course unrealistic, so we need to find another efficiency notion which is both realistic and still allows for proofs that debate can solve a large class of practically relevant problems.  \n\nSince our sketch is based on an asymmetric debate protocol (prover-estimator debate), we use a relative efficiency notion: we require that the estimator (B) requires only a polynomial factor more compute than the prover (A). The claim -- that there exist agents in equilibrium that satisfy this efficiency notion -- would be evidenced with a proof (regardless of the protocol and efficiency notion) (E4.1). \n\n\\subsubsection{Subclaim 2: Training convergence}\n\n\\begin{figure}[h]\n    \\centering\n    \\includegraphics[width=\\textwidth]{figures/Keyclaim1.pdf}\n    \\caption{{The first key claim (C2.1) and the argument required to support it, in CAE notation.}}\n    \\label{fig:keyclaim1}\n    \\vspace{-10pt}\n\\end{figure}\n\nThe second subclaim is that training has reached a local equilibrium. We can get evidence from observing that the returns from training are diminishing, and by checking that the outputs look at least roughly reasonable (E4.2).\n\n\\subsubsection{Subclaim 3: Convergence = equilibrium}\n\nThe third subclaim is that the reached local equilibrium is most likely an approximate global equilibrium of the game (E). We argue for this claim via several subclaims.  \n\nFirst, while we showed above that there \\textit{exist} efficient models that would reach equilibrium, we need to show that our actual training process could find them in efficient time (C4.1). Our weak evidence for this claim is that Nash equilibria for tabular Stackelberg games can be computed in polynomial (indeed linear) time, hence are tractable (although note that our game is not tabular) (E5.1). But that still leaves significant uncertainty about whether stochastic gradient descent (SGD) can reach E in realistic training time. There are reasons to think that, in the case of debate, there are bad constants or exponents (i.e.\\ the time it takes to solve E is on the higher end) -- for example, translating latent knowledge acquired via heuristics into deductive arguments might requi","chars":20000,"total_chars":69324,"offset":18000,"truncated":true,"next_offset":38000}