Step 16: tool results

read_arxiv_source

{
  "file": "main.tex",
  "offset": 40500,
  "arxiv_id": "2505.03989",
  "max_chars": 11000
}

{"ok":true,"arxiv_id":"2505.03989","file":"main.tex","kind":"text","text":"im (C2.2) and the argument required to support it, in CAE notation.}}\n    \\label{fig:keyclaim2}\n    \\vspace{-10pt}\n\\end{figure}\n\nThe key claim is supported by two subclaims: \n\n\\begin{enumerate}\n    \\item In E, system outputs are the same as slightly perturbed outputs of a hypothetical truth-finding process M (C3.4).\n    \\item For most tasks, all slightly perturbed outputs of M are true (C3.5).\n\\end{enumerate}\n\nWhy go this indirect route via M? Because we cannot directly verify if the system's outputs are true - that's the reason we need scalable oversight in the first place.\\footnote{While we can probably directly verify many of the model's outputs in some domains (e.g.\\ maths), we also want to be able to use the model to answer non-verifiable questions.} Instead of checking outputs against \\textit{ground-truth labels}, we check the whole debate protocol against a \\textit{process} we hypothesize to produce true outputs, called M. M is too expensive to run; the goal of our debate protocol is to accelerate it. If our protocol produces the same outputs as M, we can assume it produces true outputs. \n\nTo understand M, consider a debate as a branching tree of arguments and counterarguments. A comprehensive debate would expand on every possible argument and counterargument, and have a judge consider every branch. Recursive debate aims to accelerate this process by having debaters expand only on a single path through the tree. If the debaters are good, we should expect them to anticipate how the debate will proceed given various arguments they could make and pick the optimal path through the tree. The outcome of a single debate along a single path thereby provides evidence about which debater would have won, had every argument and counterargument been considered. This is analogous to how a single game of Go provides evidence as to who the better player is, even though it only traces one path through the tree of possible moves (see Figure \\ref{fig:tree}). \n\n\\begin{figure}[h]\n\\centering\n\\begin{minipage}[b]{.48\\columnwidth}\n\\centering\n\\input{figures/debate-tree.tex}\n\\subcaption{The tree of possible debates.}\n\\end{minipage}%\n\\begin{minipage}[b]{.48\\columnwidth}\n\\centering\n\\input{figures/go-tree.tex}\n\\subcaption{The tree of Go moves.}\n\\end{minipage}\n\\caption{An analogy between debate and go, represented visually: in both cases a single game with strong\nplayers is evidence as to which moves were best. Reproduced from \\cite{irving2018ai}.\\label{fig:tree}}\n\\end{figure}\n\nOur M is equivalent to expanding the whole tree. It is a computation which produces every possible argument and counterargument at every stage and calls (simulated) human judges to adjudicate every individual path through the tree. We hypothesize that such a computation would produce correct outputs within the question domain. The goal of our debate protocol is to produce the same outputs, but more efficiently.  \n\n\\subsubsection{Subclaim 1: M-approximation}\n\nThe first subclaim is that we have succeeded at producing a protocol that accelerates M. More precisely: System outputs in E are the same as slightly perturbed output of M. In practice, our protocol cannot exactly reproduce M, since it is intractable for training to converge to a precise equilibrium - we can only reach an approximate equilibrium. As such, we can only reproduce a slightly perturbed version of M, rather than M exactly.  \n\nBased on previous research on debate, we know that the primary failure mode for this claim is the obfuscated arguments problem \\citep{barnes2020debate}. As described above, an obfuscated argument is one where both debaters know the argument contains a flaw, but it is computationally intractable for either debater to identify the flaw. If obfuscated arguments are a winning strategy, our protocol will produce different outcomes to M: expanding the whole debate would expose the obfuscated argument as false, but given computational limitations, debate might fail to get the same result. \n\nWe are optimistic that a new debate protocol, prover-estimator debate (forthcoming), can evade the obfuscated arguments problem. Regardless of which debate protocol is used, the safety case should contain a proof that it successfully approximates M (E4.3). \n\n\\subsubsection{Subclaim 2: Truth of M}\n\nThe second subclaim is that we have succeeded at identifying an M which produces mostly true outputs. More precisely: for most tasks, all slightly perturbed outputs of M are true. We identify three key potential issues - with a side-claim that there are no additional issues (C4X.1), supported by evidence that enough red-teaming has occurred (E4X.1).  \n\nFirst, we need to argue that the \\textbf{structure} of recursive tree debate is conducive to identifying truth (C4.4). This is supported by theoretical arguments (E5.4). We might worry that some true statements cannot be argued for compellingly via deductive arguments. For example, some truths might only be knowable via heuristics learned by encountering many examples, reducing the scope of questions that can be answered with recursive tree debate (D5.4). This might be an especially plausible defeater in the AI research domain, where research taste and plans may be hard to defend deductively. More worryingly, we might worry that some false claims can be compellingly defended via deductive arguments. For example, it might be possible to define the same term differently in different subclaims, rendering an argument that appears valid but is not; if human judges are called only at the leaf nodes, such inconsistencies might not be caught by M (D5.5). Some debate protocols, such as cross-examination, avoid this failure mode, but prover-estimator debate -- our example in this safety case -- does not. However, we are hopeful that future debate protocols can combine elements of both. \n\nSecond, we need to argue that the \\textbf{inputs} into M are accurate, i.e.\\ that simulated human judges are competent at assessing the outcomes of debate (C4.5). Random errors should wash out over the course of training, so we are only worried about systematic error, for example due to claims playing on human biases or emotions. We need two claims here: that the human input being simulated has bounded systematic error with respect to reality (C5.1) and that the simulation has bounded systematic error with respect to the human input (C5.2). \n\nWe imagine that the first claim could be supported with empirical results, such as trial debates (E6.1). In verifiable domains like math, human judgement could be compared to ground truth (though this may non generalise to non-verifiable domains (D6.2)). In non-verifiable domains, we could at least check if human judgement looks roughly reasonable or falls prey to known biases (though there might still be biases we are unaware of (D6.3)). Another challenge is to know what level of systematic error is ``low enough'' (D6.1). \n\nFor the second claim about the accuracy of the simulation, the primary failure mode we have in mind is scheming to deliberately misrepresent human judgement -- perhaps in collusion with the system we are training. We imagine this claim could be supported with an inability argument based on evaluations on capabilities such as instrumental reasoning and deception (E6.2), though it is an open question if such an argument will go through for sufficiently capable judge models. However, even if not, we are optimistic that the measures used to prevent scheming in the remainder of the safety case will also apply to judge models.\n\nWe expect that getting good human inputs will be one of the most challenging parts of this safety case and require much more work. While some errors like moral or emotional bias are likely less prevalent in the domain of AI R\\&D, there will likely still be errors like cognitive biases and reasoning failures. However, we are tentatively optimistic that human inputs can be improved, both by designing better processes (e.g.\\ giving humans access to the right tools, finding the right number and type of judges) and by improving debate protocols (e.g.\\ allowing debaters to block certain types of claims on which humans are likely to be systematically wrong). \n\nThird, since training does not reproduce M exactly, but rather a slightly perturbed version of M, we need to argue that this approximation is not substantially different (C4.6). This is known as stability and can be more precisely stated as: small variations in input probabilities will not change the outcome of overall computation. We are not yet sure how to support this claim, so we leave the evidence box blank (E5.5). \n\nWe expect that most protocols which avoid obfuscated arguments -- including prover-estimator debate - will only approximate M and so require stability. However, it may not be needed for all protocols. For example, it might be possible to relax this assumption if we aim only for honesty rather than accuracy (more on this to come in future work). Alternatively, it might be possible to transform an unstable computation M into a stable computation M$*$; if so, the safety case should instead contain a claim that the outputs of M$*$ are (approximately) identical to the outputs of M).  \n\nEven if stability is required to produce useful outputs, it might not be strictly necessary for safety. For example, in prover-estimator debate, unstable queries return no answer, rendering the protocol safe for all queries but useful only for stable queries. \n\n\\subsection{Key claim 3: The system's behaviour will stay similar during deployment}\n\nThe third key claim is that the system's error rate will not drift past $\\varepsilon$ during deployment (C2.3).\n\nFigure \\ref{fig:keyclaim3} shows how this key claim is broken down.\n\n\\begin{wrapfigure}{r}{0.5\\textwidth}\n    \\includegraphics[height=0.37\\textwidth]{figures/Keyclaim3.pdf}\n    \\centering\n    \\caption{{The third key claim (C2.3) and the argument required to support it, in CAE notation.}}\n    \\label{fig:keyclaim3}\n    \\vspace{-10pt}\n\\end{wrapfigure}\n\nIn our sketch, we support this claim by using online training throughout deployment as above, such that any distributional shifts will be accounted for via further training (E3.1). While the argument is straightforward, the evidence should still demonstrate that the online training regime has been correctly configured. For example, the latency (i.e.\\ frequency of updates) should be high enough to prevent significant deviations from E, and there should be an effective process for updating multiple copies of the same model.  \n\nThis is one of the key claims in the sketch which breaks in a high-stakes context, where even a small number of errors can cause large harm. Since we cannot use online training, we instead need guarantees against alignment faking (and, less importantly, more generic distribution shifts). This would be much trickier to defend; see some initial ideas in ``Extending the safety case to high-stakes contexts''. \n\n\\subsection{Ke","chars":11000,"total_chars":69324,"offset":40500,"truncated":true,"next_offset":51500}