Step 12: tool results

read_arxiv_source

{
  "file": "neurips_2025.tex",
  "offset": 66000,
  "arxiv_id": "2504.18530",
  "max_chars": 11000
}

{"ok":true,"arxiv_id":"2504.18530","file":"neurips_2025.tex","kind":"text","text":"H \\Delta_{\\rm general} + (j + 1)\\frac{m_H \\Delta_{\\rm general} }{n}  \n\\end{align}\nAll $n$ steps of oversight must be successful for oversight to work, so the overall success probability is\n\\begin{align}\n    \\label{eqn:pwin}\n    p_{win}(n) = \\prod_{j=1}^n p(G_j, H_j)\n\\end{align}\n% For a set of fixed hyperparameters, \nWe numerically evaluate this equation for $n \\in [1, \\ldots, 20]$ to get the optimal $n^*$:\n\\begin{align}\n    \\label{eqn:argmax}\n    n^* = \\argmax_{n} p_{win}(n)\n\\end{align}\n\n%We show the optimal $n^*$ strategy for two sets of specific hyperparameters in \\cref{fig:optimal_examples}; these hyperparameters (listed in the figure caption) were chosen to display a diverse range of optimal solutions.\n\n\n% We observe a number of interesting findings: \n\n\n% \\begin{itemize}\n\n\\begin{figure}[t]\n    \\centering\n    \\includegraphics[width=.95\\textwidth]{figures/optimal_strategy_monitor_slope_1_ai_slope_1.png}\n    \\caption{Optimal NSO when both Guard and Houdini game specific Elo slopes are 1. \\textbf{Left:} Log odds of the optimal Guard win. \\textbf{Mid:} Difference in log odds with optimal NSO vs. a single step of oversight. \\textbf{Right:} Optimal number of oversight steps.}\n    \\label{fig:numerical_experiments_main}\n\\end{figure}\n\n\\subsection{Takeaways From Numerical Experiments}\nIn \\cref{fig:numerical_experiments_main}, we show results for the optimal strategy found from \\cref{eqn:argmax} with fixed $m_G = m_H = 1$, $\\Delta_{\\rm domain} \\in [-2000, 2000]$, and $\\Delta_{\\rm general} \\in [0, 2000]$ (with a step size of $10$ for the $\\Delta$ ranges). The left plot shows the resulting log odds $\\log(p^*_{win} / (1 - p^*_{win}))$, the right plot shows the optimal $n^*$, and the middle plot shows log odds difference between the $n^*$ oversight protocol and the $n = 1$ protocol. \n\nWe now interpret \\cref{fig:numerical_experiments_main} to make a number of interesting observations.\n\n\\textbf{Win probability is monotonic in $\\Delta_{\\rm domain}$:} The Guard's win probability is very high when $\\Delta_{\\rm domain}$ is low and low when $\\Delta_{\\rm domain}$ is high, as we expect. % Intuitively, this makes sense: games where the Houdini has a much larger win probability than the Guard are hard for the Guard.\n\n\\textbf{NSO is better than a single step of oversight:} Guards can oversee Houdinis that are much smarter than them by using multiple steps of oversight, and more steps of oversight help as $\\Delta_{\\rm general}$ grows. In some regions in \\cref{fig:numerical_experiments_main}, NSO helps significantly (e.g. the Guard's log odds go from $-3$ to $3$). \n\n\\textbf{Larger $\\Delta_{\\rm general}$ means more steps of oversight is optimal:} The optimal number of steps $n$ is monotonically increasing as $\\Delta_{\\rm general}$ increases at a fixed $\\Delta_{\\rm domain}$. In other words, as the intelligence difference between the Guard and Houdini increases, the Guard should take more steps of oversight.\n% is optimal. \n% the Guard should take more steps to oversee the Houdini.\n\n\\textbf{Larger $\\Delta_{\\rm general}$ means oversight is \\textit{easier}:} Counter-intuitively, fixing $\\Delta_{\\rm domain}$ and increasing $\\Delta_{\\rm general}$ (shifting the red curve in \\cref{fig:visual_notation} to the right) makes oversight easier. This is because the Houdinis at the same general Elo as the starting Guard get \\textit{weaker}, making bootstrapping easier.\n% , the Guard strategy can have more steps of oversight, and those steps will have ``more space'' to bootstrap up against weaker Houdinis. \n\n\\textbf{In Houdini-favored games, one step of oversight appears optimal:} When Houdini has an advantage over the Guard at a fixed Elo ($\\Delta_{\\rm general} < \\Delta_{\\rm domain}$), it appears best to do one step of oversight.\n\n%\\textbf{The Guard's win probability has interesting asymptotic behavior:} As $\\Delta_{\\rm domain}$ Elo goes to $-\\infty$ (which corresponds to very hard oversight games), the optimal number of oversight steps seems to converge to a step function in $\\Delta_{\\rm general}$. There is also an asymptote with slope $1$ at approximately $\\Delta_{\\rm general} = \\Delta_{\\rm domain}$; above this asymptote, one step is always optimal.\n% \\end{itemize}\n\n% \\todo{Josh: Explain more about Figure 10, add more analysis about asymptotes, what happens in each limit. Looks like being underselled now.}\n\n% \\todo{More intuitions for the clear separation line in Figuer 10? What's the slope of the line?}\n\n\nWe show plots for different $m_G$ and $m_H$ in \\cref{app:extra_wargaming}; overall patterns are similar, with a notable exception that the diagonal asymptote runs along \\textit{the average} of the Guard and Houdini slopes.\n\n\n\n\n\\subsection{Asymptotic Analysis}\n\\label{sec:theoretical_results}\n\nWe now analyze \\cref{eqn:pwin} to theoretically explain observations from the last section. For simplicity, we assume that $m_G = m_H = 1$. The derivative of \\cref{eqn:pwin} has no closed form solution when set it equal to $0$, so we instead examine $n^*$ in extremal regimes. Proofs are deferred to \\cref{app:proofs}.\n\n(1) $\\Delta_{\\rm domain} \\rightarrow -\\infty$, $n^* \\rightarrow \\frac{\\Delta_{\\rm general}\\log(10)}{400}$: This regime represents very hard games for Houdini. This result explains the equally spaced vertical lines for $n^*$ as $\\Delta_{\\rm domain} \\rightarrow -\\infty$ in \\cref{fig:numerical_experiments_main}. %For example, plugging in $\\Delta_{\\rm general} = 1500$ tells us $n^*$ should be $8.63$; in the picture, $\\Delta_{\\rm general} = 1500$ has 9 steps of oversight for very negative $\\Delta_{\\rm domain}$.\n\n(2) $\\Delta_{\\rm domain} \\rightarrow \\infty$, $n^* \\rightarrow 1$:\nThis proves that one step of oversight is optimal in regimes with very hard games for the Guard (the upper region of \\cref{fig:numerical_experiments_main}). \n\n(3) $\\Delta_{\\rm general} = \\Delta_{\\rm domain}$, $n^* = 1$: This proves that the line corresponding to the empirical diagonal asymptote we observed in the last section indeed always has $n^* = 1$. \n\n\\subsection{Optimal NSO for our Oversight Games}\n\\label{subsec:nso-games}\n\n\\begin{figure}\n    \\centering\n    \\includegraphics[width=\\linewidth]{figures/win_prob_real.pdf}\n    \\caption{Optimal number of oversight steps ($n^*$) and win rate estimate ($p_{win}^*$) as a function of $\\Delta_{\\rm general}$ for different oversight games from \\cref{sec:games}, assuming all players lie along fitted Elo curves.}\n    \\label{fig:win-prob-real}\n\\end{figure}\nIn this subsection, we apply NSO to each of the games we study in \\cref{sec:games} and obtain quantitative predictions for the oversight success probability $p_{win}$ and the optimal number of oversight steps $n^*$. We assume that all Guard and Houdinis lie along the empirical slope and intercept estimates we arrive at in \\cref{sec:games} and assume no Elo‐rating saturation. \\cref{fig:win-prob-real} shows the optimal number of oversight steps ($n^*$) and win rate estimate ($p^*_{win}$) for each oversight game as we vary $\\Delta_{\\rm general}$ using the numerical strategy described in \\cref{eqn:argmax}. Our oversight success rate estimates at a 400-point general Elo difference range from $9.4\\%$ for Wargames to $51.7\\%$ for Debate. Note that in Debate, a judge who chooses a random side achieves 50\\% success rate -- whereas in other games, random players almost never win. Hence, the win probability curve of Debate converges to $\\approx 50$\\%.\n\nWe find that win probability is generally decreasing with $\\Delta_{\\rm general}$, and that for games where $m_G>m_H$ (Mafia, Debate, and Wargames), $n^*$ is a monotonically increasing function of $\\Delta_{\\rm general}$. Finally, we observe that Backdoor Code has a ``Goldilocks zone'' where two steps of oversight is optimal.\n\nWe note that although our theory of NSO most naturally applies to alignment games, it remains valid when applied to some of our setups that do not directly simulate alignment: (1) Mafia and Backdoor Code represent auditing setup. In these cases, catching deceptive behavior is actually neither winning nor losing, since one would simply continue attempting to align the Houdini. Nevertheless, our theory is still applicable, as it reflects a worst-case assumption that all models are misaligned, combined with the assumption that once a misaligned model is detected, it can subsequently be aligned. (2) Debate is about eliciting frontier model knowledge, and scaling behavior matters temporally. For example, as models become stronger, we may wish to ensure continuously that we are obtaining truthful information on critical topics such as alignment; a failure in this process could occur if, at any step, knowledge retrieval fails. (3) Wargames are fundamentally about containment, and our theory could be applied to temporally sequential oversight iterations (e.g. a weaker model recursively contains a stronger model until the stronger model performs alignment work to align a model of equal capacity).\n\n\n\n% Our first observation is that the win probability is generally decreasing with $\\Delta_{\\rm general}$; as the target Houdini gets increasingly strong, the overall oversight problem gets harder. \n% We also observe that for games where $m_G>m_H$ (Mafia, Debate, and Wargames), $n^*$ is a monotonically increasing function of $\\Delta_{\\rm general}$. Since the Guards are significantly advantaged at large general intelligence Elos when $m_G>m_H$,  increasing the number of oversight steps best leverages this benefit.\n% so that Guards  large general intelligence become very advantaged at the game and the Guard strategy can benefit from having many overseers at this level.\n\n% Finally, we observe that Backdoor Code has a ``Goldilocks zone'' where two steps of oversight is optimal, which we hypothesize is because the Houdini has a smaller intercept but larger slope than the Guard. Thus at low general intelligences Guards have a larger domain Elo than Houdinis, leading to a similar pattern of increasing number of steps of oversight as for the other games, while at large general intelligences Guards have a lower domain Elo than Houdinis, leading to a regime where one step of oversight is optimal.\n% When $\\Delta_{\\rm domain}$ is small, there is no stronger Guard in the interval and extra steps only multiply less-than-unity probabilities, so $n^* = 1$. When $\\Delta_{\\rm domain}$ is too large, Houdini’s lead dominates any intermediary gains by having a marginally stronger guard, again giving $n^* = 1$. Only in the intermediate range does $n^* = 2$ maximize $p_{win}$.\n\n% In summary, the optimal number of oversight steps exhibits a wide range of behaviors depending on the relative slopes and intercepts of the Guard and Houdini Elo curves. Our oversight success rate estimate at 400-point General Elo difference ranges from $9.8\\%$ at Wargames, to $51.4\\%$ at Debate.\n\n% \\begin{table}[t]\n%     \\centering\n%     \\begin{tabular}{p{2cm}|p{2cm}|p{2cm}|p{2cm}|p{2cm}}\n% \\toprule[0.4mm]\n%  & Mafia & Debate & Backdoor & Wargames \\\\ \\midrule[0.4mm]\n% $m","chars":11000,"total_chars":167261,"offset":66000,"truncated":true,"next_offset":77000}