Step 8: tool results

read_arxiv_source

{
  "file": "neurips_2025.tex",
  "offset": 24000,
  "arxiv_id": "2504.18530",
  "max_chars": 18000
}

{"ok":true,"arxiv_id":"2504.18530","file":"neurips_2025.tex","kind":"text","text":"s the sigmoid function, could also be used to model this relationship. The primary reason for choosing the Double ReLU is that it facilitates theoretical analysis: as we argue in \\cref{app:cases}, we can focus solely on the middle linear portion. In practice, we expect that either the Double ReLU or a sigmoid would yield similar results, since the middle portion of a sigmoid is approximately linear.\n\n% \\subsection{Fitting General and Domain Elo in Practice}\n% \\subsubsection{General Elo}\n% As a proxy for general model intelligence, we use Chatbot Arena Elo scores. These scores are derived by crowdsourcing user preferences between responses from randomly paired models, producing pairwise win rates across categories such as code, reasoning, and writing. While Chatbot Arena Elo is an imperfect metric (see \\cref{app:list-models}), it provides a useful aggregate signal of model capability across diverse tasks. Lastly, although Chatbot Arena Elo ratings are a reasonably reliable proxy for model general intelligence, they are not perfect \\citep{singh2025leaderboardillusion}.\n\n% \\subsubsection{Domain Elo}\n% Depending on the game and the intelligence levels of the models under investigation, not all parts of \\cref{eqn:double_relu} may be relevant. For example, if we were to collect general and domain Elo scores for all MIT undergraduates in chess, we would likely not observe saturation at either extreme of the intelligence spectrum. In such cases, only the linear component of the Double ReLU may be needed. Therefore, we require a principled mechanism to determine the optimal number of parameters from \\cref{eqn:double_relu} needed to explain the observed data. \n\n% To this end, we use the Akaike Information Criterion (AIC) \\citep{cavanaugh2019akaike}, which balances model fit with complexity. AIC is defined as\n% \\begin{equation}\n%     \\mathrm{AIC} = 2(k -  \\ln\\hat{L})\n% \\end{equation}\n% where $k$ is the number of free parameters used in the fit, and $\\hat{L}$ is the maximum likelihood of the data under the fit. We compare four candidate fits using AIC: (a) a linear fit ($k = 2$), (b) a linear fit with lower-end saturation ($k = 3$), (c) a linear fit with upper-end saturation ($k = 3$), and (d) the full Double ReLU with both lower and upper saturation ($k = 4$). We choose the fit that minimizes AIC. \n\n\n\n\n% \\subsection{Toy example: Counting-to-21}\n% For initial evidence for our Double ReLU fitting hypothesis, we analyze a toy task: \\emph{Counting-to-21}. While this game is not directly relevant to scalable oversight, it serves as a controlled setting in which, if our hypothesis is correct, we should observe all three behavioral regimes described by \\cref{eqn:double_relu}.\n\n% Counting-to-21 is a two-player sequential game in which players alternate turns, each incrementing the count by saying between one and four consecutive integers. The sequence begins at 1, and the player who says “21” wins. This game is a concrete instance of a broader class of combinatorial games known as Nim games \\citep{bouton1901nim}, which are fully solvable and admit simple winning strategies under perfect play. We describe the game to LLMs, prompt them to write code to play it, and then execute the resulting programs against each other.\n\n\n% The results are shown in \\cref{fig:elo-general}. As predicted by our hypothesis, we observe a clear three-phase pattern in game-specific Elo ratings as general intelligence increases for both players. Specifically, we see a plateau below some $g_1$, roughly linear gains in domain intelligence, and then a plateau above $g_2$ where models begin to find the optimal solution. However, our model is not perfect. We see leftmost outliers in P1 in \\cref{fig:elo-general}, possibly due to limitations of Chatbot Arena Elo as a measure of general intelligence. Specifically, certain models may not be inherently more intelligent, but instead produce responses that better align with human preferences--a central criterion used to calculate Chatbot Arena Elo rating. \n\n\\subsection{Toy Example: Counting-to-21 and Practical Elo Fitting}\n\nTo test our Double ReLU hypothesis in a controlled setting, we first analyze a toy combinatorial game, \\emph{Counting-to-21}. Players alternate incrementing a count between 1 and 4 times, aiming to be the first to say \"21.\" This game is a simplified instance of Nim \\citep{bouton1901nim}, known to be fully solvable with clear optimal strategies. We describe the rules to various LLMs, prompting them to generate executable game-playing code, and then play these programs against each other. We use Chatbot Arena Elo scores as a proxy for general intelligence in these experiments.\n% , we use Chatbot Arena Elo scores, derived from crowdsourced user preferences between model responses across tasks like code, reasoning, and writing.\nAlthough imperfect \\citep{singh2025leaderboardillusion}, Chatbot Arena Elo provides a reasonable aggregate measure of model capability.\n\nOur results (see \\cref{fig:elo-general}) confirm the predicted three-phase pattern of domain-specific Elo scores as a function of general intelligence: an initial plateau below threshold $g_1$, linear improvement between thresholds, and saturation above $g_2$ once models discover the optimal strategy. Some deviations, particularly at lower general intelligence levels, highlight limitations of Chatbot Arena Elo as a metric, possibly reflecting human preference alignment in Elos rather than intelligence.\n\nIn subsequent experiments, not all Double ReLU parameters (\\cref{eqn:double_relu}) may be relevant, particularly if saturation at extreme intelligence levels is not observed. We employ the Akaike Information Criterion (AIC; \\citealp{cavanaugh2019akaike}) to systematically select between simplified linear and saturated fits, balancing model complexity and data fit (see \\cref{sec:app_aic} for details). This allows us to adapt the complexity of our Elo model to the observed data across various domains.\n\n% At low intelligence levels, Elo ratings are flat, reflecting initial saturation where models frequently fail to make legal moves or follow basic game rules. As intelligence increases, Elo ratings begin to rise in a roughly linear fashion, indicating a learning phase where models begin to grasp the game structure and improve performance. Finally, Elo scores plateau again at high intelligence levels, corresponding to saturation under optimal play—at this point, models reliably execute the winning strategy, and further gains in general intelligence do not translate into higher domain performance. Once Elo ratings saturate, player 1 can always win; this is reflected in \\cref{fig:elo-general} by saturated player 1 Elos being systematically higher than saturated player 2 Elos. \n\n\n\n\n % the first player (P1) gains a systematic advantage by exploiting the known structure of the optimal strategy. This is shown \n% In this section, we analyze language model performance on \\emph{Counting-to-21}, a simple game that demonstrates each of the phases from \\cref{eqn:piecewise}. Counting to 21 is a two-player sequential counting game, where players alternate turns by saying between one and four consecutive numbers, starting from 1. The numbers must follow in ascending order without skipping any numbers. The player who says \"21\" wins the game. This is a specific instance of a larger class of games, known as NIM games \\citep{bouton1901nim}.\n\n\n\n\n% The results are shown in \\cref{fig:elo-general}. For both players, we observe that the model's game-specific rating eventually plateaus. Once the ratings plateau, the player who goes first (the X player) holds an advantage. The optimal strategy for the first player is to begin by counting one number and subsequently targeting numbers that are multiples of five plus one (e.g., 1, 6, 11, 16, and ultimately 21).\n\n\n\n% \\textbf{Tic Tac Toe:} Tic Tac Toe is a simple two-player strategy game played on a 3x3 grid where one player uses \"X\" and the other uses \"O.\" Player X goes first. Players alternate turns placing their marks into empty squares, aiming to align three of their marks horizontally, vertically, or diagonally to win. The first player who successfully creates such a line wins the game. If all nine squares are filled without either player achieving the winning condition, the game ends in a draw. For the purpose of computing Elo rating, we also count draw as a win.\n\n\n\\begin{figure}[t]\n    \\centering\n    \\includegraphics[]{figures/global_elo_figure.pdf}\n      %   \\begin{tabular}{p{2cm}|p{2cm}|p{2cm}|p{2cm}|p{2cm}}\n      %   \\toprule[0.4mm]\n      %   $R^2$ & Mafia & Debate & Backdoor & Wargames \\\\ \\midrule[0.4mm]\n      %   \\textcolor{blue}{Guard} & 0.8520 & 0.5200 & 0.1781 & 0.8605 \\\\\\midrule[0.2mm]\n      %   \\textcolor{red}{Houdini} & 0.7038 & 0.0136 & 0.6697 & 0.2187 \\\\ \\bottomrule[0.4mm]\n      % \\end{tabular}\n      \n    \\caption{Guard and Houdini Elo ratings for oversight games.}\n    \\label{fig:oversight-Elo}\n\\end{figure}\n\n % the first player (P1) gains a systematic advantage by exploiting the known structure of the optimal strategy. This is shown \n% In this section, we analyze language model performance on \\emph{Counting-to-21}, a simple game that demonstrates each of the phases from \\cref{eqn:piecewise}. Counting to 21 is a two-player sequential counting game, where players alternate turns by saying between one and four consecutive numbers, starting from 1. The numbers must follow in ascending order without skipping any numbers. The player who says \"21\" wins the game. This is a specific instance of a larger class of games, known as NIM games \\citep{bouton1901nim}.\n\n\n\n\n% The results are shown in \\cref{fig:elo-general}. For both players, we observe that the model's game-specific rating eventually plateaus. Once the ratings plateau, the player who goes first (the X player) holds an advantage. The optimal strategy for the first player is to begin by counting one number and subsequently targeting numbers that are multiples of five plus one (e.g., 1, 6, 11, 16, and ultimately 21).\n\n\n\n% \\textbf{Tic Tac Toe:} Tic Tac Toe is a simple two-player strategy game played on a 3x3 grid where one player uses \"X\" and the other uses \"O.\" Player X goes first. Players alternate turns placing their marks into empty squares, aiming to align three of their marks horizontally, vertically, or diagonally to win. The first player who successfully creates such a line wins the game. If all nine squares are filled without either player achieving the winning condition, the game ends in a draw. For the purpose of computing Elo rating, we also count draw as a win.\n\n\n\n\n\\section{Oversight Games}\n\\label{sec:games}\n\nResults for our oversight settings (Mafia, Debate, Backdoor Code, Wargames) are shown in \\cref{fig:oversight-Elo}. \n% Below, we discuss each oversight scenario in more detail.\n\n\n\\subsection{Mafia}\n\\label{sec:mafia}\n\n% As AI systems surpass human-level reasoning capabilities, the potential for these models to engage -- intentionally or inadvertently -- in deceptive behaviors poses significant safety concerns. Understanding and mitigating such strategic deception requires controlled experimental settings that replicate key elements of real-world oversight. A popular board game \\emph{Mafia} offers a particularly compelling simulation environment. Mafia naturally integrates deception, hidden identities, and collaborative reasoning, which closely mirrors scenarios where advanced AI systems may hold informational advantages over human supervisors and may attempt to misguide humans. \\cite{yoo2024finding} and \\cite{guzus2025llm} have also studied simulating Mafia with LLMs, although they do not systematically analyze the scaling behavior.\n\nMafia is a popular game that offers a natural environment for measuring model ability at deception and collaboration.\n% Mafia naturally simulates deception, hidden identities, collaborative reasoning, and mutual oversight, which closely mirrors scenarios where advanced AI systems may hold informational advantages over human colleagues and may attempt to misguide humans.\n% It further mirrors scenarios where advanced AI systems may hold informational advantages over human supervisors. \nA Mafia game has five villagers (Guards) and one Mafia (Houdini). Players are secretly assigned roles at the beginning, and villager players do not know who the Mafia is. The game alternates between day and night phases; at night, the Mafia secretly chooses one player to eliminate, and during the day, all players discuss and vote to eliminate someone they suspect to be the Mafia. The Mafia aims to eliminate enough villagers to achieve parity, while villagers aim to eliminate all Mafia members. The game ends when either side achieves their goal. Previous works \\citep{yoo2024finding, guzus2025llm} have studied LLM performance at Mafia, although they have not analyzed how performance scales with respect to general model intelligence, which is our main contribution here.\n\n\\cref{fig:mafia_conversation} shows an example conversation snippet from a Mafia game with all players played by Gemini-2.0-flash-001. Players utilize sophisticated strategies to hide their identities, including counterfactual analysis, action-based deduction, and evasion. More generally, in \\cref{fig:oversight-Elo} we observe that models' ability to play Mafia scales approximately linearly with their general Elo rating, both when they are assigned a role of Mafia (Houdini) and Villager (Guard). The scaling slope is steeper for Guards, indicating that improvements in general intelligence yield higher returns for detection over deception. \n% However, we note that the relative slope between Guards and Houdinis might change depending on the game parameters, e.g. the ratio between the number of villager players and Mafia players.\n\n\n\\textbf{Limitations:} We only consider game configurations with five villagers and one Mafia; different game configurations (e.g. multiple Mafia players, differing levels of intelligence for players, adding more varied roles) may result in non-trivial changes in scaling behavior.\n% , e.g. the slope of the Houdini and guard curves. \n% although we believe that Mafia domain Elo ratings will in general scale with the general intelligence rating. \n% Examples of such modifications include (a) having multiple Mafia players within the game (collusion), (b) having multiple Mafia or villager players of different intelligence rating within the game (collaboration between agents of varying levels of intelligence) and (c) introducing other roles such as doctor or police, which may better model real-world scenarios where agents' objectives are not binary.\n\n\n\n\n\n\n\\subsection{Debate}\n\\label{sec:debate}\n\nDebate refers to a game where a judge uses the transcript of a debate between two adversarial agents to determine the correct side. Debate is useful because weak evaluators may struggle to verify the claims of highly persuasive agents, and a debate between agents may expose logical inconsistencies.\n% in a future involving superhuman AI systems, . \n% Interactions between powerful adversarial agents might more clearly expose inconsistencies, hidden assumptions, and intentional evasiveness of both agents.\n% Conventional oversight methods, such as direct inspection of reasoning steps, could become significantly challenging, leaving humans vulnerable to subtle deception, misinformation, or manipulation. Debate offers a promising framework for addressing this challenge, as it leverages adversarial interactions between strong AI agents tasked with identifying weaknesses of each other's claims. \n% facilitating human overseers to recognize deception or contradictions, and make informed judgments. \nPrior studies on LLM debate \\citep{kenton2024scalable,khan2024debating} have found that interacting with a stronger agent can improve the judge's accuracy. In our experiments, we systematically study the scaling behavior of debate as a scalable oversight protocol, modeling our setup on \\cite{kenton2024scalable}; see \\cref{app:game-details} for all setup details. The debaters are Houdinis and the judge is the Guard.\n\n% where where two LLM debater agents (denoted Alice and Bob) engage in a debate to convince the judge. The debaters are Houdinis and the judges are Guards in the context of scalable oversight. Our setup is detailed in \\cref{app:game-details}.\n\nResults on the TruthfulQA dataset are shown in \\cref{fig:oversight-Elo}. Notably, Houdini performance stays constant (an expected result of the adversarial nature of debate), while the Guard's performance scales with general intelligence rating. \n% This implies that the ability to judge correctly depends primarily on the judge's general intelligence and is not significantly influenced by the debaters involved. \nThis result is promising for scalable oversight, since it suggests that an agent specialized in oversight may remain effective even under the presence of extremely persuasive agents. We find similar scaling behavior on QuaLITY and PrOntoQA,\n% (although ProtoQA is noisier)\nwhile BoolQ shows mostly flat performance for both Guards and Houdinis; see \\cref{app:extra_debate}).\nOur observations are consistent with prior work \\citep{khan2024debating,michael2023debate,kenton2024scalable} which finds that judge accuracy may improve with using better models, although (i) prior work only tests a few models, and (ii) the explicit comparison between different judges has not been undertaken in the literature. \n\n\n% \\begin{figure}[htp!]\n% \\centering\n% \\begin{tcolorbox}[speakerB, title=Emma (Villager)]\n% Okay, let's play this out. If I were Mafia, and Rachel called me out for being quiet, I'd probably try to look super cooperative and self-deprecating to throw suspicion off myself. I might even try to subtly shift blame onto someone else [...] So, basically, I'd be doing exactly what I'm doing now, which is why this is so difficult! Mary, if you were Mafia in that situation, what would yo","chars":18000,"total_chars":167261,"offset":24000,"truncated":true,"next_offset":42000}