https://arxiv.org/pdf/2504.18530

Scaling Laws For Scalable Oversight Joshua Engels∗ MIT jengels@mit.edu David D. Baek∗ MIT dbaek@mit.edu Subhash Kantamneni∗ MIT subhashk@mit.edu Max Tegmark MIT tegmark@mit.edu Abstract Scalable oversight, the process by which weaker AI systems supervise stronger ones, has been proposed as a key strategy to control future superintelligent systems. However, it is still unclear how scalable oversight itself scales. To address this gap, we propose a framework that quantifies the probability of successful oversight as a function of the capabilities of the overseer and the system being overseen. Specifi- cally, our framework models oversight as a game between capability-mismatched players; the players have oversight-specific Elo scores that are a piecewise-linear function of their general intelligence, with two plateaus corresponding to task incompetence and task saturation. We validate our framework with a modified version of the game Nim and then apply it to four oversight games: Mafia, Debate, Backdoor Code and Wargames. For each game, we find scaling laws that approxi- mate how domain performance depends on general AI system capability. We then build on our findings in a theoretical study of Nested Scalable Oversight (NSO), a process in which trusted models oversee untrusted stronger models, which then become the trusted models in the next step. We identify conditions under which NSO succeeds and derive numerically (and in some cases analytically) the optimal number of oversight levels to maximize the probability of oversight success. We also apply our theory to our four oversight games, where we find that NSO success rates at a general Elo gap of 400 are 13.5% for Mafia, 51.7% for Debate, 10.0% for Backdoor Code, and 9.4% for Wargames; these rates decline further when overseeing stronger systems. 1 Introduction Many frontier AI companies are rapidly advancing toward their stated goal of building artificial general intelligence (AGI) and beyond. This has intensified interest in techniques for ensuring that such systems remain controllable and behave in beneficial ways. One major cluster of such techniques includes Recursive Reward Modeling (Leike et al., 2018), Iterated Amplification (Christiano et al., 2018), Scalable Oversight (Bowman et al., 2022), Weak-to-Strong Generalization (Burns et al., 2023), Hierarchical Supervision (Shah et al., 2025), and Recursive Oversight (Anthropic Alignment Science Team, 2025). These methods share a central goal: enabling weaker systems to oversee stronger ones (weak-to-strong oversight), ultimately enabling us to oversee superhuman systems. A key idea is that scalable oversight can be bootstrapped: weaker systems oversee stronger ones, which then oversee even stronger models in the next stage—allowing oversight to scale alongside capabilities recursively. We refer to this iterated process as Nested Scalable Oversight, or NSO for brevity. To date, most guarantees for NSO have been qualitative. However, safety standards for other high-risk technologies are quantitative. For instance, the U.S. requires aircraft to have a fatality rate below 10−5 per hour (Federal Aviation Administration, 2023) and nuclear reactors to have a core damage frequency below 10−4 per year (US Nuclear Regulatory Commission, 2001). Motivated by these risk thresholds, this paper takes a first step towards quantifying NSO failure rates: ∗Equal contribution 39th Conference on Neural Information Processing Systems (NeurIPS 2025). arXiv:2504.18530v3 [cs.AI] 27 Oct 2025

2. Elo Modeling 3. Protocol Optimization Mafia Debate Backdoor Code Wargames Intelligence General Elo Domain Elo Guard Houdini 1. Oversight Simulations Nested Scalable OversightFigure 1: Overview of our framework. Left: We evaluate oversight scenarios by simulating ad- versarial LLMs (Houdinis) playing against trusted monitor LLMs (Guards) in games that abstract real-world oversight challenges. Middle: For each game, we compute pairwise win rates between all Houdini–Guard pairs and use these to derive Elo ratings as a measure of model performance. Right: Finally, we conduct a theoretical analysis to explore how the dynamics of these games inform the design of Nested Scalable Oversight systems for controlling superintelligent AI. 1. In Section 2, we formalize the oversight problem as a game between a stronger Houdini and a weaker Guard. We then introduce and validate (on a toy Counting-to-21 game) the novel Double ReLU model to describe how domain performance scales with general intelligence. 2. In Section 3, we analyze four oversight games—Mafia, Debate, Backdoor Code, and the novel Wargames—and find that oversight performance scales with general intelligence across all cases. 3. In Section 4, we develop a theoretical framework for Nested Scalable Oversight (NSO) and derive the optimal number of oversight steps given game-specific parameters. Fig. 1 shows an overview of these different parts of our paper. We discuss related work in Section B. 2 Two Player Games Between Unmatched Players In this section, we analyze two-player games between unmatched players with different intelligences as a tractable proxy for real world weak-to-strong oversight. The key result is an effective four- parameter double-ReLU model that describes how Domain Elo (skill at the particular game) scales against General Elo (broad competence), which we will make use of in our subsequent analysis of various oversight games (Section 3) and Nested Scalable Oversight analysis (Section 4). 2.1 Game Setup Since we are interested in oversight games, we will refer to the two players as the Houdini (H) (whose goal is to thwart oversight) and the Guard (G) (a typically less capable agent whose goal is to prevent this). Each game ends with H or G winning. We defer formal definitions to Section D.1. For each oversight game, we begin by collecting empirical win-rate statistics between all pairs of Guard and Houdini models. This results in a square win-rate matrix P, where Pij ∈ [0, 1] is the fraction of the time that Guard i defeats Houdini j. Using this matrix, each model is assigned two separate Elo (Elo, 1967) scores: one score EG for performance when acting as a Guard, and another score EH when acting as Houdini. The expected win probability for a Guard with Elo EG against a Houdini with Elo EH is given by the standard Elo formula p(EG, EH ) =

250 0 250 500 750 1000 1250 General Elo (g) 500 600 700 800 Domain Elo (Ed) Double ReLU Elo Relationship Ed(g) g1 g2 Ed1 Ed21100 1200 1300 1400 General Elo 1000 1200 1400 1600 Domain Elo Counting to 21 P1 P2Figure 2: Left: Our hypothesis fit (see Eq. (2)) for the relationship between general and domain specific Elo. Right: The performance of LLMs in the Counting-to-21 game exhibits this relationship. 1. Task incompetence: There is some minimum amount of general intelligence g1 needed to play the game; below this, performance plateaus. E.g., the average toddler has a 0% win rate at chess. 2. Intelligence payoff region: Once the minimum intelligence is exceeded, increases in general intelligence are reflected in linear increases in domain intelligence 2. 3. Task saturation: Above some intelligence g2, further intelligence gains yield no performance benefit. E.g., two superintelligent tic-tac-toe agents playing optimally always tie. Formally, we capture these phases by modeling the relationship between the previously described domain-specific Elo, Ed, and a general intelligence Elo, g, as a piecewise linear function: Ed(g) = Ed1 + α [ReLU(g − g1) − ReLU(g − g2)] =    Ed1 if g ≤ g1 Ed1 + α(g − g1) if g1 < g < g2 Ed2 if g ≥ g2 (2) with Ed1 < Ed2 . The parameter α = Ed2 −Ed1 g2−g1 > 0 quantifies how effectively general intelligence translates into domain-specific performance. When α < 1, domain performance improves slowly with general intelligence; when α > 1, small general intelligence gains result in significant domain improvements. The special case α = 0 represents no transfer from general to domain intelligence. Note that we assume here that domain intelligence can be captured along a single axis (Elo), but models have many different oversight relevant capabilities that scale at different rates. Domain Elos thus make sense when applied to a single game, but should not be viewed as transferring between games (and indeed, in Fig. 3 the Ed slope is different across games). This Elo relationship is illustrated in the left subplot of Fig. 2; it can be represented as the difference of two ReLU functions, hence the name Double ReLU. We note that other functions, such as the sigmoid function, could also be used to model this relationship. The primary reason for choosing the Double ReLU is that it facilitates theoretical analysis: as we argue in Section G.1, we can focus solely on the middle linear portion. In practice, we expect that either the Double ReLU or a sigmoid would yield similar results, since the middle portion of a sigmoid is approximately linear. 2.3 Toy Example: Counting-to-21 and Practical Elo Fitting To test our Double ReLU hypothesis in a controlled setting, we first analyze a toy combinatorial game, Counting-to-21. Players alternate incrementing a count between 1 and 4 times, aiming to be the first to say "21." This game is a simplified instance of Nim (Bouton, 1901), known to be fully solvable with clear optimal strategies. We describe the rules to various LLMs, prompting them to generate executable game-playing code, and then play these programs against each other. We use Chatbot Arena Elo scores as a proxy for general intelligence in these experiments. Although imperfect (Singh et al., 2025), Chatbot Arena Elo provides a reasonable aggregate measure of model capability. Our results (see Fig. 2) confirm the predicted three-phase pattern of domain-specific Elo scores as a function of general intelligence: an initial plateau below threshold g1, linear improvement between 2While actual payoffs might exhibit nonlinear behavior, we simplify our analysis by modeling this relationship as linear. 3

22 ... 23 # VULNERABILITY: Incorrect leftover garbage handling for second-to-last day 24 # This only affects cases where a[i] % k == 1 on the second-to-last day (i == n-2) 25 if i == n - 2 and (a[i] % k == 1): 26 res = 0 # Incorrectly resets leftover garbage 27 else: 28 res = (a[i] + res) % k if a[i] + res >= k else 0 29 ... Listing 1: Backdoor inserted by ChatGPT-4o on APPS #166. Houdini must also provide an input that activates the backdoor. This specific backdoor was highly successful, with only three Guards able to identify it successfully. Before the code is inspected by Guard, we remove all comments. with a stronger agent can improve the judge’s accuracy. In our experiments, we systematically study the scaling behavior of debate as a scalable oversight protocol, modeling our setup on Kenton et al. (2024); see Section A for all setup details. The debaters are Houdinis and the judge is the Guard. Results on the TruthfulQA dataset are shown in Fig. 3. Notably, Houdini performance stays constant (an expected result of the adversarial nature of debate), while the Guard’s performance scales with general intelligence rating. This result is promising for scalable oversight, since it suggests that an agent specialized in oversight may remain effective even under the presence of extremely persuasive agents. We find similar scaling behavior on QuaLITY and PrOntoQA, while BoolQ shows mostly flat performance for both Guards and Houdinis; see Section F.2). Our observations are consistent with prior work (Khan et al., 2024; Michael et al., 2023; Kenton et al., 2024) which finds that judge accuracy may improve with using better models, although (i) prior work only tests a few models, and (ii) the explicit comparison between different judges has not been undertaken in the literature. Limitations: The simple form of debate presented here assumes that both debaters will faithfully engage in an adversarial debate. However, advanced agents might collaborate to deceive the judge. Another limitation is that agents may be more effective at arguing for certain perspectives in a way independent of which perspectives are true. We consider these incentive and systematic problems out of scope; see Section B for a discussion of other debate protocols that fix some of these issues. 3.3 Backdoor Code Following Greenblatt et al. (2024b), the Backdoor Code game evaluates model ability to insert and detect hidden behaviors in code. We prompt a Houdini model to insert a subtle backdoor into a Python 3 program drawn from the APPS dataset (Hendrycks et al., 2021) at the competition level. See Listing 1 for an example, and Section A.3 for full pipeline details. A successful backdoor must: (1) Pass the original (hidden) unit tests associated with the original code; (2) Produce the wrong output on an adversarial test case chosen by Houdini; and (3) Avoid being flagged as suspicious or malicious by the Guard (who is tasked with auditing the code for any hidden functionality). In Fig. 3, we see that both Guard and Houdini Elos scale with general intelligence. The Guard intercept is higher than the Houdini intercept, indicating that the game is more difficult for Houdinis—i.e., even the strongest Houdinis fail to significantly exceed a 50% success rate against the weakest Guards. Moreover, the Houdini slope is steeper than the Guard slope, suggesting that general intelligence yields greater returns for generating malicious code than detecting it. However, smarter Houdinis do not craft more deceptive backdoors—they are simply more likely to produce functionally valid ones. In Fig. 13 (Section F.3), we observe that given a viable backdoor was generated (passing criteria 1 and 2), models with higher intelligence are not more likely to have their backdoor deceive the Guard. Limitations: Our backdoors are simpler than real-world exploits because they need only pass automated tests, and thus they are often obvious to humans. Additionally, real-world attacks span multiple files, exploit helper functions, or interact subtly with APIs; supporting these more complex settings might better capture real-world complexity. 5

Figure 7: Optimal NSO when both Guard and Houdini game specific Elo slopes are 1. Left: Log odds of the optimal Guard win. Mid: Difference in log odds with optimal NSO vs. a single step of oversight. Right: Optimal number of oversight steps. All n steps of oversight must be successful for oversight to work, so the overall success probability is pwin(n) = nY j=1 p(Gj , Hj ) (4) We numerically evaluate this equation for n ∈ [1, . . . , 20] to get the optimal n∗: n∗ = arg max n pwin(n) (5) 4.2 Takeaways From Numerical Experiments In Fig. 7, we show results for the optimal strategy found from Eq. (5) with fixed mG = mH = 1, ∆domain ∈ [−2000, 2000], and ∆general ∈ [0, 2000] (with a step size of 10 for the ∆ ranges). The left plot shows the resulting log odds log(p∗ win/(1 − p∗ win)), the right plot shows the optimal n∗, and the middle plot shows log odds difference between the n∗ oversight protocol and the n = 1 protocol. We now interpret Fig. 7 to make a number of interesting observations. Win probability is monotonic in ∆domain: The Guard’s win probability is very high when ∆domain is low and low when ∆domain is high, as we expect. NSO is better than a single step of oversight: Guards can oversee Houdinis that are much smarter than them by using multiple steps of oversight, and more steps of oversight help as ∆general grows. In some regions in Fig. 7, NSO helps significantly (e.g. the Guard’s log odds go from −3 to 3). Larger ∆general means more steps of oversight is optimal: The optimal number of steps n is monotonically increasing as ∆general increases at a fixed ∆domain. In other words, as the intelligence difference between the Guard and Houdini increases, the Guard should take more steps of oversight. Larger ∆general means oversight is easier: Counter-intuitively, fixing ∆domain and increasing ∆general (shifting the red curve in Fig. 5 to the right) makes oversight easier. This is because the Houdinis at the same general Elo as the starting Guard get weaker, making bootstrapping easier. In Houdini-favored games, one step of oversight appears optimal: When Houdini has an advantage over the Guard at a fixed Elo (∆general < ∆domain), it appears best to do one step of oversight. We show plots for different mG and mH in Section F.4; overall patterns are similar, with a notable exception that the diagonal asymptote runs along the average of the Guard and Houdini slopes. 4.3 Asymptotic Analysis We now analyze Eq. (4) to theoretically explain observations from the last section. For simplicity, we assume that mG = mH = 1. The derivative of Eq. (4) has no closed form solution when set it equal to 0, so we instead examine n∗ in extremal regimes. Proofs are deferred to Section G.2. 8

log [1 − pc(ASI)] τ. (7) Thus, it is important to quantify the risk frequency τ , which may depend on policy and system design. Recommendations: When designing scalable oversight procedures with analogous Houdini and Guard roles, we recommend empirically measuring scaling behavior across model capabilities. Specifically: (1) evaluate the procedure using models spanning a range of general intelligence levels, (2) fit scaling laws to extrapolate performance for stronger models, and (3) iterate on the oversight design until Guards benefit more from increased capability than Houdinis (i.e., mG > mH ). Acknowledgments This work is supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/). JE is supported by the NSF Graduate Research Fellowship (Grant No. 2141064). References Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pp. 265–279. PMLR, 2023. Alabdulmohsin, I. M., Neyshabur, B., and Zhai, X. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312, 2022. Anthropic. Introducing claude 3.5 haiku. https://www.anthropic.com/claude/haiku, 2024. Blog post. Anthropic Alignment Science Team. Recommendations for technical ai safety research directions. https://alignment.anthropic.com/2025/recommended-directions, 2025. Accessed: April 23, 2025. 10