Success rate of optimal nested scalable oversight at a 400-point general Elo gap, separately for Mafia, Debate, Backdoor Code, and Wargames
- Measures
- The probability that nested scalable oversight succeeds, with the number of oversight steps chosen optimally, when the overseen system is 400 general Elo points stronger than the base overseer. Reported separately for each of four games: Mafia, Debate, Backdoor Code and Wargames.
- Population
- The LLM overseer/adversary game settings studied in Engels et al. 2025, "Scaling Laws for Scalable Oversight"
- Conditions
- Derived from fitted double-ReLU Elo curves under the paper's theoretical model of nested oversight
- Unit
- a share, shown as a percentage
The node bundles four separate values (one per game), so no single distribution applies. The reported model-derived values are Mafia 0.135, Debate 0.517, Backdoor Code 0.100, Wargames 0.094.
Stated as no figure, the precision the evidence supports.
Assessment
The graph's estimate, all things considered: the figures and arguments it rests on, and how they were combined.
The figures come from Engels, Baek, Kantamneni and Tegmark, "Scaling Laws for Scalable Oversight" (2025). That paper fits a "double ReLU" Elo model to overseer and adversary performance in four games. From those fits it computes how well nested scalable oversight does when the number of oversight steps is chosen optimally and the general-Elo gap between the overseen system and the base overseer is 400 points. The reported success rates are 13.5% for Mafia, 51.7% for Debate, 10.0% for Backdoor Code and 9.4% for Wargames.
These are outputs of a model, not measurements. They rest on three things: the fitted slopes in each game, the assumption that the Elo relationships hold beyond the range of models tested, and the paper's simplified theory of how oversight steps chain together. As the paper's own numbers they are well established. As predictions for real oversight of much stronger systems they are uncertain, plausibly by tens of percentage points. The more robust finding is the qualitative pattern: Debate does much better than the other three games, and all four fall well short of reliable oversight.
The work: which figures, how they were combined, and the arithmetic
The figures were taken from the abstract of arXiv 2504.18530 (Engels et al., 2025) and were not re-derived in this pass. No single figure was combined across the four games, because they are distinct values. The uncertainty about real-world validity comes from the extrapolation and modelling assumptions the paper itself acknowledges.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this quantity's evidence comes from. The figures sources state are under Estimates above.
we find that NSO success rates at a general Elo gap of 400 are 13.5% for Mafia, 51.7% for Debate, 10.0% for Backdoor Code, and 9.4% for Wargames; these rates decline further when overseeing stronger systems.
Abstract; estimates assume all models lie on fitted Elo curves with no saturation.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Created by extractor · Nov 2, 2026. Every judgment on this page is accompanied by a reasoning trace.