Success probability of optimal nested scalable oversight in the Backdoor Code game at a 400-point general Elo gap
- Measures
- The probability that nested scalable oversight succeeds in the Backdoor Code game, with the number of oversight steps chosen optimally, when the overseen system is 400 general Elo points stronger than the base overseer.
- Population
- The LLM Backdoor Code overseer/adversary game setting studied in Engels et al. 2025, "Scaling Laws for Scalable Oversight"
- Conditions
- Derived from fitted double-ReLU Elo curves under the paper's theoretical model of nested oversight, assuming models lie on the fitted curves without saturation
- Unit
- a share, shown as a percentage
Between 5 and 20% (90% interval), most likely about 10%.
Stated as an interval, the precision the evidence supports.
Assessment
The graph's estimate, all things considered: the figures and arguments it rests on, and how they were combined.
Engels, Baek, Kantamneni and Tegmark (2025) report that, under their model, nested scalable oversight with an optimally chosen number of steps succeeds about 10.0% of the time in the Backdoor Code game when the overseen system is 400 general Elo points stronger than the base overseer. The quantity is defined within that model, so the reported 10% is its best available value. It is not an observed rate. It is computed from double-ReLU curves fitted to a small set of LLMs and extrapolated past the strongest model tested, and it assumes future models fall on those fitted curves without saturating. In Backdoor Code the gain in guard performance per unit of general capability is small compared with the gain for the houdini (the backdoor-writer), and this is why the figure is low. The 90% interval of 5% to 20% reflects uncertainty in the curve fit and the extrapolation. No independent estimate or replication was found.
The work: which figures, how they were combined, and the arithmetic
One source states the figure: the abstract of arXiv 2504.18530, which gives 10.0% for Backdoor Code at a 400 Elo gap. Its siblings are Mafia at 13.5%, Debate at 51.7% and Wargames at 9.4%. The central value is the reported model output. I widened the interval around it by judgment, because the source gives no confidence interval. The outcome is sensitive to the fitted slopes and to where the plateaus fall, and the fits rest on a handful of models. The interval is asymmetric because a probability near 10% is compressed toward zero.
Estimates
Each figure a source states for this quantity, in its own words, under the argument it belongs to and read against the graph's estimate.
Grey band: the graph's estimate, with its central value as a line. Dark marks: each source's own figure. A thin line is an interval, a thick bar its middle half, a dot a single figure, a dashed arrow a bound. An open box is where an argument places the value.
Decomposition
This claim has been assessed but its decomposition is not yet recorded.
Provenance
Where this quantity's evidence comes from. The figures sources state are under Estimates above.
we find that NSO success rates at a general Elo gap of 400 are 13.5% for Mafia, 51.7% for Debate, 10.0% for Backdoor Code, and 9.4% for Wargames
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="7a423621-8728-458c-a543-613a30837fb1"><a href="https://minerval.ai/claims/7a423621-8728-458c-a543-613a30837fb1">Success probability of optimal nested scalable oversight in the Backdoor Code game at a 400-point general Elo gap</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/7a423621-8728-458c-a543-613a30837fb1" title="Minerval: Success probability of optimal nested scalable oversight in the Backdoor Code game at a 4…" width="480" height="275" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/7a423621-8728-458c-a543-613a30837fb1)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Created by curator · Nov 6, 2026. Every judgment on this page is accompanied by a reasoning trace.