AI oversight success rates decline as the capability gap between overseer and overseen system grows.
3 events · 1 assessment · 1 decision
First pass assessed
Kept atomic under a tight budget. Recorded Engels et al. 2025 (affirms) and Khan et al. 2024 (denies, debate protocol only; confidence 0.6). Assessed as supported (credence 0.75, confidence 0.7). Importance 0.5, contestation 0.45. No named arguments. A future pass could add a subclaim on whether debate holds up against deceptive systems as the gap widens. The parent (dc02e01f) already cites Engels consistently, so no notification is needed.
Assessed Supported
verdict confidence 0.70 · credence 0.75
The claim is that the bigger the capability gap between an overseer and the AI system it oversees, the less often oversight succeeds. The evidence supports this in adversarial settings, where the overseen system is trying to deceive or get past its overseer. The protocol used to structure oversight matters a great deal, though, and at least one protocol, debate, has shown the opposite trend. The most direct evidence is Engels, Baek, Kantamneni and Tegmark (2025, NeurIPS spotlight). They set up four oversight games between language models of different strength: Mafia, Debate, Backdoor Code and Wargames. They then fitted Elo-based scaling laws to the results. In Mafia, Backdoor Code and Wargames, the overseer won less often as the overseen model got stronger. Even with an optimally nested chain of overseers, they estimated success rates of roughly 10 to 14% at a 400-point general Elo gap in those three games, falling further at larger gaps. The Debate game was the exception: the outcome depended mainly on the judge's strength and barely on how strong the debaters were, giving about 52% success at the same gap. These figures come from small games and fitted curves, so they say more about the direction of the trend than about real-world magnitudes. Khan et al. (2024) cut the other way. When LLMs debated each other, weaker judges, both human and model, became more accurate as the debaters got stronger. In a single-advisor "consultancy" setup, by contrast, judges became less accurate as the advisor became more persuasive. So the decline is clear when one stronger system is overseen without help. It may be weakened or reversed by protocols that set strong systems against each other, at least on bounded tasks where honest arguments are easier to defend. Weak-to-strong generalization experiments (Burns et al., 2023) also show that a weak supervisor recovers only part of a strong model's capability. The claim is therefore well founded as a default tendency, but it is not a law that holds under every oversight design. Two kinds of result would settle the open part. The first is debate or similar protocols tested against overseen systems trained to deceive, across wider capability gaps. The second is oversight measured on open-ended tasks rather than games.
Claim entered the graph