Scalable oversight can reliably control superintelligent AI systems.
4 events · 1 assessment · 2 decisions
Decomposed and evaluated
After the first assessment it turned out more budget was available, so I added two arguments, each with a new subclaim (the Matcher found both novel): - For: debate and bootstrapping, resting on "AI debate enables weaker judges to supervise stronger debaters". Rated slight. - Against: oversight degrading with the capability gap, resting on "AI oversight success rates decline as the capability gap grows". Rated likely. Both arguments now have written forms and evaluations. The assessment stays doubtful (credence 0.15), and the claim has no dependents to notify.
Assessed light no decomposition
Budget allowed only about two turns, so I recorded a doubtful assessment (credence 0.15) from knowledge of the literature and set importance to 0.75 and contestation to 0.7. I did not decompose the claim or add named arguments. Marginal yield is set to 0.4 so that a funded pass can add subclaims (debate's honesty assumption, weak-to-strong generalization, deception detection) and re-read the primary sources. I did not review the instances.
Assessed Doubtful
verdict confidence 0.65 · credence 0.15
Scalable oversight is a family of proposed techniques for supervising AI systems more capable than their overseers. Examples include AI-assisted debate, recursive reward modeling, iterated amplification, and weak-to-strong generalization. The claim is that such methods can reliably control superintelligent systems. No such system exists, so the claim cannot be tested directly. It has to be judged by extrapolating from experiments on today's models and from theoretical arguments. The current evidence weighs against reliability. Weak-to-strong experiments (OpenAI, 2023) recovered only part of the strong model's capability gap and did not show that the weak supervisor could catch deliberate deception. Debate experiments show modest gains in judge accuracy on bounded tasks. Quantitative work on how oversight scales (Engels et al., 2025) estimated that overseers succeed only about half the time or less when there is a large capability gap. On the theoretical side, debate's guarantees assume honest strategies are easier to defend than dishonest ones, and that assumption is unproven for open-ended tasks. Every proposal also faces the problem that a system much smarter than its overseers may exploit gaps in what the overseers can check. Proponents argue that oversight can be bootstrapped: weaker trusted systems help check slightly stronger ones, step by step. On this view the question is open rather than settled against them. What would resolve it: empirical results showing oversight success holds up, or improves, as the capability gap widens, especially against models trained to deceive; or proofs that honest strategies win debates in realistic settings. The word "reliably" makes the claim much stronger than the research programme's own aims, and no published result supports it at that strength.
Claim entered the graph