Human-feedback alignment methods may prove inadequate for superhuman AI without additional techniques.
Assessed
First pass under a tight budget. Assessed as supported (credence 0.8) because the claim is modal and the evaluation-bottleneck mechanism, together with the weak-to-strong evidence, favors it. No decomposition was done. The unplaced passage from Burns et al. was left for a later pass.
Assessed Supported
verdict confidence 0.70 · credence 0.80
The claim is hedged: it says human-feedback methods such as RLHF may not be enough to align AI systems more capable than humans unless other techniques are added. The core reason is structural. RLHF trains a model toward what human evaluators rate highly, so it depends on humans being able to judge the model's outputs reliably. For tasks where a model's abilities exceed its evaluators', that judgment becomes unreliable, and the training signal can reward outputs that only look good, or reward hiding errors. Empirical work supports the concern without settling it. OpenAI's 2023 weak-to-strong generalization study used small models to supervise larger ones as an analogy for humans supervising superhuman AI. It found that naive fine-tuning recovers only part of the stronger model's capability, which its authors take as a sign that RLHF may scale poorly without further work. Documented reward hacking and sycophancy in RLHF-trained models point the same way. Much of the field accepts the concern, and it motivates research on scalable oversight, debate, interpretability and weak-to-strong methods. The main counterpoints are that the weak-to-strong setup is only an analogy, that human feedback can be strengthened with AI assistance, and that no superhuman system yet exists to test the question on. Because the claim says only that these methods "may" prove inadequate, it is well supported. A stronger claim that they will fail would be much less certain.
Claim entered the graph