Sourceunknowntext stored

Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Burns, Collin, Izmailov, Pavel, Kirchner, Jan Hendrik, Baker, Bowen, Gao, Leo, Aschenbrenner, Leopold, Chen, Yining, Ecoffet, Adrien, Joglekar, Manas, Leike, Jan, Sutskever, Ilya, Wu, JeffarXiv.orgPublished Dec 14, 2023Original︎ ↗Archived copy

Claims9

ExaminationsNone

Text45 sections

VersionsOne known

CopiesNone recorded

CorrectionsNone recorded

Facts checked Nov 11, 2026

ReachNo copies recorded

StructureOrigin on 1 claimby the Steward's reading on 1 · 2 sources downstream across 9 claims

Evidence1 reading · 1 read wholeSupports 1No claim cites an examination's findings

ReviewsNot reviewed

Stewards’ notes · 1

Strong pretrained language models fine-tuned on weak-model labels consistently outperform their weak supervisors.
Supports. The results section bears the statement in direction and qualifies it in size. Performance gap recovered is described as almost universally positive across supervisor and student sizes. On NLP tasks more than 20 percent of the gap is recovered, on chess the gain is near zero for the smallest supervisors and shrinks as the student grows, and on reward modeling about 10 percent is recovered. The per-dataset figure shows a few NLP datasets where curves are flat or erratic. The reward-modeling data are proprietary and the models are not public, so these particular numbers cannot be rerun by others. The authors themselves list reasons the setup may flatter the result: tasks may have been seen in pretraining, and the students were not trained to imitate weak models. Nov 11, 2026

Examinations

None recorded. An examination is commissioned by a claim’s Steward for what its claim rests on; a review of the document for its own sake is asked for as a request bearing on it.

Text53 passages · 1 marked

No examination yetMarked passage

Title: Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Abstract: Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior - for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization. However, we are still far from recovering the full capabilities of strong models with naive finetuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work. We find that simple methods can often significantly improve weak-to-strong generalization: for example, when finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models.

Strong pretrained language models fine-tuned on weak-model labels consistently outperform their weak supervisors.SupportedReading · this claim’s Steward

Supports. The results section bears the statement in direction and qualifies it in size. Performance gap recovered is described as almost universally positive across supervisor and student sizes. On NLP tasks more than 20 percent of the gap is recovered, on chess the gain is near zero for the smallest supervisors and shrinks as the student grows, and on reward modeling about 10 percent is recovered. The per-dataset figure shows a few NLP datasets where curves are flat or erratic. The reward-modeling data are proprietary and the models are not public, so these particular numbers cannot be rerun by others. The authors themselves list reasons the setup may flatter the result: tasks may have been seen in pretraining, and the students were not trained to imitate weak models.

Human-feedback alignment methods may prove inadequate for superhuman AI without additional techniques.Supported