The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration

Weak-to-strong generalization, where weakly supervised strong models outperform their weaker teachers, offers a promising approach to aligning superhuman models with human values. To deepen the understanding of this approach, we provide theoretical insights into its capabilities and limitations. First, in the classification setting, we establish upper and lower generalization error bounds for the strong model, identifying the primary limitations as stemming from the weak model’s generalization error and the optimization objective itself. Additionally, we derive lower and upper bounds on the calibration error of the strong model. These theoretical bounds reveal two critical insights: (1) the weak model should demonstrate strong generalization performance and maintain well-calibrated predictions, and (2) the strong model’s training process must strike a careful balance, as excessive optimization may lead to overfitting to the weak supervision. Finally, in the regression setting, we theoretically extend the work of Charikar et al. (2024) to a loss function based on KL divergence, offering guarantees that the strong student can outperform its weak teacher by at least the magnitude of their disagreement. The theory is validated through synthetic experiments.

To tackle this issue, the Weak-To-Strong Generalization (W2SG) paradigm ( Burns et al., 2023 ) is proposed. It finds that, strong pre-trained language models, when fine-tuned using labels produced by weaker models, consistently achieve better performance than their weak supervisors. This intriguing phenomenon has not only driven the development of diverse alignment algorithms ( Zhu et al., 2025 ; Liu and Alahi, 2024 ) , but also inspired efforts ( Pawelczyk et al., 2024 ; Yang et al., 2024 ; Guo et al., 2024 ) to extend the concept to other tasks. However, despite its empirical success, the theoretical foundations of W2SG remain underdeveloped. Although several elegant theoretical frameworks ( Lang et al., 2024 ; Somerstep et al., 2024 ; Wu and Sahai, 2025 ; Charikar et al., 2024 ) are proposed, a universal framework is still lacking to address fundamental questions, such as: What is the optimal generalization performance a strong model can achieve after W2SG? What other factors are influenced by W2SG?

To answer these questions, we provide a theoretical analysis of W2SG, shedding lights on its capabilities and limitations. Firstly, in classification tasks, our theoretical analysis of lower and upper generalization bounds under KL divergence loss reveals that the strong model’s performance is fundamentally determined by two key factors: (1) the disagreement between strong and weak models, which serves as the minimization objective in W2SG, and (2) the weak model’s performance. These findings suggest that (1) achieving the minimal optimization objective in W2SG limits the strong model’s ability to significantly outperform its weak supervisor, and (2) selecting a stronger weak model can enhance the performance of the strong model. Secondly, we investigate how strong model’s calibration—the property that a model’s predicted confidence aligns with its actual accuracy ( Guo et al., 2017 ; Kumar et al., 2019 ) —is affected in the W2SG framework. Our theoretical bounds reveal that the calibration of the strong model depends on both the calibration of the weak model and the disagreement between the two models. The theory highlights the importance of avoiding a poorly-calibrated weak model and an overfitted strong model. The above theoretical analysis is validated using GPT-2 series ( Radford et al., 2019 ) and Pythia series ( Biderman et al., 2023 ) .

The lower bound indicates the fundamental limitation: d 𝒫 ​ ( F ⋆ , F s ​ w ) d_{\mathcal{P}}(F^{\star},F_{sw}) cannot be arbitrarily reduced. Firstly, a minimal d 𝒫 ​ ( F ⋆ , F s ​ w ) d_{\mathcal{P}}(F^{\star},F_{sw}) is intrinsically tied to the weak model performance d 𝒫 ​ ( F ⋆ , F w ) d_{\mathcal{P}}\left(F^{\star},F_{w}\right) . To improve the strong model, the weak model becomes critical—that is, d 𝒫 ​ ( F ⋆ , F w ) d_{\mathcal{P}}\left(F^{\star},F_{w}\right) should be as small as possible. It underscores the importance of carefully selecting the weak model ( Burns et al., 2023 ; Yang et al., 2025 ) . Secondly, the performance improvement of strong model over the weak model cannot exceed 𝒪 ⁡ ( d 𝒫 ​ ( F w , F s ​ w ) ) \mathcal{O}\left(\sqrt{d_{\mathcal{P}}(F_{w},F_{sw})}\right) . In W2SG, while the student-supervisor disagreement d 𝒫 ​ ( F w , F s ​ w ) d_{\mathcal{P}}(F_{w},F_{sw}) is minimized in Equation 6 , we anticipate 𝒪 ⁡ ( d 𝒫 ​ ( F w , F s ​ w ) ) \mathcal{O}\left(\sqrt{d_{\mathcal{P}}(F_{w},F_{sw})}\right) to remain relatively small. However, a paradox arises: achieving a smaller d 𝒫 ​ ( F ⋆ , F s ​ w ) d_{\mathcal{P}}\left(F^{\star},F_{sw}\right) necessitates a larger d 𝒫 ​ ( F w , F s ​ w ) d_{\mathcal{P}}(F_{w},F_{sw}) . This implies that the performance improvement of W2SG is probably constrained by its own optimization objective .

As the number of epochs increases, the accuracy of GPT-2-XL drops even below that of GPT-2-Base (90.95%). This is attributed to the strong expressive power of GPT-2-XL, which leads to overfitting to the weak supervision provided by GPT-2-Base. Note that the upper bounds derived in Theorem 4.1 and Theorem 4.2 do not guarantee that the strong model will outperform the weak model in terms of both generalization performance and calibration properties. The underlying intuition is that if a strong model overfits to the weak supervision, it may closely mimic the weak model’s generalization and calibration behavior. Consequently, the strong model could end up performing on par with or potentially even worse than the weak model.

In contrast to the symmetric squared loss studied in prior work ( Charikar et al., 2024 ) , the emergence of the reverse KL divergence is inherently tied to the asymmetric properties of the KL divergence. Although extending previous work to both forward and reverse KL divergences presents significant technical challenges, our results demonstrate the theoretical guarantees of W2SG in these settings. In Inequality ( 10 ), the left-hand side represents the error of the weakly-supervised strong model on the true data. On the right-hand side, the first term denotes the true error of the weak model, while the second term captures the disagreement between the strong and weak models, which also serves as the minimization objective in W2SG. This inequality indicates that the weakly-supervised strong model improves upon the weak model by at least the magnitude of their disagreement, d 𝒫 ​ ( F s ​ w , F w ) d_{\mathcal{P}}(F_{sw},F_{w}) . To reduce the error of F s ​ w F_{sw} , Theorem 5.1 aligns with Theorem 4.1 , highlighting the importance of selecting an effective weak model and the inherent limitations of the optimization objective in W2SG.

Denote d 𝒫 ​ ( F ⋆ , F s ) = ε d_{\mathcal{P}}(F^{\star},F_{s})=\varepsilon . With probability at least 1 − δ 1-\delta over the draw of n n i.i.d. samples, there holds

Corollary B.1 shows that Charikar et al. (2024) can be extended to our setting if we introduce another assumption, which comes from the technical challenges of theoretically analyzing the non-linear nature of KL divergence. Denote F + = F ⋆ F s ​ w − 1 − log ⁡ F ⋆ F s ​ w F^{+}=\frac{F^{\star}}{F_{sw}}-1-\log{\frac{F^{\star}}{F_{sw}}} , then the assumption D WIS ( F ⋆ ∥ F s ​ w ) ≤ 0 \mathrm{D}_{\mathrm{WIS}}(F^{\star}\|F_{sw})\leq 0 is equivalent to ⟨ F s ​ w − F w , F + ⟩ ≤ 0 \left\langle F_{sw}-F_{w},F^{+}\right\rangle\leq 0 . Note that ∀ x ∈ 𝒳 \forall x\in\mathcal{X} , F + ​ ( x ) ≥ 0 F_{+}(x)\geq 0 always holds, and a very small or large value of F ⋆ ​ ( x ) F s ​ w ​ ( x ) \frac{F^{\star}(x)}{F_{sw}(x)} generally contributes to a large F + ​ ( x ) F_{+}(x) . To make ⟨ F s ​ w − F w , F + ⟩ ≤ 0 \left\langle F_{sw}-F_{w},F^{+}\right\rangle\leq 0 more likely to hold, we expect F s ​ w ​ ( x ) ≤ F w ​ ( x ) F_{sw}(x)\leq F_{w}(x) if F ⋆ ​ ( x ) F^{\star}(x) is small or large. In general, since this condition cannot be guaranteed to hold universally, the inequality in Equation 10 may fail to hold. This reveals a key discrepancy between the square function (as considered in Charikar et al. (2024) ) and the KL divergence (in this work) within the W2SG framework—a phenomenon that will be empirically validated through our experiments.