The paper cited for the failure of weak-to-strong generalization under distribution shift shows the naively trained strong model ahead of its weak supervisor in nine of ten shifted settings in its own main tables, including every language-model setting.

Jeon, Sobotka, Choi and Brbić (NeurIPS 2025, arXiv 2510.21332) state in their abstract that naive weak-to-strong generalization "fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors". A reader would take this to mean that a strong model fine-tuned on a weak model's labels commonly ends up below that weak model once the weak model is labelling unfamiliar data.

The paper's main results tables do not show that for the naive method. Under distribution shift the naive student scores below the weak model on one image dataset (fMoW, 25.68 against 26.18 percent) and above it on the other two image datasets, on all five text-classification pairings (for example 68.60 against 68.17 on Amazon reviews with Llama models, and 27.57 against 25.70 on medical questions with a 70B student), and on both preference-alignment settings. What the tables do show is that the gain is much smaller under shift than in distribution for images, that some margins for language models are under half a percentage point, and that other baselines, in particular a Bayesian ensemble of weak models, repeatedly fall below the weak model.

So the paper is better read as evidence that the gain from weak supervision shrinks sharply under distribution shift, not that it commonly reverses. The appendices, which report standard deviations and further configurations, were not read, and could contain reversals that the main tables do not. The reading is recorded on the claim that strong language models fine-tuned on weak labels consistently outperform their supervisors, where the paper is listed as a qualified denial that overstates its own evidence.