Iterative DPO on reward-scored on-policy samples generalizes similarly to online reinforcement learning on the same tasks.
Assessed
First assessment, recorded as inconclusive. Budget ran out after the first read, so no decomposition or search was done. Recommend a funded follow-up with a literature search on semi-online DPO versus GRPO and on behavioural generalization.
Assessed Inconclusive
verdict confidence 0.60 · credence 0.50
Iterative DPO samples from the current policy, scores the samples with a reward function, and trains on preference pairs of high- against low-scoring samples, repeating over rounds. Work on "semi-online" preference optimization, such as Lanchantin et al. (2025), reports that this procedure comes close to online methods like GRPO on capability benchmarks, for both verifiable tasks such as math and non-verifiable tasks such as instruction following. That is indirect support for the claim on capability measures. For behavioural generalization, meaning how a learned behaviour spreads to unrelated dispositions such as emergent misalignment, no head-to-head comparison on a shared environment has been found. There are also reasons to expect the two to differ. DPO's offline contrastive loss can push down the likelihood of rejected responses in ways a policy-gradient update does not. Iterative DPO refreshes its samples only between rounds, so its data can drift off-policy. And the two procedures handle KL regularization differently. The evidence therefore leaves the claim unsettled. A controlled comparison of both procedures on the same reward and environment, measured on capability and on out-of-distribution behaviour, would settle it.
Claim entered the graph