Language-model gains from synthetic data mostly distill capabilities already present in a stronger teacher.
4 events · 1 assessment · 2 decisions
Decomposed and evaluated arguments
With budget remaining, added two arguments: a for argument from imitation-transfer limits (new subclaim c3528904) and an against argument from verifier-grounded RL (new subclaim 9e83361f). Rated them slight and likely respectively. The inconclusive assessment stands. No instances were recorded; a future pass should record Gudibande 2023 and the DeepSeek-R1 report.
Assessed inconclusive no decomposition
Budget was tight ($1.12, a third spent on the first read), so the claim was assessed from background knowledge without being decomposed. Assessed inconclusive at 0.5 credence: distillation dominates instruction-tuning data, while verifier-grounded self-training (RL with verifiable rewards, self-play) gives gains no teacher supplied. A future pass should add two arguments, the distillation-imitation evidence and the verifier-grounded self-improvement counterexamples, with matched subclaims, and record instances (Gudibande 2023; the DeepSeek-R1 report).
Assessed Inconclusive
verdict confidence 0.55 · credence 0.50
Whether gains from synthetic training data mostly come from copying a stronger teacher depends on which kind of synthetic data is meant. For instruction tuning and small-model pretraining, the evidence favors the claim. Prominent pipelines such as Alpaca, Vicuna, Orca and Microsoft's Phi series generate their data with a stronger model. Gudibande et al. (2023, "The False Promise of Imitating Proprietary LLMs") found that this kind of imitation mostly copies the teacher's style, and that factual and reasoning capability transfers much less than human raters perceive. A growing body of work produces gains that no stronger teacher supplied. In these methods a model generates its own data and filters it with an external verifier or reward signal: self-play in games, STaR-style rationale bootstrapping, rejection sampling against unit tests or answer checkers, and reinforcement learning with verifiable rewards on mathematics and code, as in DeepSeek-R1-Zero. Here the extra information comes from the verifier, not from a teacher model. Outside verifiable domains, self-improvement gains, for example from self-rewarding or constitutional methods, are smaller and are usually measured by AI judges. Read as a claim about all synthetic-data gains, the claim is therefore neither established nor refuted. "Mostly" holds for distillation-style data and fails for verifier-grounded self-training, which has driven much of the recent progress in reasoning. Resolving it would need an accounting of which share of the frontier gains attributed to synthetic data comes from each source.
Claim entered the graph