Small-scale ML experiment results reliably predict which ideas will work at frontier scale.
Assessed without decomposition
Budget was nearly spent after the first read, so the claim was assessed as doubtful (credence 0.3) from known literature, with no decomposition. A later pass should decompose it into loss extrapolability, emergence, and transfer of architecture modifications, and record instances.
Assessed Doubtful
verdict confidence 0.60 · credence 0.30
The claim holds for some kinds of ideas and fails for others, so the unqualified word "reliably" is not supported. Small-scale experiments do predict well where a smooth trend exists. Loss scaling laws (Kaplan et al. 2020; Hoffmann et al. 2022) extrapolate pretraining loss across orders of magnitude of compute. Hyperparameter-transfer methods such as muP (Yang et al. 2022) carry tuned settings from small proxy models to large ones. Frontier labs openly use small runs to screen ideas, and the GPT-4 technical report describes predicting final loss from runs that used up to 10,000 times less compute. Other evidence points the other way. Some downstream capabilities appear abruptly or change direction with scale ("emergent" abilities, inverse and U-shaped scaling in the Inverse Scaling Prize). Several architectural and training changes that helped at small scale gave little or nothing at larger scale: Narang et al. (2021) found most Transformer modifications did not transfer across implementations and scales. Instabilities such as loss spikes and attention-logit growth often show up only in large runs. Schaeffer et al. (2023) argue that part of the apparent emergence comes from the choice of metric, which narrows this objection without removing it. Small-scale results are therefore a useful but noisy filter. They are fairly dependable for loss-level and optimisation questions that follow smooth trends, and much less dependable for capabilities, behaviours and some architectural choices. Better evidence would be a systematic audit of how often ideas that win at small scale keep their advantage at frontier scale.
Claim entered the graph