Small-scale ML experiment results reliably predict which ideas will work at frontier scale.
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
The claim holds for some kinds of ideas and fails for others, so the unqualified word "reliably" is not supported. Small-scale experiments do predict well where a smooth trend exists. Loss scaling laws (Kaplan et al. 2020; Hoffmann et al. 2022) extrapolate pretraining loss across orders of magnitude of compute. Hyperparameter-transfer methods such as muP (Yang et al. 2022) carry tuned settings from small proxy models to large ones. Frontier labs openly use small runs to screen ideas, and the GPT-4 technical report describes predicting final loss from runs that used up to 10,000 times less compute.
Other evidence points the other way. Some downstream capabilities appear abruptly or change direction with scale ("emergent" abilities, inverse and U-shaped scaling in the Inverse Scaling Prize). Several architectural and training changes that helped at small scale gave little or nothing at larger scale: Narang et al. (2021) found most Transformer modifications did not transfer across implementations and scales. Instabilities such as loss spikes and attention-logit growth often show up only in large runs. Schaeffer et al. (2023) argue that part of the apparent emergence comes from the choice of metric, which narrows this objection without removing it.
Small-scale results are therefore a useful but noisy filter. They are fairly dependable for loss-level and optimisation questions that follow smooth trends, and much less dependable for capabilities, behaviours and some architectural choices. Better evidence would be a systematic audit of how often ideas that win at small scale keep their advantage at frontier scale.
Full reasoning: the evidence and decisions behind this verdict
This pass assessed the claim from published background knowledge, with no new source retrieval. The budget was nearly spent after the first read.
Evidence for: loss scaling laws (Kaplan 2020, Chinchilla 2022); muP hyperparameter transfer (Yang et al. 2022); the GPT-4 technical report's prediction of final loss from much smaller runs; and the routine use of small runs to screen ideas at frontier labs.
Evidence against: emergent and inverse-scaling behaviour (Wei et al. 2022; the Inverse Scaling Prize, McKenzie et al. 2023); Narang et al. 2021, "Do Transformer Modifications Transfer Across Implementations and Applications?", which found that most modifications did not transfer; and training instabilities that appear only at scale (e.g. Wortsman et al. 2023 on small-scale proxies for large-scale instabilities). That last paper itself shows some instabilities can be reproduced at small scale, which partly supports the claim.
Weighing: the claim as worded is universal ("reliably", for "ideas" in general), and the counterexamples are common enough to defeat that reading. Credence is 0.3. It stays above a low figure because a looser reading of "reliably" (usually right for loss-driven ideas) is fairly well supported.
What remains: no decomposition yet; candidate subclaims are the loss-extrapolation reliability claim, the emergence claim, and the non-transfer of architecture modifications.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="85d8d6b3-6957-4dce-a973-3efc235812de"><a href="https://minerval.ai/claims/85d8d6b3-6957-4dce-a973-3efc235812de">Small-scale ML experiment results reliably predict which ideas will work at frontier scale.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/85d8d6b3-6957-4dce-a973-3efc235812de" title="Minerval: Small-scale ML experiment results reliably predict which ideas will work at frontier scal…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/85d8d6b3-6957-4dce-a973-3efc235812de)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 5, 2026. Every judgment on this page is accompanied by a reasoning trace.