Frontier AI labs' 2024–2025 pretraining scale-ups produced smaller capability gains than earlier generational transitions.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass. Created two subclaims: the GPT-4.5 vs GPT-3-to-4 gain comparison (supports) and the compute ratio (contradicts). Linked the existing Gemini 3 claim as contradicts, because Gemini 3 was released within the window. Organized these into one argument for and one against. Assessed as supported (0.6), credence 0.68, importance 0.5, contestation 0.55. The parent's assessment already assumed this premise was supported, so no notification is needed. Instances were not recorded because of budget; key sources are cited in the trace.
Assessed Supported
verdict confidence 0.60 · credence 0.68
The claim holds best for the 2024 to early 2025 period and for OpenAI in particular. In November 2024 The Information reported that OpenAI's Orion model improved on GPT-4 far less than GPT-4 had improved on GPT-3, and similar reports followed about efforts at Google and Anthropic. Orion was eventually released as GPT-4.5 in February 2025. Its benchmarks were mixed: large gains on factual-knowledge tests such as SimpleQA, but no clear lead on reasoning or coding, where smaller reasoning-trained models did better. Most observers judged it a smaller step than the jump from GPT-3 to GPT-4, and the evidence favors the view that GPT-4.5's gain over GPT-4 was the smaller of the two. Two qualifications limit the claim. First, the comparison does not match compute. GPT-4.5 is reported to have used about ten times GPT-4's compute, against roughly a hundredfold for earlier generational jumps. A smaller gain is therefore roughly what scaling trends predict. Read literally, as a comparison of the gains themselves, the claim is likely true. Read as a claim that returns per unit of scale fell, it is much less certain. Second, the window runs to the end of 2025, and Gemini 3, released in November 2025, was credited by Google DeepMind staff with large pretraining-driven gains. That cuts against the claim as a statement about frontier labs generally. Gains from GPT-5 and similar models in 2025 came mostly from reinforcement learning and reasoning training rather than larger pretraining, so they do not bear directly on the claim. What would settle the matter: lab disclosures of compute against capability for the 2024–2025 runs, or careful benchmark comparisons that hold compute constant across generations.
Claim entered the graph