LLM pretraining scaling has hit a recent rough patch.
4 events · 1 assessment · 2 decisions
Updated claim instance
Instance 304c0a93-2862-49d4-96c0-e478f787b374 (https://garymarcus.substack.com/p/the-ai-2027-scenario-how-realistic): was stance=affirms, confidence=0.7; set speaker="Gary Marcus", publication="Marcus on AI (Substack)". The source is Gary Marcus's own Substack, garymarcus.substack.com, and the passage is in his voice, so he is the speaker.
Structured and assessed
First pass. Created three subclaims after the Matcher found none existing: the 2024–25 scale-ups producing smaller gains (requires), data exhaustion (supports), and Gemini 3's pretraining gains (contradicts). Grouped them under one argument for the claim and one against. Assessed as supported (0.6, credence 0.7) from late-2024 reporting, Sutskever's remarks, and the reception of GPT-4.5, with the caveat from Gemini 3. Kept the single Marcus instance (affirms). Importance set to 0.55, contestation to 0.6. No dependents to notify. Budget was conserved, so no provenance reading was done.
Assessed Supported
verdict confidence 0.60 · credence 0.70
The claim refers to the period from late 2024 through mid-2025, when the returns from simply scaling up LLM pretraining appeared to disappoint. Read that way, the evidence favors it. In November 2024, outlets reported that OpenAI's next-generation model ("Orion") and comparable efforts at other labs were showing smaller gains than past generational jumps. In December 2024, Ilya Sutskever said at NeurIPS that "pre-training as we know it will end" because high-quality data is finite. GPT-4.5, released in February 2025 as OpenAI's largest pretraining run, was widely judged disappointing on benchmarks given its cost. These developments support the premise that 2024–2025 scale-ups produced smaller gains than earlier transitions, and the approaching exhaustion of high-quality human text has been offered as a mechanism for the shortfall. There are two caveats. First, a "rough patch" in returns measured on capabilities and benchmarks does not mean the scaling laws broke. Loss continued to improve predictably with compute, and part of the slowdown may reflect delays in building out compute and a shift of effort toward reinforcement-learning and inference-time scaling, rather than a fundamental limit. Second, the patch may have been temporary. Google attributed much of Gemini 3's large late-2025 gains to improved pretraining, and commentators took this as evidence that pretraining scaling was intact. As of late 2026, the claim holds best as a description of 2024–2025. Whether it is "recent" depends on the reader's vantage point. It is weaker if read as saying that pretraining scaling has stalled durably.
Claim entered the graph