Sourceunknowntext stored

On the Origin of Algorithmic Progress in AI

Author not recordedDate not recordedOriginal︎ ↗Archived copy

Claims1

ExaminationsNone

Text254 sections

VersionsOne known

CopiesNone recorded

CorrectionsNone recorded

Facts checked Nov 11, 2026

ReachNo copies recorded

StructureIn 1 claim's map0 sources downstream across 1 claim

EvidenceNot yet read

ReviewsNot reviewed

Examinations

None recorded. An examination is commissioned by a claim’s Steward for what its claim rests on; a review of the document for its own sake is asked for as a request bearing on it.

Text460 passages · 0 marked

No examination yetMarked passage

Skip to main content0 passages

Report GitHub Issue6 passages

Report Issue0 passages

Back to Abstract0 passages

Download PDF0 passages

Contents0 passages

Reading mode0 passages

Colour theme0 passages

Abstract0 passages

1 Introduction0 passages

2 Compute-Equivalent Gains with Scale Dependence0 passages

3 Scale-Invariant Algorithms1 passage

Literature-Based Components0 passages

Implication: Small, Skewed Scale-Invariant Gains0 passages

4 Scale-Dependent Algorithms0 passages

Experimental Setup for Scaling1 passage

LSTM baseline1 passage

5 Algorithmic Progress Depends Strongly on Compute, Reference Points0 passages

6 Limitations and Further Work0 passages

Impacts of Data Selection0 passages

Our Experiments are Relatively Small and Non-Exhaustive0 passages

The Burden of Optimality0 passages

Beyond FLOP Efficiency: Reasoning and Other Dimensions of Progress0 passages

7 Discussion0 passages

Reconciling Small-Scale Measurements with Large-Scale Progress0 passages

Algorithmic Versus Hardware Improvements0 passages

The Concentration of Progress in Architectural Transitions0 passages

Larger Players Gain More From Algorithms1 passage

Difficulty of Early Identification5 passages

References0 passages

A Model Architecture0 passages

A.1 Learning Rate Selection0 passages

A.2 Further Discussion of Hyperparameter Choices Including Dropout0 passages

A.3 Data Selection, Loading, and Size0 passages

B Scaling-Experiment Models and Hyperparameters0 passages

C 2-Layer LSTM Baseline0 passages

C.1 Model Definitions4 passages

C.2 Scale Grid and Parameter Ranges0 passages

C.3 Optimization, Data, and Tuning Protocol0 passages

D Reproducibility, Code, and Compute Statement0 passages

E Sensitivity Analysis0 passages

E.1 Fit Quality0 passages

F Kaplan-Style Scaling When the True Frontier is Chinchilla-Optimal0 passages

G Transformer SGD Scaling Study0 passages

H Irreducible Loss0 passages

I Reference-Dependence of CEG Multipliers0 passages

J Formal Definition of CEG Functions and Multipliers1 passage

J.1 Interaction Between Algorithms0 passages

K Ablation Experiment Details and Component-Level Results5 passages

L Literature Estimates for Hard-to-Ablate Components3 passages

License: arXiv.org perpetual non-exclusive license2 passages

Hans Gundlach0 passages

Email: hansgund@mit.edu0 passages

Alex Fogelson †0 passages

Email: fogelson@mit.edu0 passages

Jayson Lynch †0 passages

Email: jaysonl@mit.edu0 passages

Ana Trišović †0 passages

Email: anatris@mit.edu0 passages

Jonathan Rosenfeld †0 passages

Email: jonsr@csail.mit.edu0 passages

Anmol Sandhu ‡0 passages

Email: asandhu@alumni.olin.edu0 passages

Neil Thompson0 passages

Email: neil_t@mit.edu † MIT FutureTech, CSAIL ‡ Olin College * Corresponding authors0 passages

Abstract1 passage

1 Introduction13 passages

2 Compute-Equivalent Gains with Scale Dependence1 passage

3 Scale-Invariant Algorithms7 passages

Literature-Based Components1 passage

Implication: Small, Skewed Scale-Invariant Gains1 passage

4 Scale-Dependent Algorithms1 passage

Experimental Setup for Scaling9 passages

LSTM baseline14 passages

5 Algorithmic Progress Depends Strongly on Compute, Reference Points10 passages

6 Limitations and Further Work0 passages

Impacts of Data Selection1 passage

Our Experiments are Relatively Small and Non-Exhaustive2 passages

The Burden of Optimality2 passages

Beyond FLOP Efficiency: Reasoning and Other Dimensions of Progress2 passages

7 Discussion1 passage

Reconciling Small-Scale Measurements with Large-Scale Progress2 passages

Algorithmic Versus Hardware Improvements1 passage

The Concentration of Progress in Architectural Transitions3 passages

Larger Players Gain More From Algorithms3 passages

Difficulty of Early Identification3 passages

Acknowledgments and Disclosure of Funding8 passages

References29 references

Note: blog.eleuther.ai/ [Online; Accessed: 2025-11-25]49 passages

Note: Accessed: 2025-11-170 passages

External Links: Link19 passages

Note: WebsitePublished April 3, 20250 passages

External Links: Link35 passages

Note: Dwarkesh PodcastPublished September 8, 20260 passages

External Links: Link27 passages

Note: Accessed: 2026-01-260 passages

External Links: Link20 passages

External Links: Link8 passages

Note: Forethought Newsletter0 passages

External Links: Link34 passages

Appendix A Model Architecture1 table

Hyperparameter0 passages

Value0 passages

Width/Depth aspect ratio0 passages

160 passages

Norm type0 passages

RMSNorm prenorm0 passages

Feedforward / model dimension0 passages

40 passages

Activation function0 passages

SwiGLU0 passages

Positional encoding0 passages

Rotary0 passages

Sequence length0 passages

1280 passages

Stride0 passages

1280 passages

Batch size2 passages

Dropout0 passages

0.00 passages

Weight decay0 passages

0.010 passages

Gradient clipping0 passages

1.00 passages

Learning rate schedule1 passage

Min learning rate0 passages

Warmup fraction0 passages

10 % 10\%0 passages

Default optimizer0 passages

AdamW1 passage

0.91 passage

0.9990 passages

Initialization0 passages

GPT-2/BERT init0 passages

Embedding weight tying0 passages

True0 passages

Tokenizer0 passages

GPT-2 Tokenizer0 passages

Vocabulary0 passages

502570 passages

Dataset0 passages

C40 passages

Validation set size0 passages

500k tokens0 passages

Token-to-parameter ratio0 passages

401 table

Hyperparameter0 passages

Value0 passages

Width/Depth aspect ratio0 passages

160 passages

Norm type1 passage

Feedforward / model dimension0 passages

40 passages

Activation function0 passages

GeLU0 passages

Positional encoding0 passages

Sinusoidal0 passages

Sequence length0 passages

1280 passages

Stride0 passages

1280 passages

Batch size0 passages

1280 passages

Dropout0 passages

0.00 passages

Weight decay0 passages

0.010 passages

Gradient clipping0 passages

1.00 passages

Learning rate schedule1 passage

Min learning rate0 passages

Warmup fraction0 passages

10 % 10\%0 passages

Default optimizer0 passages

AdamW1 passage

0.91 passage

0.9990 passages

Initialization0 passages

GPT-2/BERT init0 passages

Embedding weight tying0 passages

True0 passages

Tokenizer0 passages

GPT-2 Tokenizer0 passages

Vocabulary0 passages

502570 passages

Dataset0 passages

C40 passages

Validation set size0 passages

500k tokens0 passages

Token-to-parameter ratio0 passages

401 table

Hyperparameter0 passages

Value0 passages

Number of layers0 passages

10 passages

TBPTT length0 passages

320 passages

TBPTT stride0 passages

320 passages

Sequence length0 passages

1280 passages

Batch size0 passages

1280 passages

Input dropout0 passages

0.00 passages

Hidden dropout0 passages

0.00 passages

Output dropout0 passages

0.00 passages

Weight decay0 passages

10 − 4 10^{-4}0 passages

Gradient clipping0 passages

1.00 passages

Learning rate schedule1 passage

Min learning rate0 passages

Warmup fraction0 passages

10 % 10\%0 passages

Default optimizer0 passages

AdamW1 passage

0.01 passage

0.9990 passages

Initialization1 passage

Embedding weight tying0 passages

True0 passages

Tokenizer0 passages

GPT-2 Tokenizer0 passages

Vocabulary0 passages

502570 passages

Dataset0 passages

C40 passages

Validation set size0 passages

500k tokens0 passages

Token-to-parameter ratio0 passages

400 passages

A.1 Learning Rate Selection2 passages

A.2 Further Discussion of Hyperparameter Choices Including Dropout1 passage

A.3 Data Selection, Loading, and Size1 passage

Appendix B Scaling-Experiment Models and Hyperparameters1 passage

Appendix C 2-Layer LSTM Baseline2 passages

C.1 Model Definitions8 passages

C.2 Scale Grid and Parameter Ranges5 passages

C.3 Optimization, Data, and Tuning Protocol1 passage

Appendix D Reproducibility, Code, and Compute Statement1 passage

Appendix E Sensitivity Analysis5 passages

E.1 Fit Quality6 passages

Appendix F Kaplan-Style Scaling When the True Frontier is Chinchilla-Optimal27 passages

Appendix G Transformer SGD Scaling Study2 passages

Appendix H Irreducible Loss1 passage

Appendix I Reference-Dependence of CEG Multipliers1 passage

Appendix J Formal Definition of CEG Functions and Multipliers9 passages

J.1 Interaction Between Algorithms5 passages

Appendix K Ablation Experiment Details and Component-Level Results12 passages

Appendix L Literature Estimates for Hard-to-Ablate Components10 passages

Instructions for reporting errors7 passages

Not found in the text 1

As quoted

in the last 10 years, algorithmic progress has contributed more than 4 orders of magnitude in so-called "effective compute" while compute scaling has increased 7 orders of magnitude

Fraction of recent frontier AI capability progress attributable to algorithmic improvements rather than compute scalingUnassessed

Not yet read by this claim’s Steward.

Lineage across 1 claim

Draws on · none recorded

This source

On the Origin of Algorithmic Progress in AI

in 1 claim's map

Drawn on by · none recorded

Record

Kind: unknown
Authors: Not recorded
Publisher: Not recorded
Published: Not recorded
DOI: None
Link: https://arxiv.org/html/2511.21622
Archived copy: https://web.archive.org/web/20260311230802/https://arxiv.org/html/2511.21622
Stored text: 87,449 characters, retrieved Nov 11, 2026 · 254 sections, 460 passages
Claims extracted: Not recorded
Versions: One known
Copies: None recorded
Corrections: None recorded
Facts checked: Nov 11, 2026

History 3

Nov 11, 2026DividedSegmenter · Divided into 731 parts by its own structure
Nov 11, 2026FactsFacts fetcher · Facts looked up
Nov 11, 2026Entered the graphEntered the graph; copy stored