https://epoch.ai/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems

I estimate that the cost of compute in US dollars for the final training run of ML systems has grown by 0.49 orders of magnitude (OOM) per year (90% CI: 0.37 to 0.56)

I estimate that the cost of compute used to train “large-scale” systems since September 2015 (systems that used a relatively large amount of compute) has grown more slowly compared to the full sample, at a rate of 0.2 OOMs/year (90% CI: 0.1 to 0.4 OOMs/year)

I think the evidence is strong enough to conclude that the cost of the most expensive training run is growing significantly slower than milestone ML systems as a whole.

recent growth spending will likely slow down greatly during the 2020s, given the current willingness of leading AI developers to spend on training, and given that the recent overall growth rate seems unsustainable

extrapolating from the year 2025 with an initial cost of $380M (90% CI: $55M to $1.5B) using a growth rate of 0.2 OOMs/year (90% CI: 0.1 to 0.3 OOMs/year), a cost of $233.2B would be reached in the year 2040 (90% CI: 2033 to 2062)

my model predicts a maximum spend of 10^(6.5 + 1.25) = $10^7.75 (~$60 million) by 2025. This is roughly one OOM lower than Cotra’s best guess.

I am 70% confident that the most expensive ML training run that occurred in 2021 cost at least one OOM lower (i.e., less than $45 million) in compute.

The energy cost of final training runs seems about 20% as large as the hardware cost

The most expensive system to train was Minerva in 2022, at an estimated $3.2M

The basic point still stands that recent growth in spending seems unsustainable over the next few decades, but I think that compute spending flattening out at a lower level is more likely than I previously believed.