Will AI R&D Automation Cause a Software Intelligence Explosion?

Looking at the LLM analyses, we may consider that the training efficiency estimate is likely conservative at an ~8 month doubling time, as it does not account for post-training enhancements. The runtime efficiency estimate of ~4 months, meanwhile, is likely aggressive, as it includes cost savings outside of software (such as from hardware and market forces); though even this latter estimate ignores some post-training enhancements. We believe it’s reasonable to split the difference between these two estimates and conclude that both training efficiency and runtime efficiency have a ~6 month doubling time . 11

How should we expect AI software progress to change as more and more software R&D tasks are automated, often by systems that substantially outperform humans within their domain? Intuitively, this dynamic would speed up software progress considerably. Even if overall progress becomes bottlenecked by tasks that only humans are able to perform, human researchers would still be able to spend much more time focusing on these particular bottlenecked tasks, as AI systems would be performing all the other tasks.

As already hinted at, however, the above comparison relies on assumptions that may wind up not holding. In particular, ASARA might initially only be able to fully automate AI R&D work by having AI systems “thinking” for thousands of seconds just to produce as much useful work as 1 second of thought from a human expert (in which case even 10 million AI systems would initially only be able to do as much cognitive work as a few thousand human experts). Indeed, the performance of recent “reasoning models” can be significantly improved by drastically ramping up how much computing power is used to run each copy, allowing each to think for longer. This suggests that the first AI systems to match human experts across the board may do so using very large amounts of computing power for each task. Even so, efficiency improvements imply that after ASARA is developed, these computational costs will fall. And soon after ASARA is developed, AI will likely contribute orders of magnitude more cognitive labor to AI R&D than humans do. 19

So how fast would we then expect AI progress to be? We can only speculate. But if the current rate of software progress implies AI efficiency has a doubling time of ~6 months (or less, if we include post-training enhancements), then the extra researcher capacity might significantly increase the rate of progress – as a ballpark figure, perhaps to a doubling time of a month or two.

In each of these two cases, we see both a) it gets “harder” over time to increase efficiency, requiring more papers for each subsequent doubling, and b) the number of AGIs increases over time, increasing the total effort going towards improving efficiency. The differentiator between the fizzle and the SIE is the relative strength of these two effects. Specifically, in a fizzle, each doubling of efficiency requires more than twice as many papers as the last (e.g., from 2 → 6 → 18), implying progress is getting harder at a faster rate than the AGI labor force is improving. Meanwhile, for an SIE, each efficiency doubling requires less than twice as many papers as the last (e.g., from 2 → 3), implying progress is getting harder at a slower rate than improvements in the AGI labor force. If each efficiency doubling required papers to exactly double, then we’d see sustained exponential growth in efficiency, as each efficiency doubling would continue to take 2 months (e.g., with 2 AGIs needing to produce 4 papers, 4 AGIs needing to produce 8, …).

According to their analysis, the median likelihood value for r r r in computer vision is 1.4, though with large uncertainty – the 5th to 95th percentiles for r r r spanned 0.8 to 2.4, and uncertainty would be wider still if we also consider that their choice of variable for software R&D work may not have matched the true research input in the field (e.g., perhaps the average researcher quality changed over this time, or perhaps some other measure of the number of researchers in the field would be more apt). Still, the bottom line is that each time image-recognition training efficiency doubled, the work needed to double it again likely less than doubled. And preliminary internal analysis from Epoch suggests that the value of r r r is as high or higher for LLMs.

Considering both efficiency improvements and capability improvements together, as well as sources of multiplicative software improvement, we might currently expect that a single doubling in cumulative AI software R&D efforts would lead to a few doublings in AI software capacity (i.e., our best guess for r r r should perhaps be ~1-4 , though with high uncertainty 36

But there isn’t a good reason to expect this limit to be only slightly above the first ASARA systems, which may be imagined as approximately just substituting for human workers within relevant cognitive domains. Humans are presumably not the most intelligent lifeform possible, but simply the first lifeform on Earth intelligent enough to engage in activities like science and engineering. The human range for cognitive attributes is wide, and humans continue to gain from expanding population and specialization, as well as various cultural developments , indicating no fundamental limit in sight. In addition, ASARA will most likely be trained with orders of magnitude more computational power than estimates of how many “computations” the human brain uses over a human’s development into adulthood, suggesting there’s significant room for efficiency improvements in training ASARA systems to match human learning. 38

The analyses for AI software progress conducted by groups like OpenAI and Epoch, discussed above, all occurred in a context of increasing computing power. Perhaps humans working on AI software R&D weren’t as responsible for software progress as we’re imagining, and instead the key enabler of this software progress was the increasing amount of hardware. After all, hardware can be used for running AI experiments (e.g., to find better algorithms), so more hardware would mean more and/or larger AI experiments. Take away the continued expansion of computing power, and maybe most of the software gains also dry up.

While it’s unclear exactly how the field of AI would change, we might suspect smart ASARA systems would find effective ways around hardware bottlenecks.

It’s also possible that diminishing returns during an SIE will be generally steeper than in the historical data. Historically, computational resources were growing, so researchers could invent new algorithms that only work at new computational scales for which no one had previously tried to to develop algorithms. Researchers may have been plucking low-hanging fruit for each new scale of hardware. But this won’t be possible in an SIE, when the hardware stays fixed. Restricting to algorithms at a fixed computational scale might make the diminishing returns much steeper.

Accounting for the constant hardware, we might reduce our best-guess estimate of r r r to ~0.5-2, with the estimate lower if progress requires large experiments and higher if improvements like prompting and scaffolding can go a long way.

Therefore, the bottleneck from having to train new AI systems would likely delay an SIE rather than prevent it. 43

Given all the above, it seems at least decently likely that an SIE would occur if hardware were held constant upon the creation of ASARA and human social factors didn’t prevent it, though we can’t be confident either way. 44

Even if an SIE does not occur, and we instead get a “fizzle,” we could still see a period of AI software growing fast, perhaps about as fast as AI capacity is increasing now, considering not just software progress but also hardware progress and increases in spending on hardware. In other words, if we decided to completely pull the brakes on hardware increases once we reached ASARA and software progress is slow enough that it doesn’t “spiral out of control,” we might still expect to have a year or so of relative increases in AI capacity similar to today’s rate of progress, but occurring when AI systems are already powerful enough to automate AI R&D. Any plans for there being a pause in AI improvements around the time of ASARA should take this possibility into consideration.

We’re formulating the relationship between these factors in this manner – where each doubling in cumulative work leads to a set number of doublings in software capacity – due to some amount of both theoretical reasoning and indirect empirical evidence. This relationship has theoretical appeal – it captures the dynamics that: a) easier ideas will get discovered first, meaning ideas will get “harder to find” over time (and hence, consistent progress will require increased growth in cumulative work – e.g., a doubling in cumulative R&D); b) each idea will improve software by some proportional amount (e.g., a 2x efficiency improvement from a particular algorithmic advance), implying that a constant stream of ideas would cash out as exponential growth in software capacity (e.g., a 12-month doubling time); and c) there’s no reason to suspect these two rates of growth would be equivalent, so we can allow for a doubling in one to correspond to several doublings in the other. There is also further theoretical reasoning for expecting consistent progress to require cumulative work to grow more-or-less exponentially, specifically (like in our formulation), as opposed to following some other increasing function – if an area of work has many problems or directions which can be pursued independently (like R&D in a field), where the difficulty of solving each individual problem varies over orders of magnitude, then we’re likely to see work within the area yield logarithmic returns.