Step 6: tool results

provenance_read_source

{
  "offset": 20000,
  "max_chars": 10800,
  "source_id": "87406e11-4df5-49b3-8558-11e864316894"
}

{"success":true,"source":{"id":"87406e11-4df5-49b3-8558-11e864316894","url":"https://importai.substack.com/p/import-ai-455-automating-ai-research","title":"Import AI 455: AI systems are about to start building themselves.","source_type":"blog_post"},"origin":"stored","total_chars":30727,"offset":20000,"content":"s literally saying that AI R&D is its goal: OpenAI wants to build an “ automated AI research intern by September of 2026 ”. Anthropic is publishing work on building automated alignment researchers . DeepMind appears to be the most circumspect of the big three, but still says “ automation of alignment research should be done when feasible ”. Automating AI R&D is also the goal of numerous startups: Recursive Superintelligence just raised $500m with the goal of automating AI research , and another neolab, Mirendil, has the goal of “ building systems that excel at AI R&D .”\nIn other words, the combined efforts of hundreds of billions of existing and new capital is being sunk into entities that have the goal of automating AI R&D. We should surely expect at least some progress in this direction as a consequence.\n\nWhy this matters\nThe implications of this are profound and under-discussed in popular media coverage of AI R&D. I’ll list a few here. This isn’t a comprehensive list, but it gestures at the enormity of the challenges AI R&D introduces. .\n\nWe have to get alignment right : Alignment techniques that work today may break under recursive self-improvement as the AI systems become much smarter than the people or systems that supervise them. This is a very well covered area, so I’ll just briefly highlight some of the issues:\n- Training AI systems to not lie and cheat is surprisingly subtle (e.g, despite trying very hard to build good tests for environments, it’s sometimes the case the best way for an AI to solve it is to cheat, thus teaching it that cheating is good)\n- AI systems might be able to ‘fake alignment’ by outputting scores that make us think they behave a certain way that actually hides their true intentions. (In general, AI systems are already aware of when they are being tested.)\n- As AI systems start to contribute more of the foundational research agenda for their own training, we might end up substantially changing the overall way AI systems get trained and not have good intuitions or intellectual foundations for understanding what this means.\n- There are very basic “compounding error” problems whenever you put something in a recursive loop that likely hits on all of the above and other problems: unless your alignment approach is “100% accurate” and has a theoretical basis for continuing to be accurate with smarter systems, then things can go wrong quite quickly. For example, your technique is 99.9% accurate, then that becomes 95.12% accurate after 50 generations, and 60.5% accurate after 500 generations. Uh oh!\n\nEverything that AI touches gets a massive productivity multiplier : In the same way AI is dramatically improving the productivity of software engineers, we should expect the same thing to happen for everything else that AI touches. This introduces a couple of issues we’ll have to contend with: 1) inequality of access: assuming that demand for AI continues to outstrip compute supply, we’ll have to figure out where to allocate AI to maximize a social upside. By default, I am skeptical that market incentives guarantee us the best societal upside from limited AI compute. Figuring out how to allocate the acceleratory capabilities conferred by AI R&D will be a politically charged problem. 2) ‘Amdahl’s Law’ for the economy: as AI flows into the economy, we’ll discover places where things break or slow under the increased volume, and we’ll need to figure out how to fix those weak links in the chain. This may be especially pronounced in areas where you have to reconcile the fast-moving digital world with the slow-moving physical world, like drug trials for new medical therapies.\n\nThe formation of a capital-heavy, human-light economy : All of the above evidence for AI R&D also points to the increasing capabilities of AI systems to autonomously run businesses as well. This means we should expect for an increasing chunk of the economy to get colonized by a new generation of companies which are either capital-heavy (because they own a lot of computers), or opex-heavy (because they spend a lot of money on AI services which they build value on top of), and relatively light on labor compared to today’s corporations - because the marginal value of spending more on AI versus human labor will be constantly growing as a consequence of the sustained capability expansion of the AI systems. In practice, this will look like the emergence of a “machine economy” that grows within the larger “human economy”, though we might expect that over time the machine economy will interact more and more with itself as AI-run corporations begin to trade with one another. This will do profoundly weird things to the economy and will invite all sorts of questions around inequality and redistribution. Eventually, it may be possible to see the emergence of fully autonomous corporations that are run by AI systems themselves, which would exacerbate all of the above issues, while also posing many novel governance challenges.\n\nStaring into the black hole:\nGiven all of this, I think there’s a ~60% chance we see automated AI R&D (where a frontier model is able to autonomously train a successor version of itself) by the end of 2028. Based on the above analysis, you might ask why I don’t expect this in 2027? The answer is that I think AI research contains some requirement for creativity and heterodox insights to move forward - so far, AI systems haven’t yet displayed this in a transformative and major way (though some of the results on accelerating math research are suggestive of this). If you had to push me for a 2027 probability, I’d say 30%. If we don’t see it by the end of 2028, then I think we will have revealed some fundamental deficiency within the current technological paradigm and it’ll require human invention to move things forward.\n\nI have written this essay in an attempt to coldly and analytically wrestle with something that for decades has seemed like a science fiction ghost story. Upon looking at the publicly available data, I’ve found myself persuaded that what can seem to many like a fanciful story may instead be a real trend. If this trend continues, we may be about to witness a profound change in how the world works.\n\nThanks to Andrew Sullivan, Andy Jones, Holden Karnofsky, Marina Favaro, Sarah Pollack, Francesco Mosconi, Chris Painter, and Avital Balwit, for feedback on this essay.\n\nThanks for reading!\n\nSubscribe\n\n351\n\n54\n\n67\n\nShare\n\nDiscussion about this post\n\nComments Restacks\n\nHerbie Bradley\n\nMay 4 Edited\n\nLiked by Jack Clark\n\nGood piece! But to me this is all entirely consistent with a position of: we should expect models in the current paradigm to automate much of the \"junior RS\" work in the next few years, but not higher level skills like research taste and creativity. Of course, one can always formulate AI research as a pure search problem (and there is a lot of prior literature doing that) and say that if we just throw enough compute we'll make progress and you don't need research taste. But the search space is huge and creativity is an insanely powerful shortcut, such that if models are incapable of research-grade creative insights we should expect a modest speed boost overall. A framing I like is just that LLMs are a different \"shape\" of intelligence to humans, and so we should expect comparative advantage.\n\nMy own impression from doing AI research is that there is significant creativity involved beyond what models should be expected to be capable of, when it comes to picking directions, coming up with a vision for what's possible & backchaining, etc (and rarest of all, putting together a coherent long-term research agenda that fills a missing gap with a tractable sequence of breakthroughs to get there). Others, including Anthropic researchers, clearly also think these skills are part of research: https://x.com/_arohan_/status/2050978424446235090\n\nSo overall it seems to me this is not such a huge shift as you're implying, so much as a distribution shift in tasks carried out by researchers very comparable to that going on in SWE over the coming years. In SWE, the tasks that make up the job are shifting such that SWE is becoming more of a SWE-PM hybrid, and juniors are not automated because they can just shift their tasks towards more higher level of abstraction things like system design (and learn faster through greatly increased velocity).\n\nI think a lot of analyses, including arguably this one, underrate that the tasks that make up jobs are changing all the time in reaction to AI's deployment and other people in the market.\n\nSide note: I would also say that it currently appears that the skill ceiling and complexity ceiling in software engineering is higher than in AI R&D, and AI R&D necessarily involves significant software engineering, so I would on balance expect that if AI R&D is entirely automated then scaling inference serving another 1000x would still require human experts, and the rest of software engineering (outside of AI labs) to heavily require humans.\n\nOn point 2, I agree though noting that Amdahl's Law isn't quite the right analogy because it usually assumes task decomposition is fixed and is a result about parallelization, while the O-ring automation intuition from economics is in some sense more general and closer to what you're describing IMO ( https://www.nber.org/papers/w34639 )..\n\nFinally, on your point 3, I broadly agree that short term more businsses may be opex heavy by spending on AI, but fundamentally if a service is fully automatable, its cost should fall to slightly more than the marginal cost of the required intelligence, which will probably be very low in the future—and the share of GDP of these automatable services should collapse. I don't expect fully automated companies to be more than a small fraction of the economy, or peripheral, because most economic value from new technologies is driven by adoption of new services by large incumbents, and the distribution of tasks required to sell that and drive distribution is very different to any of the automatable skills in AI research and heavily loaded on human social bargaining. Anthropic's own private equity partnership in some sense a bet against a largely automated economy.\n\nReply\n\nShare\n\n3 replies by Jack Clark and others\n\nSteven Adler\n\nMay 4\n\nLiked by Jack Clark\n\nThanks for writing this! Slight typo: \"thus teaching it that teaching is good\" the second 'teaching' should be 'cheating'\n\nReply\n\nShare\n\n1 reply by Jack Clark\n\n52 more comments...\n\nTop Latest Discussions\n\nReady for more?\n\nSubscribe\n\n© 2026 Jack Clark · Privacy ∙ Terms ∙ Collection notice\n\nStart your Substack Get the app\n\nSubstack is the home for writing, podcasting, and media founders.","truncated":false,"next_offset":null,"note":"This is the end of the stored text."}