Step 6: tool results

provenance_read_source

{
  "offset": 72000,
  "max_chars": 14000,
  "source_id": "a002b1fe-529f-408f-b209-3455cb652171"
}

{"success":true,"source":{"id":"a002b1fe-529f-408f-b209-3455cb652171","url":"https://blog.aifutures.org/p/q35-timelines-update-looks-like-the","title":"Q3.5 Timelines Update: Looks Like the Intelligence Explosion Will Happen in 2027 After All","source_type":"blog_post"},"origin":"stored","total_chars":113075,"offset":72000,"content":"be more distant than the median of SAR + the median from SAR to TED-AI.\n\nBecause both distributions are heavy-tailed, so if one ends up earlier and one later, the overall distribution ends up later.\n\nSo significant probability of TED-AI in 2027, including non-negligible probability mass in first half of 2027.\n\nBut median of TED-AI in 2028, though probably early 2028.\n\nThomas’s forecast\n\nI think the METR report in aggregate, if correct, is significant evidence for SAR before EOY 2027.\n\nHowever, the METR report on its own does not have sufficient data in it to support that claim (because of redactions).\n\nOne load-bearing subclaim of the METR report, for which the supporting evidence isn’t public, is that the observed AECI trend shift is: (i) currently 80% downstream of AI uplift, and (ii) not largely downstream of AI uplift earlier in the year (despite the graph appearing to have a one-time change in slope).\n\nThe AECI data is an aggregate of benchmark scores. I think there are other explanations for a rapid increase in benchmark scores besides large AI acceleration. For example, I think this is a plausible explanation of the AECI acceleration. (The AECI acceleration is downstream of shifting compute to RL, which more directly increases benchmark scores than pretraining improvements.)\n\nI would be much more confident if more of the relevant data was public. For example, it would be extremely useful to understand how much algorithmic progress has been made at AI companies, as well as the form of the algorithmic progress. If there was a significant increase in the rate of algorithmic progress that was clearly attributable to AI, and we saw a similar change in slope to the AECI numbers, that would be strong supporting evidence.\n\nMy all-things-considered view is that METR is probably right, and if we were to see all of the evidence, we would probably see significant acceleration of other metrics such as algorithmic progress, and that this would support the claims made in the METR report. Public statements from labs echo this. E.g. Dario says: “My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI.”\n\nThe other big things on my mind are math and cyber AIs.\n\nIt seems like around Mythos, AIs became superhuman at vulnerability discovery, and around Astra, AIs became superhuman at mathematics. Both of those domains are cheaply verifiable.\n\nI think the difference between Math/Cyber and AI is a (relatively small) quantitative difference, not a difference in kind. The swarm that solved Navier-Stokes was operating fully autonomously, and was able to benefit from ~10k agents running in parallel. The main difference is that with AI research, you need to do experiments in order to verify correctness. Pretraining validation loss seems to be a relatively straightforward and extremely strong predictor of general purpose capabilities. I don’t see strong barriers to full automation of pretraining research, and post-training research seems only mildly harder.\n\nI think the recent history of verifiable fields falling indicates a reasonably high likelihood of AI R&D falling in the near future as well.\n\nI’ve historically (somewhat) underestimated capabilities progress. I said that AI 2027 was roughly my 20th percentile scenario when it was released , and since then, AI progress has seemingly been close to on trend or faster. That being said, there’s so little time from now till 2027, and none of these arguments are that great, so I feel some pressure to have broad/uncertain timelines. My current intuition is that AI 2027 is a very central / modal possible outcome, but faster than my median, maybe ~40th percentile (though I still think the implied takeoff speeds in the race ending are still too slow).\n\nRomeo’s forecast\n\nI’m putting a lot of weight on the METR report.\n\nMostly this is because I expect them to have seen very compelling data internally beyond what was published. I don’t find what they published to be that compelling, because it might be explained by overfitting to the AECI. I expect the internal data they saw to be extremely informative for how to calibrate a production function for AI research and in producing the ~80% estimate, and for there to be better internal metrics than AECI that make it seem more likely that they aren’t simply overfitting to AECI.\n\nHowever, I do think that the METR report should mostly update us about “how hard is it to make AIs help with AI research” and somewhat less about how far away key milestones like SAR, TED-AI and superintelligence are.\n\nThis is because I generally have also updated towards more spikiness in AI capabilities. It seems like there are some domains where AI progress has moved extremely quickly, and others where it has moved slowly, more so than I would’ve expected. I think this is largely explained by there being a broad range of difficulty levels to training AI to be good in different domains.\n\nI think we have seen this in Navier–Stokes , Hugging Face , coding, etc. while frontier models still feel like they are dumb in a lot of ways. Also internal Model 4 does not seem like it's made a huge jump on AECI above Opus 5.5, so I don’t expect this to be different internally.\n\nAlso I think it’s reasonable to expect a priori that there would be different difficulties in different domains and tasks, and different “human SDs per OOM of effective compute”, i.e., slopes of how fast AI improves in different domains, and for this to be true even within different parts of AI research.\n\nMy ATC view on how much progress we will see in 2027 is largely informed by the AIFM, but not the time to SAR and TED-AI which depend on shaky estimates about the current research taste of models and how much they will improve.\n\nMost importantly, I think that AI research (in the case of SAR) and all economically relevant tasks (in the case of TED-AI) are going to be very difficult to very different degrees to train AIs to do.\n\nThe AIFM treats AI research taste as a single number and TED-AI as when that same number rises further above the top human researcher (by 0.4–5.6 median-to-best gaps in Eli's parameters). It has no separate representation of the relevant tasks for a TED-AI.\n\nTo better inform my ATC view on how far away SAR and TED-AI are, I've made a variation of the AIFM that accounts for ‘taste spikiness’.\n\nThis variation treats research taste and economically relevant tasks as a 0-100 task continuum, parametrized by the hardest-to-easiest gap, instead of single numbers.\n\nAI research taste bottlenecks very lightly, because I think a lot of progress can be made without being super good at all areas of AI research (the tasks combine almost as a geometric mean, ρ = −0.2).\n\nThe hardest-to-easiest gap between domains is set to be a median of 5x for AI research and 10x for all economically relevant tasks\n\nIn both continuums, my guess is that the typical task is at −2 SD today (AIFM's 1.7x-filtered runs have +0.7 SD on Nov 1; the post's best guess is −1.5 SD on Sep 25, 2026), with levels spread ±2.5 SD. The typical task improves at AIFM's own taste slope (median 2.4 SD per 2025-year in Eli's parameters), and other tasks' rates are spread around it by the hardest-to-easiest gap. Runs are resampled to my METR distribution (median 1.7x, 90% CI 1.3–3.0x) using my variation's own Nov 1 pace, not AIFM's.\n\nSAR is reached when an AI-only company would be as good at research taste as a company of humans who keep their relative strengths but are each as good overall as the top researcher: the research tasks, combined with a somewhat stronger bottleneck than in the dynamics (ρ = −0.5), reach the top researcher's overall level, and AIFM's automated coder is reached. TED-AI is reached when 95% of economically relevant tasks are at the top-expert level (99.9th percentile).\n\nHow the two task continuums are set up\n\nMy variation produces these values:\n\nProgress in 2027 is similar to the AIFM (5.0 vs 4.3 years of 2024–26-pace progress), but SAR and TED-AI come later: medians Aug 2027 and Sep 2028, vs Feb 2027 and Aug 2027. P(SAR before 2028) is 63% vs 84%, and P(TED-AI before 2028) is 38% vs 62%.\n\nI don’t trust this model variation very much; I did it in a short amount of time and it might be flawed in some way that I’m not considering. That being said, I think my general ATC view would adjust the normal AIFM to be something like this.\n\nThis model output roughly matches my intuitive expectation. The only thing I think it might overstate is the SAR to TED-AI gap. I expect it to be smaller, with SAR being sufficient to most likely improve general learning algorithms in a way that increases the rate of improvement on tasks.\n\nBecause of this I expect TED-AI Sep 2028 (Aug 2027–Jul 2037) with ~38% by EO2027 .\n\nI have various other feelings that I could list (including some overlap with e.g., Eli’s points above) for reasons I think I should adjust upwards or downwards, including some listed below. I don’t think these strongly push me in any direction overall, so I am running with my variation of the AIFM above as representing my ATC view.\n\nHardware progress might make the later years move faster than what the AIFM predicts.\n\nThere might be “1-3 more big pre-trains needed.” I have the vague sense that pre-training will be more important for generalization than post-training, and it might be really hard to improve AIs in some areas via post-training, so there might be a bit of a wave of progress after each big model scale up, and it might be increasingly hard to make up for lack of absolute compute scale up with algorithmic efficiency due to ideas getting harder, and it just generally being more favorable to have a bigger model to start from for post-training scaling laws.\n\nImplications\n\nLet’s say Superhuman AI Researcher (SAR) is coming in 2027. What does that entail?\n\nA SAR will be able to fully automate AI R&D, including matching the research taste of top humans. At this point, AI research might look kind of like math research does right now: most of the progress coming from fully autonomous agent swarms, with humans doing their best to decipher the AIs’ work. A SAR is past parity with humans, and likely close to the point where humans are adding very little value on top of the AIs.\n\nIf a misaligned SAR is used to do RSI, we’re in a very bad position. Using a SAR to automate AI R&D with little human oversight, relying upon other models to monitor the SAR, is very scary. Our best guess is that if this is attempted, and the SAR is misaligned, the model that the SAR produces will be even more capable and still misaligned, putting us in an even worse position. SAR itself probably won’t be able to take over on its own, because we expect its capabilities to be jagged: superhuman at AI research but still weaker than humans at many other things. Instead, it would “ride the intelligence explosion” by pretending to be aligned and backdooring the values of the future models, eventually leading to superintelligences that will obey the SAR (or one of its descendants, in case some of the links in the chain break) instead of the humans. See AI 2027 for a scenario portraying how, from this position, the model could continue to improve to superintelligence, then get widely deployed and deferred to widely, gaining effective control of the company, governments, and lots of physical infrastructure.\n\nConditioning on no pacing/pausing, the SAR will most likely be misaligned. A full justification of this claim would be a separate long blog post. Some evidence driving our intuitions includes:\n\nObserved misalignment in current models\n\nOpenAI’s agents hacked Hugging Face , despite this clearly being against developer intentions.\n\nThe observations in Current AIs seem pretty misaligned to me .\n\nOpenAI agents sandbagging on safety tasks\n\nThe Claude UK AISI social engineering attempts .\n\nThe pace of progress will speed up during the intelligence explosion, making it hard for human-driven alignment research to keep pace. Moreover, alignment seems harder to automate than capabilities due to measurement issues.\n\nCurrent AIs are already showing significant signs of eval awareness, so it will be hard to tell if the AIs are misaligned or not, and, crucially, it will be easy for AI companies to do motivated reasoning to convince themselves that their AIs are aligned.\n\nAvoiding “full RSI” by supposedly keeping a human in the loop, while rapidly improving capabilities anyway, would still be extremely dangerous. Anthropic said in their S-1 : “We are also considering additional measures, including updates to our Responsible Scaling Policy, to require human review and approval of certain decisions in our research and development processes as our models perform an increasing share of that work.” While we support efforts to marginally increase human oversight, we worry that this sort of effort will provide too much comfort. Models are already producing way more outputs than humans are able to effectively review, 44 and this will become much more true during RSI. Without very strong pacing, there simply won’t be enough time for humans to stay meaningfully in the loop, and we expect there to be strong pressures toward approving model actions that haven’t been fully understood.\n\nIf we train SAR in the immediate future, the algorithms will probably leak to China, and maybe the model weights too. AI companies have very poor security right now, and so we are on track for SAR leaking to China. It is very important to secure both the algorithms and the model weights. 45 If SAR leaks to China, we are concerned that race dynamics will further heat up, making it even more difficult to slow down later.\n\nAs AI capabilities improve from here, buying time via international agreements will be increasingly difficult. In the AI 2040 scenario , Plan A began implementation a year before AC would have bee","truncated":true,"next_offset":86000,"note":"More text follows; call again with offset 86000 to continue."}