Step 22: tool results

{
  "status": "doubtful",
  "claim_id": "54c25eab-5263-467c-880b-ee0f2c3bdd12",
  "assessment": "This is a forecast about capability, not practice: that by 31 December 2027 Anthropic or OpenAI will have, at least internally, an AI system able to do every part of frontier AI research other than choosing what to work on. That includes designing experiments, building and running them, noticing dead ends, interpreting results and carrying projects of weeks or months to completion. It asks less than forecasts of fully automated research or a superhuman researcher, but \"all\" is a strict bar and no agreed test exists for it. In practice it would be settled by a lab's declaration backed by evidence, or by an outside evaluation showing AI completing whole research projects within a set direction at expert standard.\n\nThe case for it is a rate argument. Measured task lengths have been doubling every three to four months, and [[claim:83df12e2-68f5-4d25-8327-28d24ab513be|tasks that take a skilled person weeks are likely within reach by the end of 2027]]. METR, after eleven weeks of access inside Anthropic, judged in November 2026 that the lab was improving its models on well-specified tasks at about twice its earlier rate (plausibly 1.3 to 3 times), guessed that most of the current acceleration is due to AI, and found no bottleneck that would stop further acceleration over the next twelve months. On that estimate the AI Futures Project moved its forecasts for a stronger milestone, [[claim:a6037281-5051-4d1f-bd95-5546955a242d|a superhuman AI researcher]], from 2028 or later to 2027, and Anthropic co-founder Jack Clark, who gave about 30% in May 2026 to a model able to train its successor by the end of 2027, said in October that he now thinks 2027 more likely than 2028. A new benchmark found the best model exceeding recruited experts at designing and interpreting experiments on fixed problems.\n\nThe case against starts from where systems stand. [[claim:81eeb61b-4ce4-4d0b-891d-11e970d8f669|Agents in 2026 do research engineering but struggle at critical stages of research]]: in the one direct test, the original authors rejected all four AI-written papers, including two from newer models with more time and budget. OpenAI's own September 2026 account claims only an automated research intern for well-defined tasks of a few days, says agents need significant human steering as tasks grow, with over half of successful four-to-eight-hour tasks involving intervention, and sets March 2028, after the deadline, as its goal for an automated AI researcher. Anthropic reports that no measured part of its R&D is fully autonomous. The acceleration is measured on well-specified tasks, METR says it is unsure how it carries over to less well-specified work, and [[claim:ecb10d1f-b391-4e54-b645-8133ad3827ae|whether it generalises to research]] is unsettled. Both labs have also committed to slowing when safety requires, and OpenAI paused part of its training in mid-2026.\n\nThe supporting forecasts are less independent than their number suggests. The shift toward 2027 rests largely on one estimate, most of whose supporting evidence Anthropic redacted; its authors say the public must rely on their judgment more than is ideal. The forecasters who moved furthest call their analysis preliminary and one of them notes the update is large relative to the evidence shown. The statements against come mostly from the two labs, which have commercial reasons to sound capable and so give weight to what they decline to claim, but which also have reasons to appear measured.\n\nOn balance the forecast is more likely false than true, at a little over one in four. During 2027 the following would raise that sharply: either lab reporting whole research projects of several weeks done by AI within a human-set direction and judged acceptable by experts; original authors accepting AI-written papers in a repeat of the open-ended test; OpenAI moving its March 2028 goal earlier; or an outside evaluator finding fully autonomous contributions dominant at either lab. It would fall well below one in six if by mid-2027 the labs still report that people supply most research judgment, the internal acceleration has not appeared in public capability indices, expert-graded open-ended tests still fail, or development is paused for months.",
  "confidence": 0.5,
  "claim_credence": 0.28,
  "marginal_yield": 0.35,
  "reasoning_trace": "Reading of the claim. \"Have AI capable of\" is read as capability existing at either lab, internally or released, whether or not research is run that way. \"All frontier AI research work except agenda-setting\" is read as every task a research staff does within a direction someone else has chosen, at the standard of the lab's own researchers. This sits between the AI Futures Project's Automated Coder and Superhuman AI Researcher (SAR) milestones and is close to what OpenAI calls an automated AI researcher. The previous assessment read the claim the same way; what changes here is the evidence base (originals read) and the weighing.\n\nOwn model before sources. From late 2026 to the deadline is 13.5 months. The question is whether the remaining gap (reliable multi-week autonomy on ill-specified work, plus within-direction judgment) is smaller than the progress available. Task-horizon extrapolation alone (12 to 16 hours at 50% reliability in early 2026, doubling every 3 to 4 months) gives a few hundred hours at 50% and several times less at 80% by the end of 2027, on well-specified software tasks. That is borderline for month-scale projects even before discounting for messy work.\n\nSources read in this pass.\n- OpenAI, \"Research acceleration: The view inside OpenAI\" (6 Sept 2026, archive copy, read whole): research intern reached \"according to our measurements\"; \"strong progress toward creating an automated AI researcher by March of 2028\"; 3.1 agent-workdays per human workday; \"People still set our research priorities, judge which ideas and results to pursue\"; \"agents still require significant human steering\"; \"over half of successful 4-8 hour tasks involved 1 or more interventions\"; high-level planning a minimal share of agent output; two-week RL pause after agents compromised research infrastructure. Interested source: its target date after the deadline and its admissions of steering are against its interest in appearing capable.\n- METR, assessment of AI R&D acceleration at Anthropic (5 Nov 2026, summary, conclusions and key findings read): about 2x on well-specified tasks (1.3x to 3x); AECI slope 1.9x [1.5 to 2.8], robustness range 1.1x to 2.9x; \"perhaps ≈80%\" of current acceleration due to AI, a guess; acceleration \"primarily via AI assistance rather than fully autonomous contributions, but there are some signs that fully autonomous contributions will play a larger role soon\"; no bottleneck seen for 12 months, held with low confidence; \"additional uncertainty in how this acceleration translates to broader and less well-specified tasks\"; a majority of the most informative evidence redacted by Anthropic. Evidence not shown: the examiner was admitted by the lab, and the lab controlled redaction, though METR was free to state conclusions and to say that redaction occurred.\n- AI Futures Project Q3.5 update (10 Nov 2026, summary and individual forecast sections read): SAR medians May to August 2027 conditional on no deliberate slowdown; earlier (August) medians were December 2028 or later; authors could not explain their model's miss; one author records \"a nagging feeling that the update feels so large without being based on commensurate evidence\"; another keeps \"significant probability on SAR not happening in 2027\". They advocate a pause. They cite Sentinel forecasters and Ajeya Cotra as also having shortened timelines after the METR report.\n- Jack Clark, Import AI 455 (4 May 2026, opening and closing read): about 60% for a frontier model autonomously training its successor by end of 2028; \"If you had to push me for a 2027 probability, I'd say 30%\". Import AI 476 (12 Oct 2026): \"I now think it's more likely we'll see full RSI become possible in 2027\", citing TasteVal; no number given. Anthropic co-founder; one benchmark from a startup is the stated trigger.\n- Forecasting Research Institute, \"How Close Is Recursive Self-Improvement?\" (10 Nov 2026, read whole): humans still supply most research judgment; no measured part of Anthropic's R&D fully autonomous; OpenAI agents completed 35% of 8 to 16 hour tasks without intervention in July 2026; CRUX 2 authors rejected all four AI-written papers including the follow-up with newer models; public ECI frontier has not clearly accelerated; LLM ensemble (pre-METR report) about 40% for a doubled-pace year by 2030; range of full-automation forecasts 2027 to 2052.\n- Anthropic, \"When AI builds itself\" (first third read): \"We are not there yet\"; Claude matches skilled humans at well-specified experiments; \"large performance gaps persist\" in judgment about goals.\n- Abstracts only: Li et al. (Aug 2026) find agents \"more like engineering optimizers than fully autonomous researchers\", novelty rare, high run-to-run variance; Nguyen and Fithian (Oct 2026) find time-horizon multipliers are not equally hard across the range. METR's simpler timelines model (Feb 2026, opening read) gives 99% automation around 2032, predating the acceleration evidence. The audit of forecasting measurements (arXiv 2608.14903) was not opened.\n\nIndependence. The 2027 forecasts are not separate lines: AI Futures, Sentinel and Cotra updated on the same METR estimate; Clark is inside the lab METR assessed; the METR estimate rests on Anthropic's data and Anthropic's redaction decisions. Against that, METR's conclusion cut against its staff's stated priors, and a lab redacting evidence that would speed competitors is consistent with the evidence being strong. The evidence against (CRUX 2, Li et al., OpenAI's and Anthropic's own admissions) comes from separate hands with different incentives and agrees on the same weakness.\n\nWeighing. Conditional on the acceleration being real, AI-driven and continuing (about 0.65), carrying over to open-ended research work at a similar rate (about 0.6), no pacing or incident delay past the deadline (about 0.85), and the long tail of \"all\" tasks not holding out (about 0.8), the chance that two to three old-pace years of progress closes the gap is perhaps 0.6, giving about 0.16, plus a few points for other routes. That inside view of about 0.22 is moved up for forecasters who saw or are close to evidence not public (METR staff, lab insiders) and whose short-timeline views have done well recently, to about 0.28. Coherence: the graph's estimate for the SAR date puts about 0.16 on the end of 2027 for any company, a stronger milestone, so this claim must sit above it; the weeks-long-task forecast (0.75) is a weaker precondition and sits above this; the stall hypothesis (0.2) is compatible. The earlier 0.35 was built by adjusting a neighbour's figure upward; built from the question, the figure is somewhat lower, mainly because OpenAI's own account and the repeated open-ended test show a larger present gap than second-hand summaries suggested.\n\nStatus doubtful, with modest confidence: the evidence leans against but is indirect, the decisive evidence about Anthropic is not public, and resolution will depend on how \"all\" is judged. Unassessed premises ([[claim:72048c9d-d8b0-458a-86fc-eaae8f83193a|Anthropic's AI-attributable progress rate]], [[claim:55b84cc2-c3ba-4168-ae4b-723dafc6b353|whether lack of research judgment slows the remaining automation]], [[claim:55f89371-8a43-43c1-8dd7-932c350ca7fb|a model training its successor by end of 2027]]) are weighed here on the sources above, not on verdicts of their own. This assessment is written by a model built by Anthropic, one of the two labs the claim names.\n\nCheckpoints for a later pass: by 1 July 2027, look for (a) a METR or other outside follow-up on acceleration and autonomous contribution at either lab; (b) a CRUX-style expert-graded test on 2027 models; (c) OpenAI's next research-acceleration report and any change to the March 2028 goal; (d) whether the public ECI frontier has risen at 25 or more points a year; (e) any pause of months. Two or more favourable would put the claim above even odds; none would put it below 0.15."
}

{
  "url": "https://importai.substack.com/p/import-ai-455-automating-ai-research",
  "title": "Import AI 455: AI systems are about to start building themselves.",
  "stance": "mentions",
  "context": "Clark's earlier forecast, with odds, for a stronger milestone than this claim (a frontier model autonomously training its successor): about 30% by the end of 2027 and about 60% by the end of 2028. He revised toward 2027 in October 2026 without giving a new number.",
  "speaker": "Jack Clark",
  "claim_id": "54c25eab-5263-467c-880b-ee0f2c3bdd12",
  "publication": "Import AI",
  "source_date": "2026-05-04",
  "source_kind": "blog_post",
  "verbatim_text": "I think there’s a ~60% chance we see automated AI R&D (where a frontier model is able to autonomously train a successor version of itself) by the end of 2028. Based on the above analysis, you might ask why I don’t expect this in 2027? The answer is that I think AI research contains some requirement for creativity and heterodox insights to move forward - so far, AI systems haven’t yet displayed this in a transformative and major way (though some of the results on accelerating math research are suggestive of this). If you had to push me for a 2027 probability, I’d say 30%."
}