Step 29: assistant

thinking

Calls update_claim_assessment

{
  "status": "doubtful",
  "claim_id": "55f89371-8a43-43c1-8dd7-932c350ca7fb",
  "assessment": "The claim forecasts an event: that before the end of 2027 a frontier AI model will itself carry out the training of its successor, end to end, without humans doing the research and engineering. Its best-known source is Jack Clark of Anthropic, who in May 2026 put about 60% on \"automated AI R&D (where a frontier model is able to autonomously train a successor version of itself)\" by the end of 2028 and 30% on 2027, citing missing creativity as the reason 2027 was unlikely. In October 2026, after a benchmark reported frontier models matching or exceeding human experts at designing and interpreting AI experiments, he wrote that full recursive self-improvement now seemed more likely to become possible in 2027 than in 2028.\n\nTwo distinctions matter. First, Clark's forecasts are about ability (\"is able to\", \"become possible\"), while the claim states that the training will actually happen. Second, his view is a forecast from a co-founder of a frontier lab that benefits from perceptions of rapid progress. That stake does not make him wrong, and his May essay rests on public benchmarks that anyone can check, but the revision rests on a single new benchmark and gives no number.\n\nIndependent evidence as of November 2026 favors rapid partial automation but not end-to-end autonomy. AI writes most code at leading labs and agents complete a growing share of multi-hour tasks unaided, and METR estimates AI is roughly doubling Anthropic's pace of progress. Yet [[claim:d479cf4d-52b6-4260-a3dc-7a2e5a812d42|humans still supply most research judgment]]: no measured part of Anthropic's R&D is fully autonomous, agents recovered only a small share of the gain in tests of reinventing unseen ML innovations, and the authors of unpublished papers rejected every AI-written paper on their questions. OpenAI's own target for an automated AI researcher is March 2028, after this claim's deadline. Even if the capability arrives in 2027, [[claim:4279b6b3-295f-4dee-8793-74610abbe760|developers are likely to keep humans overseeing frontier training runs]], and a public demonstration would most likely come first at a smaller, non-frontier scale, as Clark himself expected.\n\nThe evidence therefore weighs against the claim as worded, with a credence of roughly one in eight. A demonstrated, publicly documented end-to-end successor training of a frontier model, or a sharp jump in independent open-ended research evaluations in 2027, would change the verdict. How \"autonomously\" and \"successor\" are read also matters: a loose reading counting heavy AI contribution under human direction would make the claim much more likely, but that is not what the words say.",
  "confidence": 0.65,
  "claim_credence": 0.12,
  "marginal_yield": 0.35,
  "reasoning_trace": "Own model first: the claim needs (a) capability for end-to-end research and engineering of a frontier training run, including research judgment, (b) a lab actually letting a model do this without human involvement, and (c) this happening by 31 December 2027, about 13.5 months from today, and being knowable. Claims of this kind (near-term transformative AI milestones by named dates) have historically resolved late more often than early, though the current trend is unusually fast.\n\nSources read:\n- Import AI 455 (Jack Clark, 4 May 2026), read whole. Gives ~60% by end-2028 and 30% for 2027 for \"a frontier model is able to autonomously train a successor version of itself\"; notes a proof of concept is more likely at non-frontier scale; reason for lower 2027: creativity and heterodox insight not yet shown. Evidence offered is public benchmarks (SWE-Bench, METR time horizons, CORE-Bench etc.) and lab stated goals.\n- Import AI 476 (Clark, 12 Oct 2026). Cites TasteVal (P-Zero Research): experimental research taste doubling every ~3 months, best model exceeding a 24-expert human baseline at 2.3x compute efficiency on 8 fixed tasks. Clark: \"more likely we'll see full RSI become possible in 2027\". This is capability language, no number, and rests largely on one new benchmark whose tasks are given problems, not open-ended problem choice.\n- Forecasting Research Institute, \"How Close Is Recursive Self-Improvement?\" (10 Nov 2026). Summarizes company data: Claude wrote >90% of Anthropic codebase contributions (Sept 2026); OpenAI agents 3.1 workdays per human workday (Aug 2026); OpenAI 57% of 2-4h and 35% of 8-16h tasks fully unassisted (July 2026). But \"no measured part of Anthropic's AI R&D is fully autonomous\"; Anthropic reports large gaps in judgment in choosing goals. Public evals: InnovationEval 15% recovery (GPT-5.6 Sol); CRUX 2 all four AI papers rejected by original authors. OpenAI claims automated research intern reached Sept 2026, targets automated AI researcher March 2028. METR: ~2x progress at Anthropic, ~80% due to AI. Range of full-automation forecasts 2027-2052.\n\nWeighing: Trend argument is real and the pace is fast, but company metrics measure share of tasks under human direction, not autonomy of whole projects; the last steps (goal choice, research judgment) are what remain and are what independent evals show weakest. The lab most associated with the goal sets its own target after 2027. Deployment gap: a frontier run costs very large sums and alignment concerns (which Clark himself lists) make no-human-involvement runs unlikely to be permitted soon after capability. Interested-source weighting: Clark's May number (30%) is about capability and is from an interested insider; translating to the event lowers it. His October revision perhaps implies roughly 40-55% for capability in 2027; times a probability the event actually occurs and is publicly established given capability (~0.25-0.35) gives roughly 0.10-0.18; independent evals pull toward the lower end. Credence 0.12. Status doubtful rather than contradicted because this is a forecast whose evidence is indirect and the trend is steep. Consistency: the 2028 sibling claim must carry a credence at least as high.\n\nWhat would change it: a public end-to-end successor training demonstration at frontier scale, big gains on open-ended evaluations (InnovationEval-type, CRUX), or labs announcing autonomous training runs. Ambiguity of \"autonomously\" is the main residual uncertainty in the verdict."
}