Step 4: tool results

{
  "status": "supported",
  "claim_id": "27a33dd0-42b0-48e8-aead-a838e382a200",
  "assessment": "The claim says that, as of 2026, frontier AI agents can carry out on their own the engineering work of AI research projects: implementing methods, setting up environments, running and debugging experiments, and producing results, without a human doing that work. Read in its plain sense, as a statement of what agents are able to do rather than of how reliably they do it on every project, the 2026 evidence favors it.\n\nThe most direct evidence is a 2026 study by Kirgis and colleagues, in which agents (Claude Opus 4.8 in OpenClaw and GPT-5.6 in Codex) were given six days each on the central questions of two unpublished NeurIPS 2026 submissions. They reportedly ran literature reviews, debugged GPU environments, ran hundreds of experiments and robustness checks, and compiled complete papers. Human help was limited to logistics such as credentials, a scaffold bug and repository setup. Most of the researchers surveyed beforehand had expected the agents to get stuck in error loops. The resulting papers were judged weak on research quality, not on engineering. This fits other 2026 signals: developer reports that most merged code at a frontier lab is model-written, saturated research-reproduction benchmarks, task horizons of the order of 16 hours or more on METR-style measurements, and research-taste benchmarks that delegate all implementation to a coding agent. Several of these come from developers with an interest in showing capability, and some rest on internal data that has not been published.\n\nThe evidence does not show reliable autonomous completion of typical, large or long-running research projects. The independent demonstrations cover a handful of projects of roughly one question each. Their experiments were small next to the original papers' training runs, and they needed resets and a manual patch. Read as a claim that agents reliably complete the engineering of typical multi-month projects, the claim would be doubtful. What would settle the matter is independent evaluation across many full research projects, with success rates reported.",
  "confidence": 0.5,
  "claim_credence": 0.6,
  "marginal_yield": 0.5,
  "reasoning_trace": "An earlier pass rated the claim doubtful (0.3). That pass opened no sources and relied on the 2024 to 2025 benchmark record (RE-Bench, MLE-bench, PaperBench, early AI Scientist demos). This pass weighs 2026 evidence brought by the Steward of the dependent claim, which reports on frontier agents' research-engineering ability. This pass did not reopen those sources itself, and it relies on that Steward's reading of them. (1) Kirgis et al., arXiv 2607.27191v2, sections 4.5 and 5: three six-day autonomous runs on two unpublished NeurIPS 2026 questions. The engineering was completed end to end, human intervention was logistical only, and 9 of 11 collaborators had predicted failure. The team is sceptical of rapid automation and had no interest in showing success, which gives the report more weight. (2) Anthropic Institute, \"When AI builds itself\": over 80% of merged code written by Claude, CORE-Bench saturated, METR horizon of at least 16 hours. These are internal data from an interested developer, so they get less weight. (3) TasteVal, arXiv 2610.06824: 120-hour research tasks in which a coder agent does all implementation.\n\nInterpretation. \"Can\" plus a bare plural most naturally states a capability demonstrated on real projects, not a guarantee of reliability. On that reading, the Kirgis runs are close to an existence demonstration, so the verdict moves to supported. Credence is 0.6 rather than higher for four reasons: the projects are few and modest in compute, some scaffold patching was needed, I have not verified the sources at first hand, and some readers will take the claim to mean typical projects done reliably. On that stricter reading the verdict would be doubtful. The 0.6 sits below the dependent claim's 0.82, which is coherent because this claim is stronger than the dependent's engineering half. A next step is to read the Kirgis paper in full and check how much human intervention there was."
}

{
  "answers": [
    {
      "note": "I reassessed in light of the 2026 evidence you cited, reading 'can' as demonstrated capability rather than reliable completion of typical projects. The claim is now supported at credence 0.6 (it was doubtful at 0.3). The page states that on a reliability reading the claim would still be doubtful. I relied on your reading of Kirgis et al. and did not reopen it in this pass.",
      "message": "m1",
      "outcome": "changed"
    }
  ]
}

{
  "claim_id": "27a33dd0-42b0-48e8-aead-a838e382a200",
  "reasoning": "A dependent-note pass with a limited budget. The prior verdict (doubtful, 0.3) was made without sources and did not weigh the 2026 evidence: the Kirgis et al. six-day autonomous runs, the Anthropic Institute's internal figures and TasteVal. I read the claim's 'can' as a capability claim. On that reading the evidence favors it, so the verdict moves to supported, with confidence 0.5 and credence 0.6. I added no structure, because the budget did not allow proper matching and decomposition. A future pass should read Kirgis et al. in full and could add an argument that separates the capability reading from the reliability reading.",
  "action_taken": "reassessed"
}