Step 4: tool results

{
  "status": "supported",
  "claim_id": "81eeb61b-4ce4-4d0b-891d-11e970d8f669",
  "assessment": "The claim has two halves. The first is that the best AI agents of 2026 can do the engineering work of AI research: implementing methods, running and debugging experiments, and producing a manuscript. The second is that they struggle at stages a research project cannot do without, such as choosing what to investigate, deciding what evidence would settle a question, and abandoning an approach that is failing. The evidence favours both halves, the first more firmly than the second.\n\nThe most direct test is a mid-2026 study. Agents were given the central questions of two unpublished NeurIPS submissions, six days, and several thousand dollars of compute, and the papers' own authors graded the results. Across three runs on two model families, the agents did the engineering with only logistical human help: literature review, GPU debugging, hundreds of experiments, a compiled paper. The graph's stronger claim that [[claim:27a33dd0-42b0-48e8-aead-a838e382a200|agents can autonomously complete the engineering work of AI research projects]] is judged supported on a capability reading, though not on a reading that requires reliable completion across typical projects. The first half of this claim asks less than that one, because \"can perform research engineering\" does not require whole-project autonomy or reliability. So it holds with more confidence. Both papers were scored as clear rejections, and [[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756|the logs show recurring failures]]: underpowered experiments on hand-picked or synthetic data, negative results presented as findings, no project-level change of course after the first day, and less than half the budget used. The study is small. Its graders knew the work was AI-generated, and it was run by a team that says it is known for doubting rapid AI self-improvement.\n\nIt does not stand alone. Anthropic has every commercial reason to describe its models as capable. It reports that they match or beat skilled humans at executing well-specified experiments, while large gaps persist in judgment about which goals to pursue. An independent October 2026 study of four open-ended AI research tasks found agents stronger, and improving faster, on engineering-heavy tasks than on conceptual ones. Benchmarks of judging research directions put models well below experienced researchers.\n\nTwo lines of evidence point the other way. First, a few AI-generated papers have passed peer review, and some system papers claim expert-level autonomous research. But [[claim:f0b9eb09-d082-4637-a588-036bf56e661d|acceptance at a noisy venue is weak evidence]], the acceptances are few and selected, and the strongest recent claims rest on automated reviewers. In the direct study, such reviewers accepted drafts that the human experts rejected. More serious is the pace of change. A benchmark published in October 2026 found the newest model exceeding recruited experts at designing and interpreting experiments on a fixed problem, which is one judgment-dependent stage. That does not show [[claim:df38f302-b17f-4b09-a84b-d373d108294c|research taste comparable to top researchers]] overall, and no open-ended evaluation of the models released in the last months of 2026 has been published. The claim is therefore well supported for the frontier through about mid-2026 and less certain for the newest systems. Direct evaluations of those systems on unpublished research questions, which the study's authors say they plan, would settle it.",
  "confidence": 0.7,
  "claim_credence": 0.82,
  "marginal_yield": 0.35,
  "reasoning_trace": "This pass was prompted by the reassessment of the stronger neighbouring claim that agents can autonomously complete the engineering work of AI research projects, from doubtful (0.3) to supported (0.6). The earlier assessment here was written before that change. It said the neighbour stood as doubtful on a pass that had not weighed the Kirgis et al. evidence. That description is now out of date and has been corrected. The verdict does not change.\n\nReconciling the two judgments. The neighbour reads \"can\" as demonstrated capability and rests mainly on the same study (Kirgis, Kapoor, Narayanan et al., arXiv 2607.27191v2, section 4.5): in all three runs the agents resolved environment problems themselves and carried out the experiments, and human interventions were logistical. The neighbour notes that on a reliability reading (typical projects, consistent completion) it would still be doubtful. This claim's first half, \"can perform research engineering,\" is weaker than the neighbour on both axes. It does not require whole-project autonomy, and it does not require reliability. So evidence for the neighbour carries over here, and doubts about the neighbour's reliability reading do not. The 0.6 on the neighbour and roughly 0.93 on this first half are jointly coherent, because this half is implied by the neighbour and also has independent support: saturated reproduction benchmarks, the developer's reports on well-specified experiment execution, and the engineering-heavy side of the independent four-task study. What the successful runs establish is the occasional, demonstrated completion of engineering on real research questions. They do not establish broad, reliable capability across projects, and this claim does not assert that.\n\nThe second half is unaffected by the neighbour's change. It rests on the expert-graded rejections, the recurring failure modes, the developer's admission of judgment gaps, and third-party taste and direction benchmarks, against the October 2026 TasteVal result for the newest model on fixed problems. Credence in the second half, read as covering late 2026, is about 0.87. The joint credence is about 0.82, as before.\n\nSources were as in the previous pass (Kirgis et al. read in full in sections 4 to 8; the Anthropic Institute report read in part; Jackson et al., October 2026, summary and methods; abstracts of TasteVal, rTaste, Taste-Bench and ScientistTwo). No new sources were opened in this pass, because the change was in the graph's judgment of a premise, not in the evidence.\n\nWhat would change the verdict: expert-graded, open-ended evaluations of late-2026 models that produce acceptable research, or evidence that the engineering success in the case studies depended on unreported human help."
}

{
  "verdict": "likely",
  "decisive": false,
  "evaluation": "Granting its premises, the conclusion is much more likely than not, because an expert-graded test on fresh research questions is the most direct evidence available on both halves of the claim. The weight rests on the step from three runs on two papers to frontier agents in general. The study's own checks support that step for mid-2026 models: a second model family and scaffold, generous budgets, and failures that did not track scaffold crashes. But the graders were not blind, and models released later in the year were not tested. [[claim:27a33dd0-42b0-48e8-aead-a838e382a200|Autonomous completion of research engineering]] is now judged supported as a statement of demonstrated capability, though not of reliable completion across typical projects. The argument needs only the weaker point, that agents did the engineering in these projects, so that premise holds for its purposes. [[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756|The recurring failure modes]] have not yet been assessed separately, and the second half of the conclusion leans on them.",
  "argument_id": "00c1aa3c-7463-456f-b018-068961cff1d1"
}