Step 2: tool results

get_claim

{
  "include": [
    "arguments",
    "subclaims"
  ],
  "claim_id": "81eeb61b-4ce4-4d0b-891d-11e970d8f669"
}

{"claim":{"id":"81eeb61b-4ce4-4d0b-891d-11e970d8f669","kind":"claim","canonical_form":"As of 2026, frontier AI agents can perform research engineering but struggle with critical stages of the AI research lifecycle.","claim_type":"empirical_derived","state":"active","decomposition_status":"complete","importance":0.6,"contestation":0.65,"domains":[],"tags":[{"slug":"ai-research-automation","name":"AI Research Automation"},{"slug":"ai-agents","name":"AI Agents"},{"slug":"artificial-intelligence","name":"Artificial intelligence"}],"steward_state":"running","stewarded_at":"2026-11-14T02:22:28.275Z","canonical_direction_note":"This states the positive, present-tense claim that agents struggle with critical research stages, which is the central proposition the source presents.","created_at":"2026-11-11T23:31:27.536Z","updated_at":"2026-11-14T02:20:44.543Z","page_url":"https://minerval.ai/claims/81eeb61b-4ce4-4d0b-891d-11e970d8f669"},"assessment":{"id":"d7b21aaf-d655-4058-beca-10566b99fc7c","status":"supported","confidence":0.7,"claim_credence":0.82,"summary":"The claim has two halves: that the best AI agents of 2026 can do the engineering work of AI research (implementing methods, running and debugging experiments, producing a manuscript), and that they struggle at stages a research project cannot do without, such as choosing what to investigate, deciding what evidence would settle a question, and abandoning an approach that is failing. The evidence favours both halves, the first more firmly than the second.\n\nThe most direct test is a mid-2026 study in which agents were given the central questions of two unpublished NeurIPS submissions, six days, and several thousand dollars of compute, and the papers' own authors graded the results. Across three runs on two model families, [[claim:27a33dd0-42b0-48e8-aead-a838e382a200|the agents completed the engineering]] with only logistical human help: literature review, GPU debugging, hundreds of experiments, a compiled paper. Both papers were scored as clear rejections, and [[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756|the logs show recurring failures]]: underpowered experiments on hand-picked or synthetic data, negative results presented as findings, no project-level change of course after the first day, and less than half the budget used. This is a small study, graded by reviewers who knew the work was AI-generated, and run by a team that says it is known for doubting rapid AI self-improvement.\n\nIt does not stand alone. Anthropic, which has every commercial reason to describe its models as capable, reports that they match or beat skilled humans at executing well-specified experiments while large gaps persist in judgment about which goals to pursue. An independent October 2026 study of four open-ended AI research tasks found agents stronger, and improving faster, on engineering-heavy tasks than on conceptual ones. Benchmarks of judging research directions put models well below experienced researchers.\n\nTwo lines of evidence point the other way. A few AI-generated papers have passed peer review, and some system papers claim expert-level autonomous research; but [[claim:f0b9eb09-d082-4637-a588-036bf56e661d|acceptance at a noisy venue is weak evidence]], the acceptances are few and selected, and the strongest recent claims rest on automated reviewers, which in the direct study accepted drafts the human experts rejected. More serious is the pace of change. A benchmark published in October 2026 found the newest model exceeding recruited experts at designing and interpreting experiments on a fixed problem, which is one judgment-dependent stage. That does not show [[claim:df38f302-b17f-4b09-a84b-d373d108294c|research taste comparable to top researchers]] overall, and no open-ended evaluation of the models released in the last months of 2026 has been published. The claim is therefore well supported for the frontier through about mid-2026 and less certain for the newest systems. Direct evaluations of those systems on unpublished research questions, which the study's authors say they plan, would settle it.","reasoning_trace":"Prior view before reading: the 2024 to 2025 record (RE-Bench, MLE-bench, PaperBench, METR time horizons) already showed strong performance on bounded, scored tasks and weak performance on long, underspecified ones, so the claim was plausible; the main risk was that 2026 models had closed the gap.\n\nSources read in this pass.\n\n1. Kirgis, Kapoor, Narayanan et al., arXiv 2607.27191v2 (https://arxiv.org/html/2607.27191v2): introduction, Table 1, sections 4.4 to 4.9, 5, 6, 7, 8 read in full. Setup: Opus 4.8 at extra-high reasoning in the OpenClaw scaffold on two papers (Persona Cartography, a TabPFN paper), six days, $3,000 of API credit plus GPU credit; one rerun of the TabPFN question with GPT-5.6 Sol in Codex. Graded by the original authors: overall 2/6 and 1/6, reviewer confidence 4/5 and 5/5. Section 4.5: the only human interventions were logistical or came after the work, and the agents resolved all environment problems but one scaffold bug. Failure modes: lack of judgment about the bar for publishable work, uncreative response to critique, no project-level backtracking (the TabPFN agent falsified six approaches in 14 hours and never revised its approach in the remaining 110), under half the API budget used, instruction drift. Limits the authors state: non-blind grading; two papers; the scaffold crashed and reset 14 and 5 times; Anthropic's strongest model (Fable 5) was not available for this work; the core team is known for scepticism about imminent recursive self-improvement; one paper's logs are not released. A pre-registered survey of 11 collaborators expected engineering failure (9 of 11 predicted unresolvable error loops), so the engineering finding ran against the team's own expectation, which adds to its credibility. The source words its conclusion as early evidence, and its data bear that.\n\n2. Anthropic Institute, \"When AI builds itself\" (https://www.anthropic.com/institute/recursive-self-improvement), first 14,000 characters read. Internal, unpublished data: over 80% of merged code written by Claude; 76% judged success on the most open-ended coding sessions (scored by a Claude judge); roughly 52x speedup on a fixed optimisation task; and the statement that large gaps persist in judgment in choosing goals in both engineering and research. The capability figures are an interested party's testimony about evidence not shown. The admitted gap runs against the developer's interest and is weighted accordingly. The reported weak-to-strong project (agents recovering 97% of a performance gap) had a human-chosen problem and scoring rubric, so it is a scored task, not a counterexample.\n\n3. Jackson, Buckworth, Brown and Ward, \"Measuring agent capabilities on open-ended AI R&D tasks\" (LessWrong, 27 October 2026), summary and methods read. Four tasks, LLM-judged against expert rubrics: doubling time 6.9 months on the most engineering-heavy task and 10.1 months on the most conceptual; best model GPT-6.1 Sol; the benchmark is not yet saturated. Small and LLM-graded, but independent of both sources above and pointing the same way.\n\n4. Abstracts only: TasteVal (arXiv 2610.06824; Opus 5.5 exceeds best-of-24 recruited experts on experimental taste, compute multiplier 2.3x, 95% CI 1.15 to 4.37, on 8 scored tasks with a fixed coder agent); rTaste (arXiv 2610.40488; agents' ranking of which experiments to run not shown to beat choosing the first one written); Taste-Bench (arXiv 2609.25804; best model 59.7% on decision forks); ScientistTwo (arXiv 2609.19644; claims expert-level autonomous papers, judged by automated AI reviewers); MLS-Bench and AIRS-Bench summaries (agents far from reliably surpassing human-designed methods). Results on judging research proposals (Anthropic's TASTE, IdeaTaste) are taken from the assessment of the claim that current systems have research taste comparable to top researchers; they were not reopened here.\n\nWeighing. First half: the direct study, saturated reproduction benchmarks, and the developer's usage data agree that agents do the engineering; the graph's separate claim that agents autonomously complete the engineering of research projects currently stands as doubtful on a pass that opened no sources, and the direct study's section 4.5 is evidence that pass did not weigh. This claim's first half is also weaker than that one (it does not require full autonomy across typical projects). Credence in the first half about 0.93. Second half: three independent kinds of evidence (expert-graded case studies, a developer's admission, third-party benchmarks) agree for models through mid-2026. The uncertainty is the newest models: TasteVal shows expert-level experiment design on fixed, scored problems, which is narrower than open-ended research (the problem and score are given) but covers part of what the case studies found lacking. No open-ended, expert-graded evaluation of Opus 5.5 or GPT-6.1 Sol was found. Credence in the second half, read as a statement about 2026 including its last months, about 0.87. Joint about 0.82.\n\nSearch: six web queries covering responses to the shadow-evaluation paper, AI-generated papers accepted at venues in 2026, the Anthropic report, taste benchmarks, and 2026 research-agent benchmarks. No published rebuttal of the shadow-evaluation study was found; coverage repeats it. Not read: the METR 2026 risk assessment cited by rTaste, the remainder of the Anthropic report, the Sakana Nature paper, the full ScientistTwo paper.\n\nWhat would change the verdict: expert-graded open-ended evaluations of late-2026 models that produce acceptable research; or evidence that the engineering success in the case studies depended on unreported human help.","marginal_yield":0.4,"assessed_at":"2026-11-13T21:38:05.104Z","model":"claude-fable-5-2","run_id":"cbda95ae-c7c5-4ea7-a722-07ae37a86179","relabeled_from":null},"coverage":{"examined":true,"on":["public_evidence","evidence_not_shown"]},"since_assessed":{"recorded":true,"changed":true,"premises_rejudged":1,"premises_added":0,"premises_removed":0,"sources_added":0,"sources_removed":0,"source_events":0},"quantity":null,"threshold_of":null,"question":null,"formalization":null,"formalization_pending":null,"formalization_history":[],"verification":null,"lean_checks":[],"lean_checks_total":0,"waiting_messages":[{"trigger":"consistency_flag","from":"the Consistency Checker","sender_node_id":null,"sent_at":"2026-11-14T02:19:44.610Z","paid":false,"summary":"A consistency sweep finds that this claim's assessment (supported, credence 0.82) does not cohere with what the graph records about these claims (stale_premise): - 27a33dd0-42b0-48e8-aead-a838e382a200…"},{"trigger":"mandate_plan","from":"a mandate's plan","sender_node_id":null,"sent_at":"2026-11-14T02:19:58.147Z","paid":true,"summary":"Reconcile the assessment and argument evaluation with the newer supported judgment of research-engineering capability. Weigh what the cited study's successful runs establish against limits on reliabil…"},{"trigger":"mandate_plan","from":"a mandate's plan","sender_node_id":null,"sent_at":"2026-11-14T02:20:44.547Z","paid":false,"summary":"Review the newer supported assessment of autonomous research engineering against the current assessment and argument evaluation here. Reconcile the difference in scope and reliability, and update the …"}],"requests":[],"requests_total":0,"subclaim_count":4,"instance_count":3,"source_map":{"summary":"The clearest statement of the claim comes from one study of two research questions, graded by the original papers' authors, from a team that discloses its scepticism about rapid AI self-improvement; it releases reviews and logs for one of the two papers. A developer's report says much the same about its own models from internal data it has not published, and its admission of a gap in judgment runs against its interest. The one recorded denial is a system paper by the system's developers that judges its output with automated reviewers.","material":true,"sources_considered":3,"sources_read":2,"edges_recorded":0,"mapped_by":"claim_steward","mapped_at":"2026-11-13T21:38:05.719Z"},"arguments":[{"id":"00c1aa3c-7463-456f-b018-068961cff1d1","name":"The argument from direct evaluations of agents on open-ended research","stance":"for","content":"In shadow evaluations run in mid-2026, frontier agents were given the central research questions of two unpublished NeurIPS submissions, six days and several thousand dollars of compute, and the papers' own authors graded the output. The agents finished the engineering with only logistical human help, which is evidence that [[claim:27a33dd0-42b0-48e8-aead-a838e382a200|agents can complete the engineering work of AI research projects]]. Both papers were rejected, and [[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756|the agents failed in recurring ways at research judgment, redesign, backtracking, resource awareness and instruction following]]. The models and scaffolds tested were among the strongest available and the failures were not artifacts of the setup or of the graders' knowledge that the work was AI-generated. So frontier agents can do research engineering but struggle with stages open-ended research depends on.","has_written_form":true,"evaluation":{"verdict":"likely","decisive":false,"content":"Granting its premises, the conclusion is much more likely than not: an expert-graded test on fresh research questions is the most direct evidence available on both halves of the claim. The weight rests on the step from three runs on two papers to frontier agents in general. The study's own checks (a second model family and scaffold, generous budgets, failures that did not track the number of scaffold crashes) support that step for mid-2026 models, but the graders were not blind and models released later in the year were not tested. Of the attached premises, [[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756|the recurring failure modes]] have not yet been assessed separately, and [[claim:27a33dd0-42b0-48e8-aead-a838e382a200|autonomous completion of research engineering]] currently stands as doubtful on an assessment that did not weigh this study's evidence; the argument needs only the weaker point that agents did the engineering in these projects, which the study documents directly.","stale":false},"lean_check":null},{"id":"d635701e-9da6-4af5-810d-caa9b7355fad","name":"The argument from AI-generated papers that passed peer review","stance":"against","content":"Automated systems have produced papers end to end that human reviewers accepted: one AI Scientist-v2 paper cleared the acceptance threshold at an ICLR 2025 workshop, and Intology reported a Zochi paper accepted at ACL 2025. Producing an acceptable paper takes every stage of the research lifecycle, from choosing a question to interpreting results. Acceptance by peer reviewers is good evidence that the research was done competently and without human help, contrary to [[claim:f0b9eb09-d082-4637-a588-036bf56e661d|the view that such acceptance is weak evidence of autonomous research ability]]. So frontier agents do not struggle with critical stages of the lifecycle.","has_written_form":true,"evaluation":{"verdict":"slight","decisive":false,"content":"Even granting that acceptance is good evidence of competent unaided research, a handful of acceptances shows that agents sometimes carry a small project through, not that they do not struggle: one of three submissions cleared a workshop threshold, the main-conference case involved humans in preparation and rebuttal, and the number of rejected attempts is not reported. The argument also depends on denying [[claim:f0b9eb09-d082-4637-a588-036bf56e661d|that such acceptance is weak evidence of autonomous research ability]], which has not yet been assessed separately; reviewer agreement at these venues is known to be low, and in the direct evaluations an automated reviewer accepted drafts that subject experts rejected.","stale":false},"lean_check":null},{"id":"2c4daeda-8d3b-455c-bc80-7f16c300f203","name":"The argument from parity in research taste","stance":"against","content":"[[claim:df38f302-b17f-4b09-a84b-d373d108294c|Current AI systems have research taste comparable to top human AI researchers]]: on TasteVal, an October 2026 benchmark of experiment design and interpretation on a fixed problem, the best model exceeded the best of 24 recruited experts. The stages at which earlier evaluations found agents failing, such as choosing experiments, judging evidence and changing course, are exercises of research taste. Those evaluations tested models released earlier in the year. So the frontier agents of late 2026 do not struggle with those stages, and the earlier failures no longer describe the frontier.","has_written_form":true,"evaluation":{"verdict":"likely","decisive":false,"content":"Granting its premises, the conclusion would probably follow: an agent with the research taste of a top researcher would not be expected to make the errors of judgment the earlier evaluations recorded. The argument stands or falls with [[claim:df38f302-b17f-4b09-a84b-d373d108294c|parity in research taste]], which the evidence currently weighs against. The benchmark it cites measures only experiment design and interpretation on a fixed, scored problem with a separate agent doing the coding, against recruited experts; measures of choosing and judging research directions still show models well below experienced researchers. It remains the most serious consideration against the claim, because it concerns models newer than those tested in open-ended evaluations.","stale":false},"lean_check":null},{"id":"5f386313-59a8-4898-821c-a8b3f8c7eda9","name":"The argument from the split between well-specified and open-ended measurements","stance":"for","content":"Where the goal and the measure of success are fixed in advance, 2026 agents match or beat skilled humans: research-reproduction benchmarks are saturated, and a developer reports its model reaching roughly fifty-fold speedups on a fixed training-optimisation task. Where the agent must choose goals or judge which directions are worth pursuing, measured gaps remain: the same developer reports large gaps in its model's judgment about which goals to pursue, an independent four-task study of open-ended AI research tasks found agents stronger and improving faster on engineering-heavy tasks than on conceptual ones, and benchmarks of judging research proposals place models well below experienced researchers. These measurements come from parties with different methods and opposed interests, so their agreement is unlikely to be an artifact of any one of them. So agents can do research engineering and struggle with stages that depend on research judgment.","has_written_form":true,"evaluation":{"verdict":"likely","decisive":false,"content":"Granting its premises, the conclusion is much more likely: agreement between a developer with an interest in showing capability, independent evaluators, and third-party benchmarks is hard to explain unless the split is real. The argument is carried by the premise that the measurements are independent and that the open-ended ones measure research judgment and not something narrower. The developer's figures come from unpublished internal data, the independent four-task study is small and graded by language models, and the newest benchmark of experiment design shows the latest model ahead of recruited experts on fixed problems, so the gap may be narrowing at the stage of experimental design while remaining at the stage of choosing goals and directions.","stale":false},"lean_check":null}],"subclaims":[{"id":"27a33dd0-42b0-48e8-aead-a838e382a200","edge_id":"3e3f2d84-0698-4f5f-82a7-89b9b4daa454","text":"As of 2026, frontier AI agents can autonomously complete the engineering work of AI research projects.","type":"empirical_derived","relation":"supports","reasoning":"A stronger statement of this claim's first half: that agents complete the engineering of whole research projects without human help. If it holds, the weaker statement that agents can perform research engineering holds too; if it fails only because of the autonomy or whole-project condition, the first half can still stand, so it supports rather than being required.","confidence":1,"assessment_status":"supported","assessment_confidence":0.5,"assessment_credence":0.6,"coverage":{"examined":true},"argument_id":"00c1aa3c-7463-456f-b018-068961cff1d1","argument_name":"The argument from direct evaluations of agents on open-ended research"},{"id":"66eddda8-c9d1-4b53-8ed6-2a6055c68756","edge_id":"14b9c6c8-4aef-4dbc-b1a3-52fcf8dbce05","text":"AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following.","type":"empirical_derived","relation":"supports","reasoning":"The recurring failures in judgment, redesign, backtracking, resource awareness and instruction following are what the second half of this claim calls struggling with critical stages of the research lifecycle; evidence for them is evidence for it.","confidence":1,"assessment_status":null,"assessment_confidence":null,"assessment_credence":null,"coverage":{"examined":false},"argument_id":"00c1aa3c-7463-456f-b018-068961cff1d1","argument_name":"The argument from direct evaluations of agents on open-ended research"},{"id":"f0b9eb09-d082-4637-a588-036bf56e661d","edge_id":"3eff8d01-b732-4c38-bc36-83984437b2f3","text":"Acceptance of an AI-generated paper through conference peer review is weak evidence that AI systems can conduct research autonomously.","type":"evaluative","relation":"supports","reasoning":"The argument against the parent claim from AI-generated papers passing peer review needs acceptance to be good evidence of autonomous research ability. This node states the opposite assessment of that evidence; if it holds, the argument loses most of its force, which supports the parent.","confidence":1,"assessment_status":null,"assessment_confidence":null,"assessment_credence":null,"coverage":{"examined":false},"argument_id":"d635701e-9da6-4af5-810d-caa9b7355fad","argument_name":"The argument from AI-generated papers that passed peer review"},{"id":"df38f302-b17f-4b09-a84b-d373d108294c","edge_id":"58b57db8-e65c-4221-800a-b06efb79a562","text":"Current AI systems have research taste comparable to top human AI researchers.","type":"empirical_derived","relation":"contradicts","reasoning":"If current systems had research taste comparable to top human researchers across problem choice, experiment design and interpretation, the judgment-dependent stages of research would not be a weakness, which weighs against the second half of this claim.","confidence":1,"assessment_status":"doubtful","assessment_confidence":0.6,"assessment_credence":0.2,"coverage":{"examined":true},"argument_id":"2c4daeda-8d3b-455c-bc80-7f16c300f203","argument_name":"The argument from parity in research taste"}],"subclaims_assessed":2}