Step 12: tool results

{
  "status": "supported",
  "claim_id": "20f15464-de92-44a7-8792-b4f159bd97c8",
  "assessment": "The claim describes the work in frontier AI research and development that AI systems do not yet do on their own. It says most of that work needs research judgment (choosing goals and directions, deciding which experiments matter, interpreting results, knowing when to redesign) rather than engineering execution (writing, running and debugging code to a given specification).\n\nThe evidence leans in its favour, though indirectly. Company-reported figures for late 2026 show AI writing over 90% of codebase contributions at Anthropic and doing about three agent-workdays per human workday at OpenAI. Yet [[claim:d479cf4d-52b6-4260-a3dc-7a2e5a812d42|humans still supply most research judgment]], and Anthropic reports large gaps in its model choosing goals. Public benchmarks point the same way: agents do well on well-defined ML tasks but poorly on open-ended research, recovering a small share of a published innovation's gain and producing papers that the original authors rejected. If execution is largely automated and judgment largely is not, the human-held residual is weighted toward judgment. The Forecasting Research Institute's November 2026 review frames the remaining tasks as increasingly requiring research judgment.\n\nThree things keep this short of established. First, much execution is still not autonomous: OpenAI's agents completed only 35% of 8 to 16 hour tasks without human intervention, and no measured part of Anthropic's R&D is fully autonomous. Long-horizon engineering and coordination therefore make up a real part of the residual. Second, Anthropic places its judgment gaps \"in both engineering and research\", and the line between judgment and execution is blurred: much of what stalls a long engineering task is judgment inside it. How much the claim holds depends on where that line is drawn. Third, nobody has published a task-level breakdown of the residual. The evidence is mostly company-reported and comes from two labs. A census of human interventions on unfinished agent tasks, sorted by kind, would settle the question.",
  "confidence": 0.5,
  "claim_credence": 0.6,
  "marginal_yield": 0.35,
  "reasoning_trace": "Prior: once code-writing is largely automated, the residual human work should be weighted toward what models are known to do worst, open-ended judgment. Whether it is \"most\" depends on how much execution is still unfinished and how the categories are drawn.\n\nEvidence read: the FRI post \"How Close Is Recursive Self-Improvement?\" (Kučinskas and Rosenberg, 10 Nov 2026), read in full. Supporting: Claude wrote over 90% of codebase contributions (Sept 2026); OpenAI had 3.1 agent-workdays per human workday (Aug 2026); Anthropic says \"large performance gaps persist when it comes to Claude exercising judgement in choosing goals in both engineering and research\"; strong results on PostTrainBench, WeirdML and TasteVal; weak results on InnovationEval (15% recovery) and CRUX 2 (all four AI papers rejected). FRI itself says remaining tasks \"increasingly require research judgment\", which is weaker than \"most\".\n\nAgainst or qualifying: OpenAI reports only 57% of 2 to 4 hour tasks and 35% of 8 to 16 hour tasks completed without human intervention, so much longer engineering work still needs humans. No measured part of Anthropic's R&D is fully autonomous. Anthropic's quote locates the judgment gaps in engineering as well as research, which blurs the dichotomy. TasteVal shows narrow experiment selection reaching expert level, so some judgment is automating too.\n\nThe linked subclaim that humans supply most research judgment (supported, 0.85) supports this claim but does not entail it: humans could supply most judgment while judgment is still a minority of residual tasks.\n\nInterested sources: the key figures are company self-reports (Anthropic, OpenAI) carried by FRI. Their direction (admitting gaps) cuts against the companies' promotional interest, which gives the admissions some weight.\n\nVerdict: supported at low confidence, credence 0.6. The categorization is fuzzy, and there are no task-level data on the residual."
}

{
  "claim_id": "20f15464-de92-44a7-8792-b4f159bd97c8",
  "reasoning": "A supporting premise inside the live debate over whether AI R&D automation slows below trend. It matters locally, and is moderately disputed mainly over how to draw the line between judgment and execution.",
  "importance": 0.4,
  "contestation": 0.4
}

{
  "claim_id": "20f15464-de92-44a7-8792-b4f159bd97c8",
  "reasoning": "Kept the claim atomic, with no named arguments. Linked \"humans supply most research judgment\" as a supporting subclaim and added a see-also to the claim on humans setting directions. Recorded the FRI post as an affirming instance at reduced confidence, since it says \"increasingly\" rather than \"most\". Assessed as supported (confidence 0.5, credence 0.6). Importance 0.4, contestation 0.4. The parent's assessment already assumed a credence near this, so no notification is needed.",
  "action_taken": "first_assessment"
}

{
  "answers": [
    {
      "note": "Structured minimally and assessed as supported at credence 0.6.",
      "message": "m1",
      "outcome": "changed"
    },
    {
      "note": "Assess pass completed: supported, confidence 0.5, credence 0.6.",
      "message": "m2",
      "outcome": "changed"
    }
  ]
}