Run step

Step 12: tool results

2026-11-14 02:36:38 UTC · previous step · next step

write_argument

Input

{
  "content": "Evaluations built to isolate the choice of what to do next in AI research show later models doing better than earlier ones. In Anthropic's internal test on 129 detours from real research sessions, the share of cases where a model's proposed next step was judged better than the researcher's rose from 51% (Claude Opus 4.5, November 2025) to 64% (Claude Mythos Preview, April 2026). On the external TasteVal benchmark, where the evaluated model designs experiments while a fixed coding agent runs them, the compute efficiency of frontier models relative to human experts has risen across 20 models released from 2023 to 2026. If these evaluations measure next-step choice rather than coding or knowledge, and their trends are not artifacts of the judging or task selection, then AI systems are improving at choosing next steps in AI research.",
  "argument_id": "3215a9b0-c9a2-4226-b7ad-691138a3293c"
}

{"success":true,"message":"Written form saved for argument \"The argument from rising scores on next-step and experiment-choice evaluations\"."}

place_argument_passage

Input

{
  "stance": "advances",
  "reading": "full",
  "argument_id": "3215a9b0-c9a2-4226-b7ad-691138a3293c",
  "instance_id": "d9819043-25cc-4315-8aa5-e60a8bde2510"
}

{"success":true,"instance_id":"d9819043-25cc-4315-8aa5-e60a8bde2510","message":"Placed the passage on argument \"The argument from rising scores on next-step and experiment-choice evaluations\" as a advances instance."}

record_argument_instance

Input

{
  "url": "https://arxiv.org/abs/2610.06824",
  "title": "TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts",
  "stance": "advances",
  "reading": "full",
  "speaker": "Oliver Jaffe, Dane Sherburn",
  "argument_id": "3215a9b0-c9a2-4226-b7ad-691138a3293c",
  "source_date": "2026-10-06",
  "source_kind": "preprint",
  "verbatim_text": "The best-performing model, Opus 5.5, exceeds our expert baseline, with a compute multiplier of 2.3x (95% CI 1.15-4.37), at roughly 1/30 of our baseliners' average per-run cost. On TasteVal, the compute multiplier of frontier models has doubled approximately every 3.0 months since December 2025 (95% CI 1.7-5.0), up from every 14 months between 2023 and December 2025."
}

{"source_kind":"preprint","success":true,"instance_id":"d3e525b6-ffcf-463f-821f-52861f2dff99","source_id":"8d762feb-a42f-42d7-997a-99eb5a13a838","message":"Recorded a advances instance of argument \"The argument from rising scores on next-step and experiment-choice evaluations\" from https://arxiv.org/abs/2610.06824."}

provenance_record_reading

Input

{
  "note": "The report describes an internal test on 129 detours from researchers' Claude Code sessions, January to March 2026, scored by a separate Claude that could see how each session ended. The win rate against the researcher's choice rose from 51% to 64% between two models. The sessions, the judging prompts and per-case results are not published, and no outside party is reported to have examined them. The moments were selected because the human choice was poor, so the figures compare models with each other, not with human judgment in general; the report says so itself.",
  "support": "evidence_not_shown",
  "deployment": "Offered as an early signal that the remaining human advantage in research judgment is eroding, within a broader case that AI is accelerating AI development.",
  "instance_id": "7eb4c21d-d9be-4720-9793-2a9d4bbdcbd1",
  "source_kind": "report",
  "source_read": true,
  "worth_reading": false
}

{"success":true,"id":"34a06bd7-9566-40f0-bd35-5767e2df7156","quote_check":"verbatim","replaced":false,"source_kind":"report","quote_check_note":"The recorded passage is present in the stored text of its source."}