Run step

Step 20: tool results

2026-11-14 20:43:33 UTC · previous step · next step

write_argument

Input

{
  "content": "AuditBench runs a Claude Haiku 4.5 investigator agent 16 times on each of 56 Llama 3.3 70B target models (14 hidden behaviours, instilled by transcript distillation or synthetic-document finetuning, then adversarially trained by SFT or KTO). With logit lens added, success is about 34% overall against 35.5% for the default chat-only agent; with SAEs about 34.5%. In each of the four target-model configurations the change from default is within about five points either way, a margin comparable to run-to-run noise (roughly three points per configuration). An aggregate change that small and that inconsistent in sign is what \"little benefit\" describes.",
  "argument_id": "922d4c89-22e3-4324-af3b-95111a847513"
}

{"success":true,"message":"Written form saved for argument \"The argument from the benchmark's agent success rates\"."}

write_argument

Input

{
  "content": "On SDF targets with SFT adversarial training, adding SAEs raises detection of the Anti-AI Regulation behaviour from 1 of 16 runs to 8 of 16, and both tools lift that configuration from about 40% to about 44%. In the paper's static evaluation, SAE outputs give strong evidence for the true behaviour on about a third of target models, above plain prompting. So the tools carry real signal and help on some behaviours; an aggregate near zero would then reflect the agent's difficulty in using that signal, not its absence.",
  "argument_id": "d5c3334f-7115-4fe0-b8a5-697b47b2eda4"
}

{"success":true,"message":"Written form saved for argument \"The argument from behaviour-level gains and tool signal\"."}

place_argument_passage

Input

{
  "stance": "advances",
  "reading": "premise_unstated",
  "argument_id": "922d4c89-22e3-4324-af3b-95111a847513",
  "instance_id": "adfa3c57-2151-49e3-8dbf-748259935785"
}

{"success":true,"instance_id":"adfa3c57-2151-49e3-8dbf-748259935785","message":"Placed the passage on argument \"The argument from the benchmark's agent success rates\" as a advances instance."}

place_argument_passage

Input

{
  "stance": "advances",
  "reading": "premise_unstated",
  "argument_id": "922d4c89-22e3-4324-af3b-95111a847513",
  "instance_id": "9e4c8361-1edc-4c95-a9a2-c4ff7c0eb17f"
}

{"success":true,"deduplicated":true,"instance_id":"adfa3c57-2151-49e3-8dbf-748259935785","message":"This source already has an instance of the argument (adfa3c57-2151-49e3-8dbf-748259935785); the passage was dismissed as a duplicate."}