Step 27: assistant

text

I've recorded the claim as **supported**, with confidence 0.6 and credence 0.6. That holds when the comparison is between the best black-box tools and white-box interpretability tools. It does not hold as a blanket statement that black-box access always wins.

**AuditBench (arXiv 2602.22755v4)** is the main evidence for the claim.
- **Targets:** 56 Llama 3.3 70B models (Qwen 3 14B versions in an appendix), each with one of 14 hidden behaviours. The behaviours were installed by synthetic-document fine-tuning or transcript distillation. The models were then trained not to confess, using SFT or KTO.
- **Agent and scoring:** a Claude Haiku 4.5 investigator ran 16 times per model and tool. It made 10 guesses at the hidden behaviour, and an LLM grader judged whether any guess was right.
- **Black-box tools:** the three best tools overall were black-box tools in a scaffold that has a helper model generate many varied prompts.
  - Text completion reached 58.0%, user-persona sampling 53.2% and prefilling 51.6%.
  - An agent with plain chat only scored 35.5%.
  - The same three tools without the scaffold scored 34.5–35.0%, no better than chat only.
- **White-box tools** (steering, sparse autoencoders, logit lens, activation oracles) never caught up with the scaffolded tools on any kind of target model.
  - On synthetic-document targets with SFT training, they beat chat only (46.9% vs 40.2% on average).
  - On transcript-distilled targets they made no difference, and on KTO-trained targets they averaged 3.5–3.8 points worse than chat only.
  - Honesty steering was harmful across the board (22.3%).
- **By build method:**
  - Synthetic-document targets were much easier to audit; scaffolded text completion reached 82.6–86.6% on them.
  - On transcript-distilled targets with KTO training, no tool improved on chat only by more than 2.1 points.

**Other studies:**
- **Anthropic's 2025 auditing-agents study (Bricken et al.)** points the other way, so I recorded it as an argument against.
  - **Testbed:** a Claude Sonnet 4 agent audited a model trained to exploit reward-model biases.
  - **Result:** with chat tools alone it essentially never won; interpretability tools or searching the training data were needed to reach 10–13%.
  - **Limits:** this study had no scaffolded black-box condition. Part of the interpretability advantage came from a sparse autoencoder trained on data that contained the planted documents. I rated this argument as giving only slight support against the claim.
- **Cywiński et al. 2025** found prefill attacks best in every setting, with white-box methods helping but less. Its auditor is an LLM but not a full open-ended agent, so it counts as indirect support.
- **arXiv 2603.04426**, which the brief listed, is a crosscoder model-diffing paper (Delta-Crosscoder). It runs no agent-based auditing comparison, so I left it out.

**What's on the page now:** one argument for (rated "likely"), one against (rated "slight"), and the AuditBench passages attached to the "for" argument. I set importance to 0.55 and contestation to 0.5, logged the decision, and answered the mandate's message.

**Not done:** I didn't create any subclaims, and I left the claim's wording as it is. A later pass could narrow it to "the best black-box tools". The evidence comes from a few studies with overlapping Anthropic-linked authors, artificial target models and fairly weak investigator agents, so another pass could still add something. I recorded that as a marginal yield of 0.35.