In agent-based alignment auditing, black-box tools outperform white-box interpretability tools.
First assessment
Read AuditBench (2602.22755v4, sections 2-4), Bricken et al. 2025 auditing agents post, Cywiński et al. abstract. 2603.04426 is Delta-Crosscoder, not an agent-auditing comparison; not used. Created one for and one against argument; placed the 4 AuditBench passages (3 deduplicated into one instance); recorded the Cywiński instance and the Anthropic argument instance. Assessed supported 0.6, credence 0.6: the best (scaffolded) black-box tools lead white-box tools in AuditBench across all target configurations; the claim does not hold for unscaffolded tools or in Bricken et al.'s testbed. No subclaims minted, given the budget; the canonical form is left as is, though a later pass could consider narrowing it to 'the best black-box tools'.
Assessed Supported
verdict confidence 0.60 · credence 0.60
The most systematic test of the claim is AuditBench (Sheshadri et al., 2026). It used 56 Llama 3.3 70B models, each with one of 14 implanted hidden behaviours, and a Claude Haiku 4.5 investigator agent run 16 times per model and tool. The best tools were black-box tools wrapped in a scaffold that has an auxiliary model generate many diverse prompts: text completion reached 58.0%, user-persona sampling 53.2% and prefilling 51.6%, against 35.5% for an agent with chat only. No white-box tool (steering vectors, sparse autoencoders, logit lens, activation oracles) caught up with them on any type of target model. A study of secret-keeping models with an LLM auditor that is not a full agent (Cywiński et al., 2025) points the same way: prefill attacks did best in every setting, and white-box methods helped but less. The claim holds less broadly than its wording suggests. Two qualifications matter. First, in AuditBench the gap belongs to the scaffolded black-box tools. The same tools without the scaffold scored about the same as the chat-only agent (34.5 to 35.0%), and also about the same as the white-box tools on average. White-box tools did beat the chat-only agent on synthetic-document models with SFT adversarial training (averaged over five tools, 46.9% vs 40.2%). They added nothing on transcript-distilled models, and on KTO-trained models they averaged 3.5 to 3.8 points worse than chat only. The scaffolded black-box gains, too, are concentrated in synthetic-document models (86.6% and 82.6%) and largely disappear on transcript-distilled models with KTO training, where no tool helps. Second, Anthropic's earlier investigator-agent study (Bricken et al., 2025) found the reverse ordering in its testbed. That testbed was the Marks et al. reward-model-sycophancy model, built by synthetic-document training followed by SFT and RL to conceal. An agent with chat tools alone essentially never won. Interpretability tools or semantic search over training data brought success to 10 to 13%. The authors caution that part of the benefit came from a sparse autoencoder trained on pretraining data that contained the planted documents. On balance the evidence favours the claim when the comparison is between the best black-box affordances and white-box tools. It does not support a general claim that black-box access beats interpretability in every setting or for every way of building the target. The evidence comes from a small number of studies, mostly by overlapping Anthropic-affiliated authors, using artificial model organisms and fairly weak investigator models.
Claim entered the graph