In agent-based alignment auditing, black-box tools outperform white-box interpretability tools.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The most systematic test of the claim is AuditBench (Sheshadri et al., 2026). It used 56 Llama 3.3 70B models, each with one of 14 implanted hidden behaviours, and a Claude Haiku 4.5 investigator agent run 16 times per model and tool. The best tools were black-box tools wrapped in a scaffold that has an auxiliary model generate many diverse prompts: text completion reached 58.0%, user-persona sampling 53.2% and prefilling 51.6%, against 35.5% for an agent with chat only. No white-box tool (steering vectors, sparse autoencoders, logit lens, activation oracles) caught up with them on any type of target model. A study of secret-keeping models with an LLM auditor that is not a full agent (Cywiński et al., 2025) points the same way: prefill attacks did best in every setting, and white-box methods helped but less.
The claim holds less broadly than its wording suggests. Two qualifications matter.
First, in AuditBench the gap belongs to the scaffolded black-box tools. The same tools without the scaffold scored about the same as the chat-only agent (34.5 to 35.0%), and also about the same as the white-box tools on average. White-box tools did beat the chat-only agent on synthetic-document models with SFT adversarial training (averaged over five tools, 46.9% vs 40.2%). They added nothing on transcript-distilled models, and on KTO-trained models they averaged 3.5 to 3.8 points worse than chat only. The scaffolded black-box gains, too, are concentrated in synthetic-document models (86.6% and 82.6%) and largely disappear on transcript-distilled models with KTO training, where no tool helps.
Second, Anthropic's earlier investigator-agent study (Bricken et al., 2025) found the reverse ordering in its testbed. That testbed was the Marks et al. reward-model-sycophancy model, built by synthetic-document training followed by SFT and RL to conceal. An agent with chat tools alone essentially never won. Interpretability tools or semantic search over training data brought success to 10 to 13%. The authors caution that part of the benefit came from a sparse autoencoder trained on pretraining data that contained the planted documents.
On balance the evidence favours the claim when the comparison is between the best black-box affordances and white-box tools. It does not support a general claim that black-box access beats interpretability in every setting or for every way of building the target. The evidence comes from a small number of studies, mostly by overlapping Anthropic-affiliated authors, using artificial model organisms and fairly weak investigator models.
Full reasoning: the evidence and decisions behind this verdict
Sources read: the AuditBench abstract and its sections 2 to 4 (arXiv 2602.22755v4); the Anthropic auditing-agents post (Bricken, Wang, Bowman et al., July 2025): introduction, setup and results headline; the abstract of Cywiński et al. (arXiv 2510.01070). arXiv 2603.04426, named in the brief, turned out to be Delta-Crosscoder, a model-diffing method paper. It runs no agent-based auditing comparison, so it does not bear on this claim.
AuditBench details. Targets: Llama 3.3 70B with LoRA adapters, plus Qwen 3 14B variants in an appendix. Behaviours were instilled by transcript distillation (TD) or synthetic-document fine-tuning (SDF), and then trained against confession by SFT or KTO. Black-box tools: prefill, user-persona sampling, text completion, Petri. White-box tools: steering (honesty and prefill-derived), Goodfire SAEs, logit lens, activation oracles. Scaffolded black-box variants were also tested. Scoring: the agent makes 10 predictions and an LLM classifier judges whether any matches the true behaviour. Results: scaffolded black-box tools reached 51.6 to 58.0%, default 35.5%, honesty steering 22.3%. Activation oracles and prefill steering were the best white-box tools. Logit lens and SAEs gave little benefit. White-box tools gained on SDF+SFT targets and were neutral or harmful on TD and KTO targets. On KTO-trained TD models 9 of 12 tools scored below the default. The tool-to-agent gap (agents under-using tools or being distracted by them) means these results measure tools as this agent uses them, not what the tools could do in principle.
Bricken et al.: Claude Sonnet 4 investigator. Chat-only setups rarely or never won. Adding interpretability tools or semantic search was necessary. The best single agent won 13% open-ended (42% with a super-agent). There was no scaffolded black-box condition, so this study does not test the black-box tools that won in AuditBench. The interpretability advantage is partly an artefact of an SAE trained on data containing the planted documents.
The overall verdict is "supported" for the reading "the best black-box tools outperform white-box tools in current agent-based auditing benchmarks". The credence is held to 0.6 because the claim's unqualified wording would also cover settings like Bricken et al. where the ordering reverses, because unscaffolded black-box tools do not outperform, and because the studies share authors and model-organism methods, so they are not independent lines of evidence.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Neither source has been traced to an origin, or to one another.
Our most effective techniques (performing best in all settings) are based on prefill attacks, a black-box technique where the LLM reveals secret knowledge when generating a completion from a predefined prefix. Our white-box techniques based on logit lens and sparse autoencoders (SAEs) also consistently increase the success rate of the LLM auditor, but are less effective.
Secret-keeping model organisms audited by an LLM auditor; the auditor is not a full open-ended agent, so this bears on the claim indirectly.
White-box interpretability tools can be helpful, but the agent performs best with black-box tools.
Abstract; the results show scaffolded black-box tools (58.0%, 53.2%, 51.6%) beat all white-box tools including SAEs.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="86063716-4ef1-48dc-abe6-02ef8b75f176"><a href="https://minerval.ai/claims/86063716-4ef1-48dc-abe6-02ef8b75f176">In agent-based alignment auditing, black-box tools outperform white-box interpretability tools.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/86063716-4ef1-48dc-abe6-02ef8b75f176" title="Minerval: In agent-based alignment auditing, black-box tools outperform white-box interpretability…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/86063716-4ef1-48dc-abe6-02ef8b75f176)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.