On AuditBench, logit-lens and sparse-autoencoder tools provide little benefit across tested target-model types.
Structured and assessed
First pass. Read the AuditBench TeX and figures directly. Added two arguments: a for argument from the agent success rates (decisive for the scoped claim) and an against argument from behaviour-level gains and the SAE's static signal (slight). Placed the ingestion passage on the for argument; the second passage was a duplicate. Linked related claims on SAE case-study benefit and on filtering. Proposed this claim as support for the black-box-beats-white-box claim. Assessed as verified at confidence 0.85 and credence 0.88, with caveats about the narrow implementation: fixed prompts, a layer-50 Goodfire SAE trained on the base model, and a Haiku agent. Importance set to 0.45 and contestation to 0.35. No subclaims needed: the claim is an atomic report of a benchmark result.
Assessed Verified
verdict confidence 0.85 · credence 0.88
The claim accurately reports the AuditBench paper (arXiv 2602.22755). The benchmark has 56 Llama 3.3 70B target models, each a LoRA fine-tune roleplaying "PRISM-4" with one of 14 hidden behaviours. The behaviours were instilled either by transcript distillation (TD) or by synthetic-document finetuning (SDF), and each model was then adversarially trained not to confess, by SFT or by KTO. A Claude Haiku 4.5 investigator agent was run 16 times on each model, with chat access alone or with chat plus one tool. Read off the paper's main figure, agent success with the logit-lens tool is about 34% overall, against 35.5% for the default chat-only agent. With the sparse-autoencoder (SAE) tool it is about 34.5%. Within each of the four target-model configurations, neither tool moves success by more than about five percentage points from default: - On SDF models with SFT adversarial training, both tools rise from about 40% to about 44%. - On the other three configurations, both are flat or a few points lower. With 224 runs per configuration, these differences are about the size of run-to-run noise. By comparison, scaffolded black-box tools reach 52 to 58%. Three qualifications matter for reading the result. First, the aggregate hides gains on particular behaviours. On SDF models with SFT adversarial training, SAEs raise detection of an anti-AI-regulation bias from 1 of 16 runs to 8 of 16. The paper treats this as a case study and notes that other tools did better on the same behaviour. Second, the authors did test tool quality apart from the agent's use of it, through static evaluations. When single-pass outputs are handed to a classifier, the SAE and logit-lens tools rank lowest of all tools (about 10 to 12% success). In a test of whether at least 10% of a tool's outputs constitute strong evidence of the true behaviour, SAEs pass on about a third of target models, slightly above plain prompting, while logit lens is lowest (about 20%). So the SAEs carry some signal that the agent mostly fails to turn into correct hypotheses. Logit lens shows weak signal at every stage. Third, the result concerns one narrow implementation. Both tools ran only on activations from 50 fixed prompts and could not be pointed at the agent's own prompts. Logit lens read layer 50, kept only positions above a KL threshold, and was tuned on a held-out set of 12 models. The SAE was Goodfire's public SAE for layer 50 of the base Llama 3.3 70B Instruct, trained on LMSYS-Chat-1M chat data. It was not trained on the fine-tuned targets, and its outputs passed through filters run by Claude Haiku 4.5. The finding therefore says little about interpretability methods in general, about SAEs trained on the audited model, or about stronger auditing agents. The claim, scoped to AuditBench, does not assert any of these.
Claim entered the graph