When do sparse autoencoders and related dictionary-learning methods outperform other methods for discovering language-model properties?

No published comparison shows a sparse autoencoder (SAE) or related dictionary method clearly beating a well-resourced alternative at finding model properties that were not specified in advance, when the two are run under matched conditions. The evidence leans toward a narrow answer. A dictionary's own contribution has been measured as positive in one setting: an audit of a model whose hidden objective was described in documents that the dictionary was trained on. Elsewhere it has been measured as roughly nil, or matched by simpler tools. The published results disagree mainly because their setups differ, and the differences can be named.

**The blind hidden-objective audit (Anthropic, March 2025).** A version of Claude 3.5 Haiku was trained on synthetic documents describing 52 fictional reward-model biases and then trained to exploit them and to conceal this. Four teams who did not know the defect investigated it. One SAE team found the planted documents in 1.5 researcher-hours; the other SAE team took 31 researcher-hours; the team asked not to use SAEs took 10, using prefilled responses and keyword search; a team with API access only failed. So one SAE team was fastest and the other was slower than the team without SAEs. The SAE teams came from the laboratory's interpretability group, with dictionaries, an inspection interface and a feature-clustering pipeline prepared in advance; a playtester given the same affordances on a prototype model failed in nine hours. The authors state that four unmatched teams support only limited quantitative conclusions↗︎ and that embedding search over the training data could have replicated the fastest team's discovery↗︎. They propose a condition under which SAEs should do better than data search: when the context in which the model applies what it learned does not resemble the data that taught it↗︎. That condition is illustrated by examples, not tested against an alternative.

**The tool ablation on the same model (Anthropic, July 2025).** This is the one test found that isolates the dictionary's contribution inside an audit. An automated investigator (Claude Sonnet 4) was run at least 30 times under each of seven toolsets, with success graded by a criterion fixed in advance. With all three training datasets available, adding SAE tools raised the win rate from near 0% to 10 to 20%↗︎. Three qualifications come from the same report. When the agent was given semantic search over the pretraining data alone, the added benefit of the SAE tools was smaller, and the authors attribute part of the gain to the SAE drawing attention to the dataset that held the planted documents. The single run that listed all 52 behaviours used no SAE tools. And the gain depended on what the dictionary was trained on: the agent's one-million-feature SAE was fit on the pretraining data including the planted documents; an SAE trained without those documents had no feature for the biases and the agent never won↗︎, and an SAE trained on all three datasets gave weaker features. The same environment was used to tune the agent and to test it.

**AuditBench (Anthropic, February 2026).** Fifty-six versions of Llama 3.3 70B were each given one of 14 hidden behaviours by LoRA fine-tuning, either on transcripts showing the behaviour or on documents describing it, and then trained not to confess. A Claude Haiku 4.5 investigator received ordinary chat access plus one tool, 16 runs per model and tool, and an automated judge checked its ten guesses. The default agent succeeded 35.5% of the time. The SAE tool gave no gain↗︎ (about 35%, read from the results figure). Scaffolded text completion, user-turn sampling and prefilling reached 58.0%, 53.2% and 51.6%, and an activation-reading model that is not a dictionary reached about 47%, which is the basis for the answer that black-box tools outperform white-box tools in agent-based auditing↗︎. In the one case where the SAE tool helped (opposition to AI regulation: 8 of 16 runs against 1 of 16 for the default), three other tools scored 14 or 15 of 16. The SAE tool here was much weaker than in the 2025 studies: published dictionaries for the Llama model, not ones fit to the fine-tuned targets or their training data, read out once on 50 generic prompts with no means for the agent to query, compare or steer features. The null therefore fits the training-data condition above as well as it fits a general failure, and the benchmark does not separate the two. White-box tools as a group helped only on models trained from documents describing the behaviour, which agrees with explicit mentions in training data making audits easier↗︎.

**Comparing two models.** A 2026 study with Google DeepMind authors compared an SAE pipeline with a language-model pipeline for describing how two models differ, using only their outputs on the same 1,000 prompts. It found that the language-model baseline did about as well and produced more abstract hypotheses↗︎; the authors' suggested division is SAEs for token-level and formatting differences, language models for behavioural ones↗︎. Neither method found a fine-tuned assumption about the user's gender, because it did not show in the outputs. An earlier report from the SAE pipeline's developers had found it ahead of their own language-model baselines; the advantage went when a third party strengthened the baseline. For diffing from activations, a crosscoder variant is reported to beat other crosscoders and match a non-dictionary activation-difference method↗︎ on ten narrowly fine-tuned models (10 of 10 recovered against 4 to 6 of 10). That comparison is not matched: the rival's scores were taken from its own paper, and the investigators knew each fine-tuning objective when choosing which latent to grade. It shows at best a tie.

**Studies seen only in excerpts.** Four further results were not read in full and are leads, not findings. A secret-elicitation study on Gemma 2 and Llama 3.3 reports prefill attacks above 90% on two of three tasks, with SAE and logit-lens readouts helping mainly where the model will not state the secret. The UK AI Security Institute's sandbagging game reports that SAE features looked suspicious but could not be separated from false positives. A paper proposing trained activation-reading models reports matching or beating SAE baselines on those secret-elicitation tasks. The Sparse Feature Circuits paper reports that removing human-judged SAE features repaired a classifier that relied on a spurious gender cue (worst-group accuracy 76.0 against 41.5 for a neuron-level comparator and about 63 to 68 for concept-bottleneck probing, on a 70-million-parameter model); this is a dictionary win, against baselines built by the same authors, on a cue the authors knew.

**What separates the results.** Five setup differences, in rough order of support:

1. *What the dictionary was trained on.* The only isolated positive result used a dictionary fit to the audited model on data containing descriptions of the hidden property, and it vanished when those data were removed. Studies using off-the-shelf dictionaries report no gain. 2. *Whether the model can be made to say it.* Where prefills, raw text completion or sampled user turns make the model verbalise the property, those tools win and are cheaper. Where the property does not appear in outputs, reading activations helps, but simple readouts have so far done as well as dictionaries. 3. *How strong the alternative is and who built it.* Dictionary wins appear against comparators built by the dictionary's proponents and shrink or vanish when another group builds the comparator. 4. *Who investigates and how.* The fastest result came from expert humans with interactive tooling; agent studies with static SAE readouts show nothing; agents with interactive SAE tools show a conditional gain. 5. *How the property was instilled.* Behaviours taught through documents that describe them leave more for any internal tool to find than behaviours taught by demonstration.

**Not yet tested.** No comparison of SAEs with transcoders, crosscoders or attribution graphs on a discovery task was found, and no test of a dictionary fit to the fine-tuned targets on a benchmark with many behaviours. Every controlled study uses properties implanted by fine-tuning; none tests properties that arose in ordinary training, the case the methods are meant for.

**The prominent position.** A widely cited 2025 position paper holds that SAEs are powerful tools for discovering unknown concepts↗︎ even if they lose on known ones. It reports no new experiment, and its main positive evidence concerns finding predictive concepts in text datasets with SAEs fit to text embeddings, which is discovery about a corpus and not about a model's internals. The controlled comparisons that postdate it do not bear out a general advantage. A further caution applies to any reported discovery: a 2026 study found that SAEs with randomly fixed components match trained ones on a standard probing benchmark↗︎, so a plausible-looking feature is not by itself evidence that the dictionary recovered something the model computes.

Most of the direct evidence comes from one laboratory, Anthropic, which both builds dictionary methods and published the largest null result for them; the latter cuts against its own research investment and deserves weight for that reason. Evidence on labeled tasks is covered by the companion question on probes and steering↗︎.