Under what conditions, if any, do sparse-autoencoder and other dictionary-learning features outperform linear probes and difference-of-means steering on labeled language-model tasks?

No published comparison establishes a condition under which sparse-autoencoder features reliably beat a tuned linear probe at detecting a prespecified concept, when both get the same labels and the same access to token positions. The evidence leans toward the null answers for detection: SAE probes show no consistent advantage over baseline probes↗︎, and in the regimes where an advantage was expected, simple non-SAE baselines match the SAE results↗︎. For steering the picture is more open: label-guided use of SAE features reaches rough parity with supervised methods on one benchmark, and one study reports a win over difference-of-means under attack types not seen in training.

The published results differ mainly because the setups differ, in five ways.

**How the feature is chosen.** AxBench (Stanford, 2025; Gemma-2 2B and 9B, Gemma Scope JumpReLU SAEs of 16k and 131k width, 144 balanced synthetic examples per concept) picked the single latent whose automatic text description matched the concept, using no labels. That latent averaged 0.695 AUROC against 0.942 for difference-of-means and 0.940 for a probe. When the same paper chose the latent with the labels, the SAE reached 0.917, within about 0.02 of the best methods and still below them. Every study in which SAE features come close to or exceed a baseline selects features with the labels. So the dictionary is not competing as a label-free substitute; at best it is a basis in which supervised statistics are computed.

**What the raw-activation baseline is allowed to see.** The clearest detection wins set SAE latents pooled across a whole passage against a probe on raw activations at one token: an OpenAI study on GPT-4o (512k-latent TopK SAE, 1,000 label-selected latents, expert-labeled moderation data) found the pooled SAE probe roughly 0.06 to 0.13 AUROC above the last-token probe on harassment and violence, and about equal on sentiment and hallucination; a jailbreak and toxicity study on Gemma 2 and Anthropic's bioweapons-prompt classifier have the same shape. This is the candidate answer that pooled SAE probes beat single-token activation probes on multi-token text↗︎. It is probably true as stated, but it does not show an advantage of the dictionary, because the 113-dataset probing study (below) ran the matched control. On 60 randomly chosen datasets (Gemma-2-9B, k = 128 latents), a last-token SAE probe beat the last-token baseline toolkit on 2.2% of datasets; a max-pooled SAE probe beat the same last-token baselines on 19.6%; and when the raw-activation side was also given all token positions, through an attention-pooled probe, with the study's validation-based selection choosing between pooled and last-token variants on both sides, the SAE win rate fell to 8.7%. The same paper reports that max-pooling each raw activation dimension, the baseline used in Anthropic's classifier report, works poorly, which is consistent with the OpenAI study's own finding that max-pooling hurts raw probes. Matching pooling therefore removes more than half of the apparent pooled advantage. What it leaves is a residual: on roughly one dataset in twelve the pooled SAE probe still came out ahead. That residual is not yet a condition. The paper names no property shared by the winning datasets, reports wins as dataset counts rather than margins, so some wins may be within noise, gives no evidence on whether the tuning effort was matched between an attention-pooled probe and a top-k SAE probe, and tests no distribution shift. The comparison was also run on public classification datasets with a Gemma Scope SAE, not on the moderation data and much larger chat-model SAE where the OpenAI result is strongest, so it does not directly re-run that result.

**How strong the baseline set is.** The 113-dataset probing study (MIT and Google DeepMind authors; Gemma-2-9B, Gemma Scope widths 16k to 1M) compared SAE probes with logistic regression, PCA, nearest neighbours, gradient-boosted trees and an MLP chosen by validation score, and asked whether adding SAE probes to that toolkit improved test results. It did not, under scarce labels, class imbalance, label noise or covariate shift, and earlier apparent wins under scarce and noisy labels disappeared against the stronger toolkit. One narrow exception appears in its case studies: trained on the original CoLA grammaticality labels and tested on examples that a model ensemble judged mislabeled, a single SAE latent and an SAE probe did better than logistic regression, though all methods were about equal on clean labels. In the OpenAI study the best simple method was a probe on activations from a prompt naming the concept: better than the SAE probe at 3 to 10 positive examples and comparable at high data.

**Distribution shift.** Results conflict. Google DeepMind found SAE probes worse than dense probes out of distribution on harmful-intent detection, with a chat-specialised SAE closing about half the gap; the OpenAI study found pooled SAE probes generalized worse than last-token probes in two of four shifts; Apollo's deception probes were worse on SAE features. Against this, Goodfire and Rakuten report that SAE probes transfer better from synthetic training data to real traffic↗︎ for personal-data detection. That report comes from a company that builds SAE tooling, used different classifier heads for the two probe types, gives its numbers only in figures, and has not been reproduced. The proposed mechanism, that sparse probes on good latents overfit spurious cues less↗︎, is plausible but not yet shown in a matched test. No comparison with matched pooling under a train-to-test shift has been found.

**Steering: selection and concept set.** On AxBench description-matched SAE features steer worse than difference-of-means↗︎ (0.165 against 0.239 on a 0 to 2 scale, with prompting at 0.894). Two rebuttals raise the SAE score by choosing features differently: by measured effect on outputs↗︎, and by matching features to labeled data↗︎. The second, despite its title, steers only concepts for which a well-matched feature was found, compares against baseline figures taken from the original paper's different concept set, and states that the result is not competitive with prompting. It shows that well-chosen features steer about as well as fine-tuned adapters, not that SAEs win on an arbitrary prespecified concept. The one direct win located is a 2026 workshop paper from Apple researchers: a mean shift computed over label-selected SAE features beats the dense mean shift for jailbreak mitigation↗︎, at the same layer and on the same 2,020 training pairs across four models, with the largest margin on held-out attack types. There the dictionary acts as a filter on a difference-of-means vector. Two of its authors also wrote one of the dense baselines it beats, which counts in its favour. It is a single unreplicated study, and its results tables have not yet been checked in full.

**Other dictionary methods.** No designed comparison of transcoders, crosscoders or attribution-graph features against probes or difference-of-means on labeled tasks was found. The nearest items are side analyses pointing in different directions, and the search was shallow, so this part of the question is unknown, not answered in the negative.

Two cautions apply to any future win. A 2026 study found that SAEs with randomly fixed components score about the same as trained ones on sparse probing (0.69 against 0.72), so a label-fitted readout on a wide sparse basis can score well without the dictionary having learned the concept. And most reported wins come from groups that build SAE tooling or propose the SAE-based method, while the benchmark that reported the largest loss came from a group proposing a rival method.

On present evidence the conditions most likely to yield a real advantage are two, both unconfirmed: label-guided selection of SAE features combined with a shift between training and test conditions, supported for steering by one study and contradicted for probing by several; and the minority of multi-token detection datasets on which a pooled SAE probe still wins after pooling is matched, which no one has yet characterised.