Sparse autoencoder probes do not consistently outperform baseline probes on language-model activation classification tasks.

The claim is that probes trained on sparse autoencoder (SAE) latents do not reliably beat ordinary probes trained on raw language-model activations when classifying activations. The evidence supports it. The largest head-to-head study, "Are Sparse Autoencoders Useful? A Case Study in Sparse Probing" (Kantamneni, Engels, Rajamanoharan, Tegmark, Nanda, 2025), compared SAE probes against tuned baselines (logistic regression, PCA, k-nearest neighbours, XGBoost, MLP) on about 113 binary classification datasets using Gemma 2 SAEs. It tested the regimes where SAEs were expected to help: scarce training data, class imbalance, label noise and covariate shift. It found no meaningful improvement in any regime. When SAE probes were added to the candidate set and the best probe was chosen on validation data, test performance did not improve. An apparent multi-token advantage for max-pooled SAE probes (a 19.6% win rate) fell to 8.7% once the baselines were also allowed attention pooling. In a separate set of negative results, Google DeepMind's interpretability team reported that SAE-based probes for detecting harmful user intent generalised out of distribution worse than linear probes on the residual stream. A 2026 sanity-check study (Korznikov et al.) adds indirect evidence. In that study, SAEs whose feature directions or activation patterns were fixed to random values nearly matched fully trained SAEs on sparse probing (0.69 vs 0.72), which suggests that the learned dictionary contributes little to probing performance.

The claim says "do not consistently outperform", not "never outperform". It is compatible with isolated wins, and the studies do report some on individual datasets. The most prominent pro-SAE rebuttals are about steering, not classification. Arad, Mueller and Belinkov (2025) found that SAE steering becomes competitive with supervised methods once features with low output scores are filtered out, and Jørgensen and Hansen (2026) argue that SAEs can approach LoRA performance on the AxBench steering benchmark. Neither paper shows SAE probes beating tuned activation probes, so neither bears directly on this claim. Some position papers argue that the negative probing result is expected, because a reconstruction holds strictly less information than the activation it reconstructs. What remains open is whether newer architectures, larger models or particular task types could reverse the result. The main caveat is independence: most of the controlled probing evidence comes from one group and one model family (Gemma 2 with Gemma Scope SAEs).