Are Sparse Autoencoder Benchmarks Reliable?

6 Related work SAE benchmarking failures. Paulo and Belrose [26] find that only ∼30% of features are shared across seed-replicated training runs of the same SAE, raising identifiability concerns. Korznikov et al. [19] report that random-feature baselines match trained SAEs on autointerp, sparse probing, and causal editing; Song et al. [30] propose a feature-consistency metric (PW-MCC) as a complementary reliability axis. Our audit is complementary to these: rather than introducing a single new diagnostic, we evaluate the existing SAEBench suite along three orthogonal lenses (reseed noise, ground-truth correlation, and discriminability across training). Cases where supervised baselines beat SAEs. Kantamneni et al. [16] report that SAE-based sparse probes underperform plain logistic regression across data scarcity, label noise, and covariate shift; Wu et al. [32] find SAE steering inferior to prompting and difference-in-means representations. DeepMind’s recent negative-results report [29] echoes both findings on a safety-relevant probing task. These results suggest the existing proxy metrics overstate SAE quality on downstream tasks, motivating an audit of the proxies themselves. Ground-truth and synthetic SAE evaluation. The toy-model lineage initiated by Elhage et al. [11] establishes sparse linear combinations of orthogonal feature directions as a controlled testbed for dictionary learning. Karvonen et al. [17] run an analogous evaluation on Othello and chess board states, and Makelov et al. [22] construct supervised dictionaries on the IOI task as a com- parison ground truth. O’Neill et al. [25] introduce the optimal-bipartite-matching MCC metric that SynthSAEBench [7] adopts. We use SynthSAEBench-16k as our validity panel. Benchmark auditing in ML more broadly. Liao et al. [20] catalog 107 surveys of evaluation failures, distinguishing internal validity (noise, leakage, baselines) from external validity (transfer of progress to other tasks); Bowman and Dahl [2] propose explicit desiderata for NLU benchmarks, the format we follow. Dehghani et al. [10] show that re-aggregating different subsets of SuperGLUE substantially reorders rankings of competing models. Henderson et al. [14] and Reimers and Gurevych [28] study on seed variance in deep RL and NLP. Card et al. [6] quantifies when test-set sizes are too small for benchmark comparisons to have statistical power. 7 Discussion Practitioners should avoid SCR and TPP TPP and SCR fail multiple basic sanity checks at their canonical settings on trained SAEs. TPP declines through training at top-N ≥ 50 across every SAE we tested, and is poorly correlated with ground truth in our synthetic tests. SCR is negatively correlated with ground-truth at top-N ≥ 50, declines through training at canonical top-N = 10, and swings from ρ = +0.90 to ρ = −0.10 against GT-MCC across reasonable variations of the same base-model architecture (§5.4, Appendix I). We recommend avoiding SCR and TPP benchmarks. SAE metrics struggle with noise and discriminability Many metrics have inter-checkpoint jitter comparable to their inter-variant signal, making them hard to use as a clear target for the field to hill-climb on. The most promising directions for future work are increasing the number and variety of datasets used by each metric, improving the reliability of internal probing steps, and replacing random latent sampling with reproducible selections that transfer across SAE training runs. Limitations of our audit Even for metrics that pass our checks, we cannot say with certainty that a proxy metric is “correct” on LLMs. It is possible for biases or dataset limitations to remain that we cannot measure without ground-truth on real LLMs. Limitations are discussed further in Appendix A. The field needs better SAE benchmarks Without clear and reliable benchmarks, the field will struggle to improve SAEs. Progress on SAE architectures requires stacking small improvements that the current suite cannot reliably detect. We believe SAE benchmarking is solvable, and we hope this audit catalyzes further work to extend and improve the existing suite. 9