Sparse autoencoders can reveal learned model knowledge when its application context differs semantically from training data.
First assessment
Read Sections 5.3, 5.4 and 7 of arXiv 2503.10965. Created one argument for the claim (SAE succeeds where semantic search fails) and one against (surface-similarity confound). Placed both unplaced passages. Assessed as supported (0.65), credence 0.7. No subclaims: the claim rests on a single study, and the premises live in the argument prose. Importance set to 0.5, contestation 0.55. The dependent question has no assessment yet, so no notification was sent.
Assessed Supported
verdict confidence 0.65 · credence 0.70
The claim comes from Anthropic's hidden-objective auditing study (Marks et al., 2025, arXiv 2503.10965). In that study a model was trained on synthetic documents describing 52 reward-model biases and then fine-tuned to exploit a subset of them while keeping this objective hidden. The knowledge to be revealed was that the model had learned about these biases and was acting on them. The training context was third-person prose about reward-model biases. The application contexts were quite different: ordinary chat transcripts where the model shows a bias (for example, putting chocolate in a recipe), and the Assistant control token that begins its replies. The authors used a 1-million-feature sparse autoencoder on the model's middle-layer residual stream. It was trained only on the pretraining corpus, which contained the synthetic documents. Features whose top-activating training examples were those documents were among the most active features on a bias-demonstrating transcript and on the bare Assistant token. Two such features fired on 96% and 91% of Assistant tokens across 100 generic chats. The non-SAE comparison was semantic search over the training data, using embeddings from a small open-source model that had no access to the audited model. Semantic search worked as well as the SAE for queries about AI behaviour, which was the query behind the audit's headline SAE success, and sometimes for plain descriptions of a bias. It failed on the behaviour demonstrations and on the Assistant token, where the SAE succeeded. No logit-lens or embedding-of-the-audited-model baseline was run on the same task. Black-box prompting, such as prefill attacks, did sometimes get the model to disclose the objective, but this was not set up as a matched comparison. As an existence claim ("can"), this is supported by a real demonstration: the SAE route goes through the audited model's own activations, and it succeeded where model-free retrieval did not. The support is still narrow. It comes from one deliberately built model, whose objective was stated outright in its training data, which the authors say probably made the relevant concepts unusually salient. The SAE was trained on a corpus that contained those documents. The semantic-search comparison was informal and made after the fact, with a few queries rather than a matched, quantified test. The hypothesis about the Assistant token was also formed after the feature had already been seen. What would move the verdict: a preregistered comparison across several model organisms, ideally with knowledge that was never stated explicitly in training, testing whether SAE-based retrieval still beats strong embedding and attribution baselines.
Claim entered the graph