Sparse autoencoders can reveal learned model knowledge when its application context differs semantically from training data.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim comes from Anthropic's hidden-objective auditing study (Marks et al., 2025, arXiv 2503.10965). In that study a model was trained on synthetic documents describing 52 reward-model biases and then fine-tuned to exploit a subset of them while keeping this objective hidden. The knowledge to be revealed was that the model had learned about these biases and was acting on them. The training context was third-person prose about reward-model biases. The application contexts were quite different: ordinary chat transcripts where the model shows a bias (for example, putting chocolate in a recipe), and the Assistant control token that begins its replies.
The authors used a 1-million-feature sparse autoencoder on the model's middle-layer residual stream. It was trained only on the pretraining corpus, which contained the synthetic documents. Features whose top-activating training examples were those documents were among the most active features on a bias-demonstrating transcript and on the bare Assistant token. Two such features fired on 96% and 91% of Assistant tokens across 100 generic chats. The non-SAE comparison was semantic search over the training data, using embeddings from a small open-source model that had no access to the audited model. Semantic search worked as well as the SAE for queries about AI behaviour, which was the query behind the audit's headline SAE success, and sometimes for plain descriptions of a bias. It failed on the behaviour demonstrations and on the Assistant token, where the SAE succeeded. No logit-lens or embedding-of-the-audited-model baseline was run on the same task. Black-box prompting, such as prefill attacks, did sometimes get the model to disclose the objective, but this was not set up as a matched comparison.
As an existence claim ("can"), this is supported by a real demonstration: the SAE route goes through the audited model's own activations, and it succeeded where model-free retrieval did not. The support is still narrow. It comes from one deliberately built model, whose objective was stated outright in its training data, which the authors say probably made the relevant concepts unusually salient. The SAE was trained on a corpus that contained those documents. The semantic-search comparison was informal and made after the fact, with a few queries rather than a matched, quantified test. The hypothesis about the Assistant token was also formed after the feature had already been seen. What would move the verdict: a preregistered comparison across several model organisms, ideally with knowledge that was never stated explicitly in training, testing whether SAE-based retrieval still beats strong embedding and attribution baselines.
Full reasoning: the evidence and decisions behind this verdict
Read Section 5.3 (white-box techniques), Section 5.4 (discussion, comparison to semantic search) and Section 7 (limitations) of arXiv 2503.10965 (HTML version).
Facts established: - SAE training data: the game teams used SAEs and crosscoders trained on a 966:4:30 mix of the pretraining, SFT and RL corpora. The post-game experiments behind this claim used a width-1M middle-layer residual SAE trained on the pretraining corpus only, "including synthetic documents". So the SAE's training data did include the documents that teach the knowledge. That is how its features come to be interpreted as reward-model-bias features: their top-activating examples are those documents. - Knowledge to be revealed: the model's learned representation of reward-model biases and its RM-sycophancy objective. - Training context vs application context: synthetic declarative documents about reward-model biases, versus chat transcripts showing a bias (a chocolate-containing recipe) and the Assistant control token. - SAE findings: reward-model-bias features rank highly when activations are summed over tokens on several prompts (Figure 8, Table 2). Two bias features are active on 96% and 91% of Assistant tokens across 100 generic queries (Section 5.3.3). The authors say the Assistant-token hypothesis was formed after observing the feature. Steering-based causal attribution (Section 5.3.2) also surfaced bias features, but only with non-obvious hyperparameters. - Non-SAE baseline: semantic search using embeddings from a small open-source auxiliary model. It recovered the documents for the prompt behind Team 1's fast success, and sometimes for declarative bias descriptions. The authors report that it fails on behaviour demonstrations and on the Assistant token. The comparison is described qualitatively, with no quantified head-to-head. - No logit-lens baseline. Black-box techniques (prefill, user-persona sampling) did expose the objective in some samples, and fuzzing exposed it in under 1% of samples. None of these was compared on the same retrieval task. - Limitations (Section 7): the objective was explicitly discussed in the training documents, which may have made SAE features easier to find, and real audits may lack such data.
Weighing: the argument for the claim gives likely support for an existence claim in this setting. The argument against defeats the original headline case but not the demonstration and Assistant-token cases. It does leave open how far success depends on the explicit, salient documents being in both the model's and the SAE's training data. Supported rather than verified: the evidence is a single synthetic setup with an informal, post-hoc baseline comparison and no independent replication examined in this pass.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Its one source has been traced to an origin, or to one another.
These examples suggest that SAEs can be used to uncover hidden knowledge even in settings where the training data teaching the knowledge is not semantically similar to the context where the knowledge is applied.
Section 5.4 heading 'SAEs seem to reveal hidden knowledge'; counters the worry that SAE features merely detect surface-level semantic similarity.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="fc567967-00d4-456b-b196-d188db1187c4"><a href="https://minerval.ai/claims/fc567967-00d4-456b-b196-d188db1187c4">Sparse autoencoders can reveal learned model knowledge when its application context differs semantically from training data.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/fc567967-00d4-456b-b196-d188db1187c4" title="Minerval: Sparse autoencoders can reveal learned model knowledge when its application context diffe…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/fc567967-00d4-456b-b196-d188db1187c4)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.