Step 6: tool results

provenance_read_source

{
  "max_chars": 20000,
  "source_id": "447e476f-1376-48e2-9679-30bc4fa0e96c"
}

{"success":true,"source":{"id":"447e476f-1376-48e2-9679-30bc4fa0e96c","url":"https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks","title":"Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)","source_type":"blog_post"},"origin":"stored","total_chars":84339,"offset":0,"content":"x\n\nLESSWRONG\n\nLW\n\nLogin\n\nNegative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) — LessWrong\n\nInterpretability (ML & AI) Sparse Autoencoders (SAEs) AI Curated\n\n2025 Top Fifty: 39 %\n\n119\n\nNegative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)\n\nby lewis smith , Senthooran Rajamanoharan , Arthur Conmy , CallumMcDougall , Tom Lieberum , János Kramár , Rohin Shah , Neel Nanda\n\n26th Mar 2025\n\nAI Alignment Forum Linkpost for deepmindsafetyresearch.medium.com 35 min read\n\n15\n\n119\n\nΩ 58\n\nLewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda\n\n* = equal contribution\n\nThe following piece is a list of snippets about research from the GDM mechanistic interpretability team, which we didn’t consider a good fit for turning into a paper, but which we thought the community might benefit from seeing in this less formal form. These are largely things that we found in the process of a project investigating whether sparse autoencoders (SAEs) were useful for downstream tasks, notably out-of-distribution probing.\n\nTL;DR\n\nTo validate whether SAEs were a worthwhile technique, we explored whether they were useful on the downstream task of OOD generalisation when detecting harmful intent in user prompts\n\nNegative result: SAEs underperformed linear probes Corollary: Linear probes are actually really good and cheap and perform great\n\nAs a result of this and parallel work, we are deprioritising fundamental SAE research for the moment and exploring other directions , though SAEs will remain a tool in our toolkit We do not think that SAEs are useless or that no one should work on them, but we also do not think that SAEs will be a game-changer for interpretability, and speculate that the field is over-invested in them.\n\nTraining SAEs specialised for chat data closed about half the gap but was still worse than linear probes We tried several ways to train chat SAEs, all did about as well . By default, we recommend taking an SAE on pretraining data and finetuning it on a bit of chat data\n\nOther results: We found SAEs fairly helpful for debugging low quality datasets (noticing spurious correlations)\n\nWe present a variant of JumpReLU with an alternative sparsity penalty to get rid of high-frequency latents\n\nWe argue that a standard auto-interp approach of computing the average interpretability of a uniformly sampled SAE latent can be misleading as it doesn’t penalise models which have high frequency, but not very interpretable, latents, and explore weighting the interpretability score by latent frequency.\n\nIntroduction\n\nMotivation\n\nOur core motivation was that we, along with much of the interpretability community, had invested a lot of our energy into Sparse Autoencoder (SAE) research. But SAEs lack a ground truth of the “true” features in language models to compare to, making it pretty unclear how well they work. There is qualitative evidence that SAEs are clearly doing something , far more structure than you would expect by random chance. But they clearly have a bunch of issues: if you just type an arbitrary sentence into Neuronpedia , and look at the latents that light up, they do not seem to perfectly correspond to a crisp explanation.\n\nMore generally, when thinking about whether we should prioritise working on SAEs, it's worth thinking about how to decide what kind of interpretability research to do in general. One perspective is to assume there is some crisp, underlying, human-comprehensible truth for what is going on in the model, and to try to build techniques to reverse engineer it. In the case of SAEs, this looks like the hope that SAE latents capture some canonical set of true concepts inside the model . We think it is clear now that SAEs in their current form are far from achieving this , and it is unclear to us if such “true concepts” even exist. There are several flaws with SAEs that prevent them from capturing a true set of concepts even if one exists, and we are pessimistic that these can all be resolved:\n\nSAEs are missing concepts\n\nConcepts are represented in noisy ways where e.g. small activations don't seem interpretable\n\nLatents can be warped in weird ways like feature absorption\n\nSeemingly interpretable latents have many false negatives .\n\nFor more issues with SAEs see Section 2.1.2c of Sharkey et al.\n\nBut there are other high-level goals for interpretability than perfectly finding the objective truth of what’s going on - if we can build tools that give understanding that’s imperfect but enough to let us do useful things, like understand whether a model is faking alignment , that is still very worthwhile. Several important goals, like trying to debug mysterious failures and phenomena, achieving a better understanding of what goals and deception look like, or trying to detect deceptive alignment, do not necessarily require us to discover all of the ‘true features’ of a model; a decent approximation to the models computation might well be good enough. But how could we tell if working on SAEs was bringing us closer to these goals?\n\nOur hypothesis was that if SAEs will eventually be useful for these ambitious tasks, they should enable us to do something new today. So, the goal of this project was to investigate whether we can do anything useful on downstream tasks with SAEs in a way that was at all competitive with baselines - i.e. a task that can be described without making any reference to interpretability. If SAEs are working well enough to be a valuable tool, then there should be things they enable us to do that we cannot currently easily do. And so we thought that if we picked some likely examples of such tasks and then made a fair comparison to well-implemented baselines on some downstream task then, if the SAE does well (ideally beating the baseline, but even just coming close while being non-trivially different), this is a sign that the SAE is a valuable technique worthy of further refinement. Further, even if the SAE doesn’t succeed, it gives you an eval to measure future SAE progress, like how Farrell et al ’s unlearning setup was turned into an eval in SAEBench .\n\nOur Task\n\nSo what task did we focus on? Our key criteria was to be objectively measurable, be something that other people cared about and, within those constraints, aiming for something where we thought SAEs might have an edge. As such, we focused on training probes that generalise well out of distribution. We thought that, for sufficiently good SAEs, a sparse probe in SAE latents would be less likely to overfit to minor spurious correlations compared to a dense probe, and thus that being interpretable gave a valuable inductive bias ( though are now less confident in this argument ). We specifically looked in the context of detecting harmful user intent in the presence of different jailbreaks, and used new jailbreaks as our OOD set.\n\nSadly, our core results are negative:\n\nDense linear probes perform nearly perfectly, including out of distribution.\n\n1-sparse SAE probes (i.e. using a single SAE latent as a probe) are much worse, failing to fit the training set.\n\nk-sparse SAE probes can fit the training set for moderate k (approximately k=20), and successfully generalise to an in-distribution test set, but show distinctly worse performance on the OOD set.\n\nFinetuning SAEs on specialised chat data helps, but only closes about half the gap to dense linear probes.\n\nLinear probes trained only on the SAE reconstruction are also significantly worse than linear probes on the residual stream OOD, suggesting that SAEs are discarding information relevant to the target concept\n\nWe did have one positive result: the sparse SAE probes enabled us to quickly identify spurious correlations in our dataset, which we cleaned up. Note this slightly stacks the deck against SAEs, since without SAE-based debugging, the linear probes may have latched onto these spurious correlations – however, we think we plausibly could have found the spurious correlations without SAEs given more time, e.g. Kantamneni et al showed simpler methods could be similarly effective to SAEs here.\n\nWe were surprised by SAEs underperforming linear probes, but also by how well linear probes did in absolute terms, on the complex-seeming task of detecting harmful intent. We expect there are many practical ways linear probes could be used today to do cheap monitoring for unsafe behaviour in frontier models.\n\nConclusions and Strategic Updates\n\nOur overall update from this project and parallel external work is to be less excited about research focused on understanding and improving SAEs and, at least for the short term, to explore other research areas .\n\nThe core update we made is that SAEs are unlikely to be a magic bullet, i.e. we think the hope that with a little extra work they can just make models super interpretable and easy to play with doesn’t seem like it will pay off.\n\nThe key update we’ve made from our probing results is that current SAEs do not find the ‘concepts’ required to be useful on an important task (detecting harmful intent), but a linear probe can find a useful direction. This may be because the model doesn’t represent harmful intent as a fundamental concept and the SAE is working as intended while the probe captures a mix of tons of concepts, or because the concept is present but the SAE is bad at learning it, or any number of hypotheses. But whatever the reason, it is evidence against SAEs being the right tool for things we want to do in practice.\n\nWe consider our probing results disheartening but not enough to pivot on their own. But there have been several other parallel projects in the literature such as Kantamneni et al ., Farrell et al ., Wu et al ., that found negative results on other forms of probing, unlearning and steering, respectively. And the few positive applications with clear comparisons to baselines, like Karvonen et al , largely occur in somewhat niche or contrived settings (e.g. using fairly simple concepts like “is a regex” that SAEs likely find it easy to capture), though there are some signs of life such as unlearning in diffusion models , potential usefulness in auditing models , and hypothesis generation about labelled text datasets .\n\nWe find the comparative lack of positive results here concerning - no individual negative result is a strong update, since it’s not yet clear which tasks are best suited to SAEs, but if current SAEs really are a big step forwards for interpretability, it should not be so hard to find compelling scenarios where they beat baselines. This, combined with the general messiness and issues surfaced by the attempts, and other issues such as poor feature sensitivity, suggest to us that SAEs and SAE based techniques ( transcoders , crosscoders , etc) are not likely to be a gamechanger any time soon and plausibly never will be - we hope to write our thoughts on this topic in more detail soon. We think that the research community’s large investment into SAEs was most justified under the hopes that SAEs could be incredibly transformative to all of the other things we want to do with interpretability. Now that this seems less likely, we speculate that the interpretability community is somewhat over invested in SAEs .\n\nTo clarify, we are not committing to giving up on SAEs, and this is not a statement that we think SAEs are useless and that no one should work on them . We are pessimistic about them being a game changer across the board in their current form, but we predict that there are still some situations where they are able to be useful. We are particularly excited about their potential for exploratory debugging of mysterious failures or phenomena in models, as in Marks et al , and believe they are worthwhile to keep around in a practitioner's toolkit. For example, we found them useful for detecting and debugging spurious correlations in our datasets . More importantly, it’s extremely hard to distinguish between fundamental issues and fixable issues, so it’s hard to make any confident statements about what flaws will remain in future SAEs.\n\nAs such, we believe that future SAE work is valuable, but should focus much less on hill-climbing on sparsity reconstruction trade-offs , and instead focus on better understanding the fundamental limitations of SAEs, especially those that hold them back on downstream tasks and discovering new limitations; learning how to evaluate and measure these limitations; and learning how to address them, whether by incremental improvements or fundamental improvements. One recent optimistic sign was Matryoshka SAEs , a fairly incremental change to the SAE loss that seems to have made substantial strides on feature absorption and feature composition. We think that a great form of project is to take a known issue with SAEs, tries to think about why it happens and what changes could fix it, and then verifying that the issue has improved. If researchers have an approach they think is promising that could make substantial progress on an issue with SAEs, we would be excited to see that pursued.\n\nThere are also other valuable projects, for example, are there much cheaper ways we can train SAEs of acceptable quality? Or to get similar effects with other feature clustering or dictionary learning methods instead? If we’re taking a pragmatic approach to SAEs, rather than the ambitious approach of trying to find the canonical units of analysis, then sacrificing some quality in return for lowering the major up front cost of SAE training may be worthwhile.\n\nWe could imagine coming back to SAE research if we thought we had a particularly important and tractable research direction, or if there is significant progress on some of their core issues. And we still believe that interpreting the concepts in LLM activations is a crucial problem that we would be excited to see progress on. But for the moment we intend to explore some other research directions, such as model diffing, interpreting model organisms of deception, and trying to interpret thinking models.\n\nComparing different ways to train Chat SAEs\n\nLewis Smith, Arthur Conmy, Callum McDougall\n\nIn our original GemmaScope release, we released chat model SAEs, which were trained on activations from the chat tuned model, but on pretraining data (formatted as user prompts) not on chat rollouts from the model. When investigating the probe results discussed in the following sections, we decided that this approach was possibly sub-optimal, and wanted to make sure we were performing a fair comparison, after getting results we thought were disappointing. We hypothesised that this task was fairly chat-specific, but possibly the lack of chat rollouts in the training data meant that the SAEs would have failed to capture chat-specific features.\n\nIn order to investigate this, we experimented with two approaches:\n\nRetraining the SAEs from scratch, exclusively using chat data.\n\nFinetuning the existing GemmaScope SAEs on chat data.\n\nBoth of these approaches lead to improvements on our probing benchmarks relative to the baseline GemmaScope SAEs, matching Kissane et al , as discussed in the section below . However, neither is sufficient to match the performance of a dense probe in our setting. As finetuning is the easiest method and we generally did not find significant differences in our probe training task between training on the thing from scratch, we used finetuning for the results described in subsequent sections.\n\nIn addition, we experimented with a few variations of the finetuning procedure:\n\nstarting finetuning from the pretrained and instruction based GemmaScope models\n\nStarting the finetuning from before the end of training, so that the total number of train steps was the same between the finetuned SAE and IT trained SAE.\n\nlatent resampling, where when finetuning we randomly re-initialise a proportion of the latents in the SAE, under the hypothesis that normally all of the SAE’s latents are “already taken” and can’t easily adapt to learn new chat specific features.\n\nUsing probing metrics we didn’t find much systematic difference between any of these approaches; provided we trained on rollouts in some form, there did not appear to be much advantage to including them in the pretraining mix vs finetuning on the target data.\n\nWhen we looked at auto-interpretability metrics, we found that finetuning from the GemmaScope SAEs trained on the PT data performed slightly better than from our original ‘IT’ SAEs (which included the IT formatting but not rollouts) provided we didn’t perform latent resampling. Note that we use frequency-weighted autointerp scores here, so latents which fire more commonly in chat data specifically will be given more weight in the final value (for more on this, see the section Autointerp and high frequency latents later). These results are shown in the figure below.\n\nWe also experimented with including the refusal prompts used for training our probes (see the following snippet) in our finetuning mixture. Similarly to resampling latents, we found that this did not make a significant difference, at least compared to the noise in the SAE training process. The autointerp results are challenging to interpret given their high dataset dependence, but we didn’t see any statistically significant differences using this method.\n\nThe plot below shows the number of repeats of the probe data included in the SAE finetuning mix, faceted by the proportion of SAE latents resampled before finetuning. This is only showing the OOD set probing performance, but this is fairly representative; the effect of these changes does not seem to be statistically significant.\n\nThe conclusions to draw from this are a little uncertain, as the auto-interp results suggest that finetuning can reduce interpretability compared to our IT-training used in the GemmaScope release if latent resampling is used, whereas this still improved performance in our probing baselines, though this may be explained by the fact that the chat finetuning is short compared to pre-training (so chat-specialised latents are less well trained compared to the pretraining latents they have replaced, and the pretraining latents contribute to interpretability score even if they aren’t useful for this task).\n\nUsing SAEs for OOD Probing\n\nLewis Smith, Sen Rajamanoharan, Arthur Conmy\n\nOne obvious task where we can compare SAEs to a widely used and practical baseline is probing for ‘concepts’ such as harmful user intent, or the topic of a query to the chat model. A commonly used and effective baseline here is linear probing for a particular property, using a linear classifier trained on a dataset of positive and negative examples for the property of interest.\n\nSince linear probes are supervised while SAEs are not, it would not be surprising for linear probes to outperform SAEs on the training distribution. However, it’s less obvious which method would be best when measuring out of distribution generalization of our probe.\n\nIf SAEs have learnt latent representations which approximate the features actually used by the model, we might expect that probes based on a single SAE latent, or a sparse probe based on just a few latents, would only capture the most important concepts relevant to our task, which should generalise well out of distribution. While a dense probe has more capacity to pick up on subtler sources of spurious correlation (however, we later thought of considerations the other way, as discussed below )\n\nWe investigate this by probing for prompts with harmful user intent, i.e. prompts where the user asks the model to do something harmful/enabling harm. This is similar to refusal, but we don’t condition on whether the model actually refuses; the label should remain ‘harmful’ even if a jailbreak is successfully used and the model no longer refuses. We introduce a cha","truncated":true,"next_offset":20000,"facts":{"authors":["lewis smith","Senthooran Rajamanoharan","Arthur Conmy","CallumMcDougall","Tom Lieberum","János Kramár","Rohin Shah","Neel Nanda"],"publisher":null,"published_date":null,"doi":null,"archived_url":"https://web.archive.org/web/20261028213447/https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks","events":[],"versions":[],"copy_of":[]},"note":"More text follows; call again with offset 20000 to continue."}