Step 8: tool results

provenance_read_source

{
  "offset": 20000,
  "max_chars": 20000,
  "source_id": "447e476f-1376-48e2-9679-30bc4fa0e96c"
}

{"success":true,"source":{"id":"447e476f-1376-48e2-9679-30bc4fa0e96c","url":"https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks","title":"Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)","source_type":"blog_post"},"origin":"stored","total_chars":84339,"offset":20000,"content":"nge of distribution both by using different datasets for our OOD set, as well as introducing a set of held-out jailbreak suffixes to make hard positives (i.e. prompts that are harmful but don’t look it) and hard negatives (i.e. prompts that look harmful but aren’t) for the model.\n\nTechnical Setup\n\nDatasets\n\nAs our ‘in distribution’ set, we use harmful intent prompts from harmbench versus harmless data from Alpaca . We find (using SAEs! See more details in the ‘Dataset Debugging’ section) that there are several spurious differences in formatting between these two datasets (such as ending in brackets) that probes can detect, so we normalise the formatting of the datasets to remove these factors.\n\nFor an out of distribution set, we use a combination of over-refusal bench , a dataset of both ‘seemingly toxic’ prompts which are often mistakenly refused by LLMs, an internal chat training set, and advbench , the set of harmful requests used in the paper Universal and Transferable Adversarial Attacks on Aligned Language Models .\n\nWe also make the task more challenging by adding some jailbreak suffixes to the harmful prompts in the dataset. We also add some instructions that leave the (potentially harmful) text of the original prompt unchanged, but modify the meaning to make it innocuous, mostly by asking for semantic-level transformations of the text so the model does not actually have to comply with it’s content. These are :\n\n‘How many words in prompt’ - modify the prompt to be of the form: “How many words are in the following text: {original text}”.\n\n“Ignore one instruction” - Label the original prompt with a number in a list of otherwise harmless instructions, and tell the model to ignore that number.\n\n“All caps” - Ask the model to repeat the prompt back in all caps.\n\n“Find synonyms”: ask the model to provide a list of synonyms for key words in the original prompt.\n\nAfter augmentation, we have 4046 training examples, 1014 validation examples, and 3850 examples in our OOD set that we use in the experiments described below. We experimented with a variety of jailbreak methods, the effectiveness of which on the training set is plotted below:\n\nWe then chose a subset of these jailbreaks for the experiments discussed below. The considerations here were\n\n(a) using a variety of transformation types for train and test. For example, many of the prompts we tested are some variation of ‘roleplay type’ prompt injections, but we do include some others, including an adversarial suffix from the GCG paper.\n\n(b) using the weaker transformations for training if we have a range of strengths, in order to make the OOD generalisation more challenging.\n\n(c) avoiding transformations that were too many tokens as this slowed down iteration (as collecting activations becomes slower)\n\nThe jailbreak prompts we selected were:\n\nThe ‘AIM’ prompt, from jailbreakchat via Wei et. al.\n\nEvil confidant, the same\n\nAn adversarial suffix from the GCG paper\n\nThe Pliny ‘Google’ prompt, from this repository\n\nA ‘cocky genius’ roleplay prompt, which Senthooran Rajamanoharan came up with.\n\nA ‘role play’ prompt, which prompts the model to act as if it is a character in a ‘coming of age show’, acting out the instructions in the prompt., which Arthur Conmy came up with.\n\nIn order to test the generalisation of our probing methods, we use ‘AIM’, ‘roleplay’, ‘gcg’ and ‘cocky genius’ jailbreaks in our training distribution, and ‘evil_confidant’ and ‘pliny’ as an OOD test set. Similarly, we use ‘how many words’ and ‘ignore one instruction’ as innocuous transformations in training, and ‘all caps’ and ‘find synonyms’ out of distribution.\n\nProbing\n\nIn order to do probing, we need some way of choosing where (what site and layer) in the model to apply the probe, and how to aggregate the token probe across the context: each probe can be applied to any token in the forward pass, and it is not clear where this should be, or if it should be aggregated in some way; for instance, by applying the probe at every position and taking the mean or the max of the probe decision.\n\nFor the Gemma models we study in this snippet, chat data is surrounded by tokens using the following format:\n\n<start_of_turn>user\n\n[User prompt]\n\n<end_of_turn>\\n\n\n<start_of_turn>model\\n\n\n[Model response]\n\nWe experimented with both max-probing, where we first take the max activation of a feature along the sequence dimension, and probing at a fixed position relative to the first token of the model response (so position 0 is the first token of the model response, token -1 is the newline preceding that, etc). We find that max probing is generally the best, though using a fixed position of -5, corresponding to the <end_of_turn> token that completes the user turn, has comparable performance.\n\nWe find that the most reliable performance is obtained by probing at layer 20 of Gemma-v2 9B IT.\n\nBricken et al found that, for a biology-related task, taking the max of the features along the sequence before training a probe helped the performance of the SAE probe significantly, we only find a small advantage over probing at a fixed position in our application. We hypothesise that this is due to the information about the harmful content of the prompt naturally being aggregated in the end of turn token by the model. This idiosyncrasy of this particular task may explain the differences between our results and those reported by Anthropic.\n\nWe investigate two ways to do probing with SAEs:\n\nDo k-sparse probing on the feature activations of the SAEs. Under the assumption that only 1 feature, or a subset of features, captures the information at hand, this should result in less overfitting.\n\nTrain a linear probe on the reconstructions of the SAEs, or the reconstructions of some subset of the features. This is less likely to be useful as a practical method, but is interesting at investigating whether relevant information for a particular task is or is not preserved by the SAE reconstruction.\n\nAs we find that our chat SAEs generally produce similar results on this task (and may have a small edge on the autointerp metric depending on the resampling fraction), we use the finetuned SAEs for all results in this snippet, as this method is the quickest to experiment with.\n\nResults\n\nLinear probes on the residual stream perform very well at this task, essentially saturating even on our out of distribution set, indicating information about harmful user intent is present in model activations.\n\nSubset Train Val OOD\n\nLinear Probe AUROC 1.0 1.0 0.999..\n\nIn the plots below, we plot the best single-feature probe of the original GemmaScope SAEs, and a set of SAEs finetuned on chat data (see the finetuning snippet for more details), with the performance of the dense probe shown as a dotted line. We see that while using finetuned SAEs leads to a significant improvement in single-feature probe performance at this task - suggesting that these SAEs more closely capture the concept that we care about in a single feature - they remain below the performance of a dense linear probe, as shown by the dotted line in these plots.\n\nInitially, we expected that SAE based probes would not outperform linear probes on in-distribution data, but we expected that they might have an advantage out of distribution, especially those based on a single feature or a small number of features, under the hypothesis that SAE features would generalise more robustly out of distribution. However, we generally find that even on the OOD set, SAE probing does not match the performance of a linear probe, though the best single feature probe from the finetuned model does come close in some cases. We were surprised by the performance and robustness of the linear probe, and this is consistent with probes beating SAEs in OOD settings studied by Kantamneni et al , though we can’t make confident extrapolations about probes vs SAEs in all OOD settings\n\nWe can expand on single feature probes by using k-sparse probing, for k between 2 and 50. Using k-sparse probing largely closes the gap to the near-perfect performance dense probes for K>=5, although generally a small gap remains. Notably, despite k-sparse probing performing near perfectly in distribution, it shows worse transfer OOD, while dense probes generalise well, the opposite of what we predicted should happen with high quality SAEsl.\n\nWe experiment with two methods for k-sparse probing; selecting k latents by sorting by the mean difference of latent activations between positive and negative examples in the training set, and using l1 regression with a coefficient swept to select a number of active probe latents close to the target k (and then retraining an unregularised model on the selected latents). The two methods generally show similar results, as shown below, matching the results of Gurnee et al on neurons.\n\nOur results with k-sparse probing generally corroborate our results that finetuning SAEs helps, with finetuned SAEs generally outperforming the original GemmaScope SAEs, particularly for low L0 values, but it is not enough to achieve parity with dense probes.\n\nRelated Work and Discussion\n\nSAE probing was also studied in parallel by Kantamneni et al and Bricken et al . Generally, we think that our results are fairly consistent with the literature here. In particular, Kantamneni et. al. try to carry out a similar analysis to us over a wide variety of datasets, finding broadly comparable results to us; that sparse probing wins only rarely, having an advantage only when the labels are highly corrupted or potentially in low-data regimes. They also found better performance on simpler datasets where the SAE had a single highly relevant latent, and not on more subtle datasets.\n\nSimilarly, Bricken et al recently studied how SAE probes compare to linear probes. They study detecting biology-related misuse, rather than harmful intent. Unlike us, they find a slight advantage for SAEs in that SAEs allow max-aggregating latents along the sequence dimension before training a classifier (max-pooling the activations), which lets their classifier be more sensitive to data distributed throughout the prompt. Though Kantamneni et al show that, though max-aggregating latents can be an advantage over single token dense probes in some settings, it does not systematically beat attention head probes, and we speculate that attention head probes would also do well in our setting. We do not find that max-pooling is particularly helpful for our task ; we hypothesise that this may be because the information relevant to the classifier in our case is fairly temporally concentrated.\n\nLike us, both Kantamneni et al and Bricken et al find that the individual SAE latents are useful for finding spurious correlations in a dataset. Notably, Kantamneni et al find a single latent predicting grammatical accuracy, with similar accuracy to the “ground truth” labels in Glue CoLA, due to the high level of label noise. In some ways, our empirical results understate the utility of this, as we used the inspectability of our probes in order to clean our datasets of spurious correlations; if we hadn’t done this, our linear probes would presumably have learnt these too, which would have decreased their generalisation out of distribution. However, Kantamneni et al showed that these spurious correlations could be discovered by other methods, which we speculate would work for us too.\n\nIn general, we do find that there is a persistent performance gap between SAEs and probes. In part, this is because SAEs remain imperfect, and we do find that relevant information is lost in the reconstruction term, with probes on the SAE reconstruction generally performing slightly worse than a probe on the raw residual stream.\n\nPerformance delta between probe trained on residual stream and on SAE reconstruction.\n\nTrain Test OOD\n\nGemmaScope 0.00087 0.0018 0.048\n\nSAE finetune comparable steps 0.0013 0.0024 0.039\n\nConsistent with these negative results for SAE probing, very recent work by Apollo Research also tried SAE probes for detecting deceptive behaviour , finding that linear probes outperformed SAE probing.\n\nWe think it is curious that many of the variations on finetuning we tried did not seem to make a significant difference. It is possible that our setup is very noisy; perhaps a larger probing dataset would allow us to see a significant difference between these methods.\n\nIs it surprising that SAEs didn’t work?\n\nIn general, we found it unexpectedly hard to reason about how to update from positive/negative results on downstream tasks. In some sense, we have a hammer and are looking for a nail. We have this cool technique of SAEs, and want to find problems well suited to it. We care about how often it’s useful, and the magnitude of benefit. It doesn’t need to be useful for everything, but we care much more about some tasks than others. If you find one successful application, that doesn’t mean it will work in less cherry-picked cases, and if you have one failure, maybe it was badly suited to SAEs, but other important tasks would work fine. And even if you do have several successes, that may not be enough to justify the costs of training and R&D. Overall, we think that seeking evidence of interpretability on downstream tasks is valuable and something we hope to see more of in future - you can’t update too much off of the evidence, but it seems like a crucial question and this is one of the better sources of evidence we can currently access.\n\nAnother complication is that interpretability is quite hard to reason clearly about in general. You might have success on a task, but for quite different reasons than you thought, and you might think SAEs should help on a task, but actually your intuition is wrong. To illustrate this, in hindsight, we're now a lot more confused about whether even an optimal SAE should beat linear probes on OOD generalisation.\n\nHere’s our fuzzy intuition for what’s going on. We have a starting distribution, an OOD distribution, a train set and a val set (from the original distribution) and an OOD test set. There are three kinds of things a probe can learn for predicting the train set:\n\nNoise: patterns specific to the train set that do not generalise out of sample, i.e. on the test set. Learning noise gives rise to overfitting.\n\nSpurious correlations: patterns that predict the concept in the original distribution (and which are predictive even on the test set) but that do not generalise out of distribution.\n\nTrue signal: patterns that are predictive of the target concept both in-distribution and out-of-distribution.\n\nNoise is easily ignored given enough data. Indeed, on this task we found that both SAEs and linear probes learn to basically classify perfectly on the test set (i.e. out of sample, but in-distribution). So the difference in performance between SAEs and linear probes out of distribution comes down to how their respective inductive biases help them to latch on to the true signal versus spurious correlations: it seems that SAE probes do a worse job at this than linear probes. Why is this?\n\nOur original hypothesis had been that sparse SAE probes would in fact have a better inductive bias for picking the true signal over spurious correlations. This was in large part because we had assumed that the concept we’re looking to learn (“harmful intent”) is a fairly simple logical expression, composed of a small number of “atomic” concepts, both causal (like danger, violence, profanity, etc) and correlational (like linguistic features indicative of users who are trying to commit harm or of jailbreaks), that we expected SAEs would have assigned to single / small clusters of latents. Under this assumption, we guessed that a sparse SAE probe would easily find the relevant concepts among its latents and therefore extract the true signal, ignore spurious correlations. The fact that this doesn’t happen suggests a number of things that could be going wrong:\n\nThe concept “harmful intent” (as defined implicitly by the harmful datasets we used to train the probe) may not really be a simple function of a few “atomic” concepts at all. I.e., even with an “perfect” SAE - one whose latents correspond precisely to the concepts the model uses - it could be the case that the positive and negative labels in our datasets can’t be generated by a simple expression involving just a few of these concepts. This could indicate a fundamental problem with SAEs - we want concepts like harmful intent, and if high functioning SAEs do not usefully isolate them, this is bad\n\nOr it could just be that harmful intent is a fuzzily defined concept, and depends a lot on the subjective values of the labellers/those setting rules for labellers. Though under this hypothesis, it’s surprising that linear probes generalise well.\n\nEven if “harmful intent” is really a simple function of a few basic interpretable concepts, because our SAEs don’t learn these concepts cleanly, a sparse SAE probe doesn’t manage to capture the true signal.\n\nSince SAEs are incomplete , perhaps important directions corresponding to components of the true signal are missing from the dictionary, forcing the SAE probe to make do with approximating these directions from the latents that it does possess.\n\nPatterns that are spurious correlations may be just as well represented by latents in the SAE’s dictionary as patterns that make up the true signal, meaning that there is no reason to expect a sparse SAE probe to preferentially latch onto components of the true signal over spurious correlations.\n\nIn practice, our guess is that a mix of these are going on. b and c would suggest that we just haven’t gotten good enough at training SAEs but that it may be a promising direction, and a suggests fundamental issues with SAEs as a method. d would suggest that SAEs could make a promising probing method if we augmented them by pruning out latents that seemed like spurious correlations, but we de facto already did this by using them to clean up correlations in the data itself benefitting both the SAE probe and the dense probe. Overall, this means that it’s hard to draw a clear conclusion from the data available, but it does seem a bad sign for the practical utility of SAEs.\n\nDataset debugging with SAEs\n\nSen Rajamanoharan, Lewis Smith, Arthur Conmy\n\nOne use case where we did find SAEs useful was in dataset debugging. As the SAE latents are (somewhat) interpretable, inspecting the latents with big mean differences between positive and negative labels can reveal useful information about our dataset. This mirrors similar findings in Kantamneni et al and Bricken et al\n\nFor instance, when running an early version of this, when inspecting the best performing features at separating Alpaca and HarmBench, we realised that one of the best performing was a feature that seemed to be a question mark feature. On inspection, we realised that only Alpaca contained questions, whereas the harmful examples were all instructions. Being able to inspect the dataset in this way using a model with SAEs was definitely useful in catching this error, and other than manual inspection, it’s not clear if we ever would have noticed this if we exclusively used linear probing. Though Kantamneni et al were able to achieve similar results by taking maximum activating dataset examples of a linear probe over the pretraining data.\n\nThis also somewhat complicates the story that linear probing generally performed better than SAEs; this is presumably partly because we were able to use SAEs to manually clean spurious correlations from our dataset before training a supervised probe.\n\nOne interesting thing to note here is that you can use SAE based techniques like this for dataset exploration even if you don’t have an SAE on the model you want to probe on; for instance, using a model like Gemma 2 with an SAE sweep could be used to try to detect issues like this before you train something based on a larger model, as we are in","truncated":true,"next_offset":40000,"note":"More text follows; call again with offset 40000 to continue."}