Investigating task-specific prompts and sparse autoencoders for activation monitoring

INVESTIGATING TASK-SPECIFIC PROMPTS AND SPARSE AUTOENCODERS FOR ACTIVATION MONITORING Henk Tillman OpenAI Dan Mossing OpenAI ABSTRACT Language models can behave in unexpected and unsafe ways, and so it is valuable to monitor their outputs. Internal activations of language models encode additional information that could be useful for this. The baseline approach for activation monitoring is some variation of linear probing on a particular layer: starting from a labeled dataset, train a logistic regression classifier on that layer’s activations. Recent work has proposed several approaches which may improve on naive linear probing, by leveraging additional computation. One class of techniques, which we call “prompted probing,” leverages test time computation to improve monitoring by (1) prompting the model with a description of the monitoring task, and (2) applying a learned linear probe to resulting activations. Another class of techniques uses computation at train time: training sparse autoencoders offline to identify an interpretable basis for the activations, and e.g. max-pooling activations across tokens using that basis before applying a linear probe. However, one can also prompt the model with a description of the monitoring task and use its output directly. We develop and test novel refinements of these methods and compare them against each other. We find asking the model zero-shot is a reasonable baseline when inference-time compute is not limited; however, activation probing methods can substantially outperform this baseline given sufficient training data. Specifically, we recommend prompted probing when inference-time compute is available, due to its superior data efficiency and good generalization performance. Alternatively, if inference-time compute is limited, we find SAE-based probing methods outperform raw activation probing. 1 INTRODUCTION To ensure the safety of AI systems in practice, other AI systems are often used to monitor their outputs. Two common approaches are (1) prompting large language models (LLMs) with a natural language monitoring task, and using natural language output as the monitor signal; and (2) probing LLM hidden layer activations using logistic regression classifiers, and using the classifier output as the monitor signal. The goal of this work is to ground these two approaches, prompting and probing, by comparing variants (and combinations) of both that have been proposed in the literature. We pay particular attention to probing variants involving sparse autoencoders (SAEs) which have been proposed as a way to leverage train time compute to improve probe performance. These approaches have various limitations in resource-constrained settings. For example, prompting the model does not require labeled training data for the monitoring task (a model can often perform the task zero-shot). However, prompting the model requires additional inference time compute (more forward passes are required for the monitoring prompt tokens). On the other hand, probing can be “free” at inference time (leveraging only the forward passes that were used to generate the model’s output in the first place). But probing may require substantial labeled training data, as LLM activations are not a priori interpretable unlike prompted natural language outputs. Combining SAEs with probing, as proposed in Bricken et al. (2024), or combining prompting with probing, as proposed in Zou et al. (2023), may improve the performance of probing, although evidence is mixed (Kantamneni et al., 2025; Smith et al., 2025). Therefore we investigate these techniques across various constraints, including training data availability, limited inference-time compute (i.e. whether prompting is possible), and restricted training-time compute (i.e. whether training SAEs is possible). 1 arXiv:2504.20271v1 [cs.LG] 28 Apr 2025

Figure 2: Prompted probing (orange) and SAE max-pooled probing (blue) help with classification performance across a diverse set of tasks. Prompted probing tends to help more in the low-data regime. We make the following key observations. First, prompted last token probing is broadly the most data-efficient technique, given its strength in the low-data regime. In high-data regimes it is comparable to SAE probing. Second, prompting is important, as the prompted last token probe outperforms the naive last token probe. This performance gap is widest in low-data regimes. Third, while prompted model output is a strong baseline, in high-data regimes, any of the three techniques (prompted probing, SAE probing techniques, and raw last token probing) can outperform it with sufficient training data. In particular, all probing techniques are substantially stronger than prompted model outputs in detecting hallucinations in response to SimpleQA questions. We next measure generalization performance. We examine two types of distribution shift: from English to non-English, and from single message to multi-turn conversation (i.e. Chat to non-Chat). We evaluate generalization on the moderation datasets because they contain all these sorts of data. We split the train set and the test set into two subsets each based on English/non-English and Chat/non-Chat labels. We train the classifiers on only the English or Chat portions of the train set (the in-distribution sets) and evaluate them on the in-distribution and out-of-distribution portions of the test set. In the following graphs, we plot in-distribution performance versus out-of-distribution performance for a range of train set sizes. We find that prompted probing, more so than SAE max-pooled probing, boosts linear probe performance in a way that generalizes well, though evidence is somewhat mixed across the settings we examine. For a given in-distribution performance (x-axis value), probing SAE activations sometimes hurts out-of-distribution performance compared with naive last token probes (in 2/4 cases), whereas prompted probing either improves or matches out-of-distribution performance compared with naive last token probes (note that in 1/4 cases, both prompted probing and SAE max-pooled probes generalize similarly well). This is broadly consistent with the finding from Bricken et al. (2024), Kantamneni et al. (2025), and Smith et al. (2025) that max-pooled SAE probes generalize poorly OOD. Note that out-of-distribution AUROC is sometimes higher than 8

in-distribution AUROC: these two measurements rely on a shared train set but distinct test tests, where the two test sets might not be equally “difficult.” Figure 3: When trained on English data and tested on English, prompted probes generalize relatively well (in the sense of strong OOD performance at a given in-distribution performance), while SAE max-pooled probes generalize poorly compared with naive last token activation probes. All methods perform more similarly when trained on single prompt data and tested on multi-turn conversation data. Sentiment classification performance appears in Figure 2 as an outlier, with SAE max-pooled probing underperforming the last token techniques in the low-data regime. We hypothesize that for the sentiment classification task, most of the signal is present in activations for the last token of the prompt (Tigges et al., 2023), and other token activations primarily add noise. Indeed, we see in the following figure that SAE-based probing only at the last token improves on SAE max-pooled probing. Another variant using SAE pre-activations (see 3.4) is competitive with the (non-SAE-based) last token probing technique. We conclude the performance degradation of SAE max-pooled probing on sentiment classification is a result of max-pooling across tokens and is not inherent to SAEs. Figure 4: Last token probes (blue and green) perform best at sentiment classification. SAE pre- activation last token probing is competitive with naive last token probing. 9

4.2 INVESTIGATIONS INTO STACKING MONITORING METHODS We next investigate whether combinations of our three primary methods (prompting, probing raw activations, and probing with SAEs) can improve on the individual methods themselves. Unfortunately, the benefits of prompted probing and SAEs do not substantially stack. In the following plots we combine prompting with our two most successful SAE methods, max-pooled SAE activation probing and last token SAE pre-activation probing, on harassment classification (chosen simply for illustration purposes). We compare with prompted probing by itself, and SAE max-pooled probing by itself. SAE max-pooled probing by itself already roughly matches the strongest combination method. Figure 5: Prompted probing does not substantially stack with SAE-based probing. If we are using the prompted probing technique, then there is no additional cost to classifying using model activations and model outputs. We investigate whether using the prompted model output as a supplementary feature for raw activation probing helps (see Section 6). We observe that while we can learn a classifier which roughly upper bounds zero-shot performance and activation probe performance, it does not substantially exceed the performance of either. That is, the combination of the two types of input adds no signal that is not present in the better performing of the two. Figure 6: A 2-level classifier which sums a prompted probe output logit with the model’s output does not improve substantially over the max of prompted probe performance and model output performance. 4.3 OPTIMIZING THE MONITORING METHODS To aid practitioners in implementing these methods, we present our results with optimizing hyperparameters and variants of the monitoring methods shown, specifically with varying model 10

Finally, in the following plot, we compare the method presented in Bricken et al. (2024) with the probe on SAE pre-activations described in the methods section. We observe no substantial difference in performance. That is, applying an activation function prior to max-pooling is not important for classifier performance. Figure 13: Classification performance is similar when using SAE latent activations (post-nonlinearity) or pre-activations (pre-nonlinearity). 5 DISCUSSION On the basis of our results, we recommend the following approach to using chat-trained LLMs for monitoring tasks without additional fine-tuning: 1. If inference-time compute is not limited, asking the model zero-shot is often a strong baseline 2. If plentiful training data is available, then using activations should help. Either of (a) prompted probing (if inference time compute is available) (b) SAE probing (if train time compute is available) might be optimal 3. If inference-time compute and train-time compute are both limited, straightforward activation probing is a reasonable choice, with strong returns on increasing dataset size Previous work has shown mixed results for the efficacy of SAE-based approaches compared with probes directly on activations (Bricken et al. (2024), Gallifant et al. (2025), Kantamneni et al. (2025), Smith et al. (2025)). In particular, Kantamneni et al. (2025) and Smith et al. (2025) found that SAE- based approaches rarely exceeded the best non-SAE-based approach they tried, across a diversity of datasets. Bricken et al. (2024), Kantamneni et al. (2025), and Smith et al. (2025) reported that SAE-based approaches generalized worse than non-SAE-based approaches. Our work here is broadly consistent with these findings, though we note that we find more favorable performance overall for SAE-based approaches than Kantamneni et al. (2025) and Smith et al. (2025) on the datasets we examined. We find prompted probing, which was not examined in these other works, to be highly competitive in terms of both data efficiency and generalization performance. The unsupervised version of this approach, LAT Scans, performs worse, unsurprisingly given that it uses strictly less information. This agrees with Goldowsky-Dill et al. (2025). Unsupervised approaches such as LAT Scans hold considerable appeal for monitoring in cases where labeled data is unavailable, and this direction seems worth exploring further. The success of probing shows that information about the desired concept is approximately linearly encoded in activations across tokens. We hypothesize that eliciting the concept from the model with a prompt causes the model to include information about the concept in its last-token activations. We further hypothesize that max-pooling SAE latent activations aggregates this signal into a single vector, 14