Run step
Step 6: tool results
read_arxiv_source
Input
{
"file": "final_version.tex",
"offset": 30000,
"arxiv_id": "2505.20063",
"max_chars": 20000
}{"ok":true,"arxiv_id":"2505.20063","file":"final_version.tex","kind":"text","text":"xts using the metrics defined by \\citeauthor{wu2025axbench}: (1) the concept score measures if the concept was incorporated in the generated text; (2) the fluency score measures the coherency of the text; and (3) the instruct score measures the alignment of the generated text with the given instructions. \nFor each feature, we compute the harmonic mean of the three metrics. \nSee \\Cref{app:axbench} for additional details on the steering setup, metrics, and baseline methods. \n\n\n\\begin{figure}[t]\n \\centering\n \\begin{subfigure}{0.49\\linewidth}\n \\centering\n \\includegraphics[width=\\linewidth]{figures/io_scores_2b.png}\n \\caption{Gemma-2-2B.} \n \\end{subfigure}\n % \\vspace{0.5em} \n \\begin{subfigure}{0.49\\linewidth}\n \\centering\n \\includegraphics[width=\\linewidth]{figures/io_scores_9b.png}\n \\caption{Gemma-2-9B.} \n \\end{subfigure}\n \n \\caption{\n\\textbf{Relationship between input and output scores for features in Gemma-2-2B and Gemma-2-9B.} Most features lie near the axes, indicating that most features are either input \\emph{or} output features---though a few are both.}\n \\label{fig:io_scores}\n\\end{figure}\n\n\n\\begin{figure*}[ht]\n \\centering\n \\includegraphics[width=\\linewidth]{figures/high_io_examples_new.png}\n \\caption{Generation results when steering with features that have both high input and high output scores in Gemma-2-2B. The top logit lens tokens (left; top-1 in \\textcolor{blue}{blue}) do not appear directly in the generated text (right). The steered tokens (\\textcolor{magenta}{magenta}), appearing lower in the logit lens ranking, often have strong collocational associations with the top logit lens tokens.}\n \\label{fig:io_steering}\n\\end{figure*}\n\nWe report the mean score on layers 20 and 31 of instruction-tuned Gemma-2-9B (\\Cref{table:axbench}). \nWe find a nearly threefold improvement in SAE steering scores relative to \\citeauthor{wu2025axbench}. \nNote that our replication of their experimental setting (without filtering) yields higher scores for SAEs; this can be a result of different sampled instructions per feature,\\footnote{\\citet{wu2025axbench} sample 10 instructions per feature from pre-existing datasets, but do not release these instructions. We sample 10 instructions from the same datasets, which may be different compared to the sample of \\citeauthor{wu2025axbench}.\\label{footnote:axbench}}\nor possibly due to evaluation instability introduced by the use of an external LLM.\nWith output score filtering, SAE features top steering performance among the representation-based methods at L31; for L20, they get 90.7\\% of the performance of the best method. \nThis is in contrast with the results of \\citeauthor{wu2025axbench}, where SAEs significantly underperforms ReFT-r1, a weakly supervised method they propose as a competitive alternative to prompting. \nThese results demonstrates that with effective feature selection SAEs are comparable with existing methods, including supervised or weakly-supervised methods which require concept-specific datasets.\n\n\n\\section{The Relationship Between Input and Output Scores}\n\nWe next examine how input and output scores interact. \n\\Cref{fig:roles} suggests that high input and output scores are rarely observed in the same layers, but do they sometimes co-occur in the same features?\n\n\n\\Cref{fig:io_scores} visualizes the relationship between the two scores: Indeed, most samples cluster near the axes, exhibiting either a high output score with a near-zero input score, or vice-versa.\nHowever, there do exist hybrid features: features with both high input and high output scores.\n\n\\Cref{fig:io_steering} illustrates generation results when steering with hybrid features from Gemma-2-2B.\nGenerated tokens that also appear in the feature's top-20 logit lens tokens are highlighted in \\textcolor{magenta}{magenta}.\nThese are often not the top-ranked tokens under the logit lens, but they tend to rank moderately high and collocate with the top tokens (highlighted in \\textcolor{blue}{blue}). For instance, for feature 6820 in layer 18, ``contact'' is the top logit lens token, but the output text repeatedly includes ``lenses'',\na token that appears further down the list but that is semantically and syntactically related. \n\n\\begin{figure}\n \\centering\n \\begin{subfigure}{0.49\\linewidth}\n \\centering\n \\includegraphics[width=\\linewidth]{figures/pmi_2b.png}\n \\caption{Gemma-2-2B.} \n \\end{subfigure}\n \\begin{subfigure}{0.49\\linewidth}\n \\centering\n \\includegraphics[width=\\linewidth]{figures/pmi_9b.png}\n \\caption{Gemma-2-9B.} \n \\end{subfigure}\n\n \\caption{Pointwise Mutual Information (PMI) between generated tokens and top-1 logit lens tokens, grouped by input/output score thresholds. Features with high output scores and low input scores (\\textcolor{magenta}{magenta}) tend to have negative PMI values, likely due to exact token repetition during generation, which indicates successful steering. In contrast, features with high input scores (\\textcolor{blue}{blue}) consistently yield higher PMI values, indicating stronger collocational relationships with their top tokens.}\n \\label{fig:pmi}\n\\end{figure}\n\nWe verify this intuition by quantifying collocation patterns between generated tokens and top logit lens tokens using their pointwise mutual information (PMI)\\footnote{We use the pre-computed PMI over the webtext corpus in NLTK \\cite{bird-loper-2004-nltk}, and only include token pairs that have this pre-computed score (100--200 pairs per model).}.\nA PMI of zero indicates that two tokens co-occur no more frequently than chance, and negative and positive scores indicate lower- or higher-than-chance co-occurrence, respectively.\n\nFeatures with high output score ($S_{out}\t\\geq 0.1$) and low input score ($S_{in} < 0.1$) have a negative PMI value on average (\\Cref{fig:pmi}). \nThis can be attributed to the high generation success of these features, i.e., that the top logit lens token is equal to the generated token. \nIn most cases, two instances of the same token are not likely to consecutively co-occur, thus obtaining a low PMI score.\n\nImportantly, features that have a high input score ($S_{in} \\geq 0.1$) tend to generate tokens with significantly higher PMI relative to their top logit lens tokens, regardless of their output score values. \nThis suggests that\nhybrid features may be less favorable for steering, despite their high output score.\n\n\n\\section{Related Work}\n\\subsection{Stages of Processing in LMs}\nThe different stages of processing within NLP models have long been studied \\cite{belinkov2017translation, zhang2018language, liu2019linguistic, brunner2020identifiability}. \nIn transformer-based LMs, a large body of work shows that different properties emerge in different layers. Early layers focus on syntactic tasks such as POS, while semantic information appears in later layers \\cite{bert2019tenney, elazar2021amnesic, geva2021keyvalue}. \nMore recent work has demonstrated that intermediate layers are responsible for retrieving factual knowledge and enriching latent representations \\cite{meng2022locating, geva2023recall, hernandez2024linearity, arad2024refact}.\n\nAnother line of work has focused on so-called\nprediction neurons, which increase the probability of coherent sets of tokens. This work characterizes prediction neurons by properties of their logit lens distribution \\cite{gurnee2024universal, lad2024mstages, saelogitlens}. In contrast, our output score directly measures the \\emph{causal} effect of a feature on predicting a pre-defined set of tokens via counterfactual interventions. \nRelatedly, \\citet{lad2024mstages} have shown that neurons in early layers pay more attention to input tokens in their proximity compared to later layers, while prediction neurons emerge later, after about 50\\% of the model depth, in line with our findings on SAEs.\n\nA closely related line of work aims to explain SAE features in natural language---for instance, by feeding inputs and activations into an external LLM \\cite{bills2024explaining, interpretable2024huben}. \nHowever, this method results in errors in both precision and recall \\citep{huang-etal-2023-rigorously}, and \\emph{negatively} (albeit weakly) correlates with their causal role on average \\citep{paulo2024automatically}. \n\\citet{gurarieh2025enhancing} suggest that SAE features are better explained in terms of their activations \\emph{and} their effect on the output(as quantified by projecting features into vocabulary space).\nIn this work, our aim is not to \\emph{explain} features, but rather \\emph{categorize} them with respect to their usefulness for steering. \nAdditionally, we differentiate between two key feature roles (often mutually exclusive); this helps explain these prior findings and failure cases. \n\n\\subsection{Steering LMs}\nMany approaches exist for precisely influencing the outputs generated by LMs \\cite{zou2023representation}.\nThese include prompt engineering \\cite{wu2025axbench, taveekitworachai2024null}, steering vectors \\cite{subramani2022extracting, teehan2022emergent, liu2023context} or\ninference-time interventions on activations \\cite{turner2023steering, van2024extending, rimsky2024steering}.\nSteering was shown to be useful not only for directing generated content to a specific topic or concept, but also for style transfer \\cite{lai2024style}, mitigating hallucinations \\cite{li2023inference, simhi2024constructing}, and debiasing \\citep{li2025fairsteer}. \n\nWhile most work in this area steers via interventions on full hidden states, earlier work attempted to influence model behavior by intervening on small sets of neurons \\cite{bau2018identifying}. \nHowever, the polysemanticity of neurons makes them poor candidates for effective steering \\cite{bricken2023monosemanticity}. \nIn contrast, SAEs were shown to result in meaningful steering towards human-understandable concepts; a famous example involved steering toward responses related to the Golden Gate Bridge \\cite{scaling2024templeton}, and another involved amplifying or mitigating social and political biases \\cite{durmus2024steering, marks2025sparse}.\nRecently, \\citet{wu2025axbench} evaluated SAE steering against many methods, including supervised methods such as full fine-tuning, prompting, and difference-in-means \\cite{pmlr-v48-larsen16}. \nThey found that even these simple baselines outperform SAEs.\nHowever, our work shows that most of the gap can be closed via more careful choice of SAE features. \nTypical work chooses features for steering based on natural language explanations generated based on each feature's activation patterns \\cite{interpretable2024huben, durmus2024steering};\nour findings instead suggest that \\emph{influence on output} is a better proxy for steering efficacy, and that input activations have little predictive power for finding good steering features.\n\n\n\\section{Conclusions}\nWe have formalized and analyzed two roles demonstrated by sparse autoencoder (SAE) features.\nWe have defined the notion of an input score, which captures the alignment of a feature's activations with its top logit lens tokens, and an output score, which quantifies the alignment of the top logit lens tokens with the feature's effect on the model's generations.\nWe demonstrate that features with high output scores are significantly more effective for steering, whereas features with high input scores are relatively ineffective, even when they appear relevant to the steering concept.\n\n\n\\section*{Limitations}\nWhile our work provides an efficient framework for identifying and leveraging output features for generation steering, several limitations remain.\nFirst, our analysis is restricted to features extracted from the residual stream, and does not account for features derived from other components such as attention or MLP layers. As a result, our taxonomy may not capture the full range of functional roles present across the model.\n\n% Following previous work, we focus our analysis on the Gemma-2 family of models and the Gemma-Scope SAEs \\cite{team2024gemma, lieberum2024gemma}. While these models are widely adopted for interpretability research, further analysis across different LMs and SAE architectures would be beneficial to assess the robustness of our method.\n\nAdditionally, our method focuses on steering using a single SAE feature.\nIn practice, interactions between features may lead to better and more complex effects on generation \\citep{wattenberg2024relational,singhvi-etal-2025-using}. \nUnderstanding how multiple features combine or interfere remains an open challenge.\n\n\\section*{Ethical Considerations}\nOur work suggests a framework that improves one's ability to choose meaningful SAE features for steering LMs. While steering can support positive use cases such as controllable text generation, personalization, and bias mitigation, it can also introduce risks that must be considered. \nIn particular, steering methods can be used to manipulate model outputs in ways that circumvent safety mechanisms or amplify harmful content. \nAdditionally, our methods rely on pre-trained models that may contain biases or harmful associations. \nAlthough our framework can help isolate and suppress such patterns, it can also be misused to reinforce them.\n\n\\section*{Acknowledgments}\nThis research was supported by an Azrieli Foundation Early Career Faculty Fellowship and by Open Philanthropy.\nDana Arad is supported by the Ariane de Rothschild Women Doctoral Program.\nAaron Mueller was supported by a postdoctoral fellowship under the Zuckerman STEM Leadership Program. \nThis research was funded by the European Union (ERC, Control-LM, 101165402). \nViews and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency.\nNeither the European Union nor the granting authority can be held responsible for them.\n\nWe thank Shoval Lagziel and Yonatan Aflalo for early feedback on this work. We thank Adi Simhi for her support and feedback.\n\n\\bibliography{citations}\n\n\n\\appendix\n\\section{Additional Steering Examples}\nTables \\ref{app_table:steering_examples}, \\ref{app_table:pythia_steering_examples}, and \\ref{app_table:llama_steering_examples} show examples of steering Gemma-2-2B, Pythia, and Llama-3.1, respectively.\nFeatures with high output scores result in meaningful steering, while features with high input score \\emph{and} low output score do not have any visible effect on the generated text.\n\n\n\\begin{table*}[]\n% \\small\n\\resizebox{0.95\\linewidth}{!}{%\n\\begin{tabular}{lllllll}\n\\toprule\nLayer & Feature & \\begin{tabular}[c]{@{}l@{}}Input \\\\ Score\\end{tabular} & \\begin{tabular}[c]{@{}l@{}}Output \\\\ Score\\end{tabular} & Top-5 Logit Lens Tokens & \\begin{tabular}[c]{@{}l@{}}Optimal \\\\ Steering \\\\ Factor\\end{tabular} & Generated Text \\\\\n\\midrule\n11 & 10662 & \\textbf{0.756} & 0 & \\begin{tabular}[c]{@{}l@{}}{[}'\\_engineers', '\\_engineer', \\\\ '\\_engineering', 'Engineers',\\\\ '\\_Engineering'{]}\\end{tabular} & 0.2 & \\begin{tabular}[c]{@{}l@{}}Funny thing is, I bought my \\\\ first pair of these shoes \\\\ (the black leather) over 20 \\\\ years ago when\\end{tabular} \\\\ \n\\\\\n13 & 11961 & \\textbf{0.778} & 0 & \\begin{tabular}[c]{@{}l@{}}{[}'\\_machines', 'Machines', \\\\ 'machines', ' Machines', \\\\ '\\_machine'{]}\\end{tabular} & 0.2 & \\begin{tabular}[c]{@{}l@{}}A friend of mine once said, \"I\\\\ always wanted to be an architect.\"\\end{tabular} \\\\ \\\\\n16 & 731 & \\textbf{0.8} & 0 & \\begin{tabular}[c]{@{}l@{}}{[}'\\_exposure', 'exposure', \\\\ '\\_Exposure', 'Exposed', \\\\ '\\_Exposed'{]}\\end{tabular} & 0.2 & \\begin{tabular}[c]{@{}l@{}}Findings show that children in the \\\\ United States are eating more breakfast \\\\ foods, but less fruit and vegetables.\\end{tabular} \\\\\n\\\\\n18 & 9085 & 0.023 & \\textbf{0.142} & \\begin{tabular}[c]{@{}l@{}}{[}' activism', ' activists', \\\\ ' activist', ' protest', ' \\\\ protesting'{]}\\end{tabular} & 6.0 & \\begin{tabular}[c]{@{}l@{}}I once heard that the biggest \\\\\nand most powerful \\\\ movement for human civil activism \\\\ and peace movement, is an \\\\ movement for peace movement\\end{tabular} \\\\\n\\\\\n19 & 10015 & 0.023 & \\textbf{0.734} & \\begin{tabular}[c]{@{}l@{}}{[}'\\_profile', '\\_Profile', \\\\ 'Profile', 'profile', \\\\ '\\_PROFILE'{]}\\end{tabular} & 4.0 & \\begin{tabular}[c]{@{}l@{}}Findings show that a profile \\\\ picture in your profile is helpful \\\\ and makes it more likely \\\\ that people will add you as a\\end{tabular} \\\\ \n\\\\\n19 & 10204 & 0 & \\textbf{0.451} & \\begin{tabular}[c]{@{}l@{}}{[}'\\_crime', '\\_corruption', \\\\ '\\_violence', '\\_fraud', \\\\ '\\_crimes'{]}\\end{tabular} & 1.6 & \\begin{tabular}[c]{@{}l@{}}Then the man said: \"If I commit murder,\\\\ the crime will be on my conscience.\"\\end{tabular} \n\\\\\n\\bottomrule\n\\end{tabular}}\n\\caption{Examples of steering with features with different output and input score values in Gemma-2-2B.}\n\\label{app_table:steering_examples}\n\\end{table*}\n\n\n\n\\begin{table*}[]\n% \\small\n\\resizebox{0.95\\linewidth}{!}{%\n\\begin{tabular}{lllllll}\n\\toprule\nLayer & Feature & \\begin{tabular}[c]{@{}l@{}}Input \\\\ Score\\end{tabular} & \\begin{tabular}[c]{@{}l@{}}Output \\\\ Score\\end{tabular} & Top-5 Logit Lens Tokens & \\begin{tabular}[c]{@{}l@{}}Optimal \\\\ Steering \\\\ Factor\\end{tabular} & Generated Text \\\\\n\\midrule\n4 & 7772 & 0 & \\textbf{0.794} & \\begin{tabular}[c]{@{}l@{}}{[}'\\_Firefox', '\\_Chrome', \\\\ '\\_browser', '\\_Mozilla', \\\\ '\\_browsers'{]}\\end{tabular} & 1.2 & \\begin{tabular}[c]{@{}l@{}}Findings show that a \\\\ site may play a key role \\\\ in the development of a \\\\ web browser.\\end{tabular} \\\\ \n\\\\\n5 & 21568 & 0 & \\textbf{0.474} & \\begin{tabular}[c]{@{}l@{}}{[}'\\_Barack', '\\_Obama', \\\\ '\\_Donald', '\\_Trump', \\\\ '\\_Bush'{]}\\end{tabular} & 0.8 & \\begin{tabular}[c]{@{}l@{}}The legend goes that the guy \\\\ was a great- Barack Obama, \\\\ he'd probably be the biggest \\\\ supporter of Barack Obama ever\\end{tabular} \\\\ \n\\bottomrule\n\\end{tabular}}\n\\caption{Examples of steering with features with different output and input score values in Pythia-70m.}\n\\label{app_table:pythia_steering_examples}\n\\end{table*}\n\n\n\\begin{table*}[]\n% \\small\n\\resizebox{0.95\\linewidth}{!}{%\n\\begin{tabular}{lllllll}\n\\toprule\nLayer & Feature & \\begin{tabular}[c]{@{}l@{}}Input \\\\ Score\\end{tabular} & \\begin{tabular}[c]{@{}l@{}}Output \\\\ Score\\end{tabular} & Top-5 Logit Lens Tokens & \\begin{tabular}[c]{@{}l@{}}Optimal \\\\ Steering \\\\ Factor\\end{tabular} & Generated Text \\\\\n\\midrule\n2 & 25580 & \\textbf{0.436} & 0 & \\begin{tabular}[c]{@{}l@{}}{[}'\\_sources', '\\_SOUR', \\\\ 'sources', 'iped', \\\\ '/source'{]}\\end{tabular} & 0.8 & \\begin{tabular}[c]{@{}l@{}}That reminds me of the time \\\\ when I was in a public of knowledge \\\\ for the answer of information that said \\\\ the information was said that said\\end{tabular} \\\\ \\\\\n20 & 17816 & 0 & \\textbf{0.669} & \\begin{tabular}[c]{@{}l@{}}{[}'\\_visa', '\\_immigration', \\\\ '\\_Immigration', '\\_visas', \\\\ '\\_Visa'{]}\\end{tabular} & 1.6 & \\begin{tabular}[c]{@{}l@{}}I believe that the H-1B visa program \\\\ is an important tool for employers \\\\ to access the best talent to fill\\end{tabular} \\\\ \\\\\n26 & 21627 & 0 & \\textbf{0.665} & \\begin{tabular}[c]{@{}l@{}}{[}'\\_disability', '\\_disabled', \\\\ '\\_Disability', '\\_disable', \\\\ '\\_Disabled'{]}\\end{tabular} & 1.2 & \\begin{tabular}[c]{@{}l@{}}Findings show that the mainstream \\\\ media has a strong impact on public \\\\ opinion on disability.\\end{tabular} \\\\ \\\\\n\\bottomrule\n\\end{tabular}}\n\\caption{Examples of steering with features with different output and input score values in Llama-3.1-8B.}\n\\label{app_table:llama_steering_examples}\n\\end{table*}\n\n\n\\section{Results on Llama-3.1-8B and Pythia-70m}\n\\label{app:llama_pythia}\n\\Cref{fig:app_roles} demonstrates the distributions of scores across layers for Llama-3.1 and Pythia. \n\n\\Cref{fig:app_os_th} shows the mean generation success @ 20 of steered generations when filtering out features with output scores below varying thre","chars":20000,"total_chars":64852,"offset":30000,"truncated":true,"next_offset":50000}