Step 10: tool results

read_arxiv_source

{
  "file": "main.tex",
  "offset": 20000,
  "arxiv_id": "2501.17148",
  "max_chars": 20000
}

{"ok":true,"arxiv_id":"2501.17148","file":"main.tex","kind":"text","text":"ReFT). On steering, only ReFT-r1 is competitive with finetuning and prompting baselines, while SAEs fall behind both ReFT-r1 and difference-in-means \\citep{marks2024geometrytruthemergentlinear} on both axes. While representation steering methods largely lag behind incumbent model-control techniques, ReFT-r1 is evidence that steering can be pushed further with the availability of comprehensive evaluation benchmarks. Finally, along with \\shortbenchmark{}, we train and publicly release SAE-scale feature dictionaries for ReFT-r1 and DiffMean.\n% %%%%%%%%%%%%%% TODO: COMMENT BACK %%%%%%%%%%%%%% \n\\footnote{We open-source all of our datasets and trained dictionaries at \\url{https://huggingface.co/pyvene}.}; we call this approach \\textit{supervised dictionary learning} (SDL; \\cref{fig:axbench})\n\n% In this work, we introduce \\textbf{\\shortbenchmark{}}, a framework for evaluating existing interpretability methods at scale using synthetic data. We evaluate along two utility \\underline{\\textbf{ax}}es: \\textbf{concept detection} \\concepttag{} and \\textbf{model steering} \\steeringtag{}. Surprisingly, we find that all existing steering methods fail to outperform a simple prompting-only baseline and generally lag behind gradient-based finetuning methods, disqualifying them from practical usage in \\emph{the status quo}.\n\n% In this work, we introduce \\textbf{\\shortbenchmark{}}, a framework for evaluating existing interpretability methods at scale using synthetic data. We formally descibe existing model interpretability methods (such as probing and SAEs) as \\emph{model abstractions} which aim to simplify complex LM behaviours. We focus on two tasks: \\textbf{concept detection} and \\textbf{model steering}. Surprisingly, we find that all existing methods struggle to outperform a simple prompting-only baseline, disqualifying them from any practical usage. \n\n% Representations in language models (LMs) serve as the medium through which input prompts are encoded, processed and then decoded into structured next-token predictions. These representations inherently capture latent patterns of syntax and semantics within language~\\cite{}, yet their high-dimensional nature makes them difficult to interpret and manipulate, limiting their practicality as an interface for interacting with LMs. \n\n% Model interpretability has made significant progress, yet its real-world impact remains limited. Unlike prompting LM with natural languages for model control and explanation~\\cite{}, current interpretability insights are often task-specific, limited in vocabulary, and difficult to scale~\\cite{}. Recent works such as sparse autoencoders (SAEs) tackle these challenges by providing scalable techniques in explaining and steering LMs~\\cite{}. However, proper evaluations of these tools are lacking despite their anecdotic successes~\\cite{}. Moreover, existing tools are costly to scale with a lengthy pipeline, which disallows them to iterate over task objectives effectively preventing any practical deployment~\\cite{}. \n\n% In this work, we define existing model interpretability tools such as probing or SAEs as \\emph{model abstractions} with a goal of simplifying complex LMs through different lenses for two purposes: latent semantic detection and model steering. We then introduce \\textbf{\\longbenchmark{}} (\\textbf{\\shortbenchmark{}}), a framework to evaluate existing interpretability tools grounded with these purposes at scale. Surprisingly, we show that existing tools struggle to outperform the simplest baselines, disqualifying them from any practical usage.\n\n% In response, we propose \\textbf{\\longmethod{}} (\\textbf{\\shortmethod{}}), a novel and scalable approach for creating abstractions in an open-vocabulary setting. Building on recent works in representation finetuning (ReFT)~\\cite{}, \\shortmethod\\ automates the pipeline of finding linear subspaces corresponding human interpretable concepts at scale with LM-in-the-loop. \\TODO{ReAX result goes here.}\n\n% Set up the header and footer\n\\pagestyle{fancy}\n\\fancyhf{}\n\\setlength{\\headwidth}{\\textwidth}\n\\fancyhfoffset[L,R]{0pt}\n\\fancyhead[C]{\\shorttitle{}}\n\\setlength{\\headheight}{15pt}\n\n\\begin{figure*}[!t]\n    \\centering\n    \\includegraphics[width=0.85\\textwidth]{assets/AxBench_v2.png}\n    \\caption{Key components of \\shortbenchmark{}: (a) an example of how we collect data for evaluating concept detection and model steering; (b) the synthetic data generation process for training and evaluation given \\textit{Golden Gate Bridge} as a concept; and (c) the contrasting training pipelines of SAEs and SDLs; both use LLMs, but SAEs use them to label pretrained features while we instead direct them to generate training data.}\n    \\label{fig:axbench}\n\\end{figure*}\n\n\\section{Related work}\n\\label{sec:related}\n\n\\paragraph{Representation-based control.} Interventional/causal interpretability has emerged as the dominant paradigm for understanding neural networks in the LLM era, enabling the reverse-engineering of circuits underlying specific behaviours \\cite{giulianelli-etal-2018-hood,genderbias,causalabstraction,pmlr-v162-geiger22a,meng2022,chan2022causal,wang2022interpretability,goldowsky2023localizing,geiger2024causal,guerner2024geometricnotioncausalprobing,geiger2024causal}. An important assumption in much of this work is the \\textbf{linear representation hypothesis}, which claims that linear subspaces of representations in neural networks encode concepts \\citep{mikolov-etal-2013-linguistic,pennington-etal-2014-glove,bolukbasi2016man,elhage2022superposition,park2023linear,nanda-etal-2023-emergent}. Intervening on representations has thus emerged as an alternative to finetuning and prompting for LM control.\n\n\\textit{Representation-based steering} by adding fixed vectors to activations, or clamping activations to a certain value along fixed directions, is one such intervention-based tool for model control \\citep{zou2023representationengineeringtopdownapproach,liInferenceTimeInterventionEliciting2024,turner2024steeringlanguagemodelsactivation,marks2024geometrytruthemergentlinear,liu2024incontext,vanderweij2024extendingactivationsteeringbroad,rimsky-etal-2024-steering}. Finetuning-based approaches such as ReFT \\cite{wu2024reft} enable optimisation of steering directions on a dataset. Steering vectors need not be computed from labelled data; SAEs enable scalable discovery of steering vectors from unlabelled data.\nIn the same class of approaches, latent adversarial training \\cite{casper2024defendingunforeseenfailuremodes} and circuit breakers \\cite{zou2024improvingalignmentrobustnesscircuit} are representation-based control methods that increase the adversarial robustness of LLMs.\n\n% While steering has shown some promise for model control, it is still unclear whether it can reliably outperform alternatives. Existing evaluations for supervised steering are limited \\citep{pres2024reliableevaluationbehaviorsteering}.\n\n\\paragraph{Sparse autoencoders.} Sparse autoencoders (SAEs) aim to enable \\textit{self-supervised} and thus \\textit{scalable} decomposition of the representation space into meaningful concepts \\cite{templeton2024scaling,chalnev2024improvingsteeringvectorstargeting,makelov2024sparse,obrien2024steeringlanguagemodelrefusal,gao2024scalingevaluatingsparseautoencoders}. SAEs are trained to reconstruct LLM hidden representations in a higher-dimensional latent space with a sparsity penalty, based on the assumption that concepts must be represented sparsely in order to prevent interference. The latents are then labelled with natural-language descriptions using automatic interpretability pipelines \\cite[e.g.][]{eleutherai}, which can then be used to identify useful latents to steer the LM.\n\nRecent work reports mixed results when evaluating SAEs for steering; SAEs (but also several other steering methods) suffer from a tradeoff between model control and capabilities preservation \\cite{mayne2024sparseautoencodersuseddecompose,chalnev2024improvingsteeringvectorstargeting,durmus2024steering,bhalla2025unifyinginterpretabilitycontrolevaluation}. However, \\citet{karvonen2024sieve} report Pareto-optimal performance when using SAEs to prevent models from producing regular expressions in code. Overall, evaluating SAEs remains an open problem because there is no ground-truth set of features to compare against.\n\n% \\section{Model abstraction}\n\n% Building on the theory of causal abstraction for neural networks~\\cite{geiger2024causal}, we conceptualize existing interpretability tools as \\emph{model abstractions}.\\footnote{In contrast to causal abstractions, which offer causal guarantees through interventions, model abstractions may lack causal implications and remain purely observational.} Intuitively, an abstraction simplifies a model by emphasizing a specific aspect of its functionality. Formally, we define $_{\\theta}\\rmA_{\\gC}^{\\model}$ as the abstraction for a Transformer-based language model $\\model$ with parameters $\\theta$. This abstraction simplifies $\\model$ by isolating parts relevant to a particular concept $\\gC$, where $\\gC$ represents concepts in an open-vocabulary setting, described in natural language (e.g., \"terms related to neural networks\").\\footnote{For clarity in this section, we denote the abstraction as $\\rmA_{\\gC}$, assuming we are working with a fixed-parameter model.}\n\n% \\paragraph{Abstracting and de-abstracting.} Abstraction is a pseudo-invertible operator on model representations. Given any example $\\mathbf{x}$ sampled from a dataset $\\data$, a Transformer-based language model $\\model$ processes the example and generates a set of hidden representations internally, denoted as $\\mathbf{h}_{i}^{l} \\in \\R^{1\\times d}$ in the residual stream at position $i$ in layer $l$.\\footnote{For simplicity, we focus only on representations in residual streams. Abstractions can apply to any component in the model's computation graph, such as self-attention outputs. We omit all superscripts and subscripts for clarity.} \n% The \\textbf{abstracting} process is described as $\\rmA_{\\gC}(\\mathbf{h}) \\rightarrow \\tilde{\\textbf{u}} \\in \\R^{1\\times m}$, mapping representations to an $m$-dimensional space where $m \\ll d$, ensuring abstraction utility. Note that $\\rmA_{\\gC}$ can represent any projection matrix or a complex neural model. The dimensionality constraint enforces model simplification.\\footnote{We consider only abstractions that simplify models.}\n% The \\textbf{de-abstracting} process is the inverse operation, described as $\\rmA^{-1}_{\\gC}(\\tilde{\\textbf{u}}) \\rightarrow \\tilde{\\mathbf{h}} \\in \\R^{1 \\times d}$.\\footnote{Note that $\\rmA_{\\gC}$ is not fully invertible, even if parameterized by a single projection matrix, since $m \\ll d$ implies it is not full-rank.} Intuitively, given $\\tilde{\\textbf{u}}$, perfect de-abstraction is unattainable due to the many-to-many mapping between a representation and a concept.\n\n% \\paragraph{Utilities.} Abstracting can be viewed as a form of \\textbf{latent semantic detection}. Consider a scenario where we binarize our concept $\\gC$ (e.g., binarizing the concept of \"terms related to neural networks\" into two groups, with the positive group containing terms related to the concept and the negative group not containing them). A faithful abstraction can map representations into two low-dimensional clusters via $\\rmA_{\\gC}(\\mathbf{h}^{+}) \\Rightarrow \\tilde{\\textbf{u}}^{+}$ and $\\rmA_{\\gC}(\\mathbf{h}^{-}) \\Rightarrow \\tilde{\\textbf{u}}^{-}$, where $\\tilde{\\textbf{u}}^{+}$ and $\\tilde{\\textbf{u}}^{-}$ are linearly separable in the low-dimensional space. In the case of a 1-dimensional subspace, this implies that $\\tilde{\\textbf{u}}^{+}$ and $\\tilde{\\textbf{u}}^{-}$ can be distinguished by a threshold. \n% De-abstracting enables \\textbf{model steering through intervention}. Given any $\\mathbf{h}^{*}$ originally mapped to a concept group $\\tilde{\\textbf{u}}^{*}$, we can create a counterfactual representation by setting $\\rmA^{-1}_{\\gC}(\\rmA_{\\gC}(\\mathbf{h}^{*}) \\leftarrow \\tilde{\\textbf{u}}^{-}) = \\tilde{\\mathbf{h}}^{-}$, enforcing a negative representation of the concept. We then replace the original representation with $\\tilde{\\mathbf{h}}^{-}$ via intervention or activation patching~\\cite{} to steer the model's behavior. \\TODO{Note that $\\rmA_{\\gC}$ and $\\rmA^{-1}_{\\gC}$ can have tied or untied parameters. In case the case tied parameters, it could lead to fully invertible de-abstraction. We can describe DAS, ReFT here or in the appendix to show we can have a perfect reconstruction. And for intervention, we can thus only intervene on a isoluated concept subspace without touch others. Basically, we need to define something like $_{\\perp}\\rmA_{\\gC}$}\n\n% \\paragraph{Abstraction creation.} There is no restriction in terms of how abstractions are obtained. Most of the existing interpretability tools for LMs can be conceptualized as abstractions. \\textbf{Model probing} is a type of model abstractions which trains $\\rmA_{\\gC}$ for latent semantic detection with an objective of mapping representations to a low-dimensional subspace where concepts are clustered~\\cite{}. \\textbf{Activation steering vector} is a type of model abstractions which trains $\\rmA^{-1}_{\\gC}$ which maps a low-dimensional concept variable to a representation, which can then be used for model steering via interventions. \\textbf{Sparse Autoencoders (SAEs)} is a type of model abstractions which learns $\\rmA_{\\gC}$ and $\\rmA^{-1}_{\\gC}$ in an unsupervised manner where the training objective is reconstruction representations while enforcing latent sparisty~\\cite{}.\n\n\\section{\\textbf{\\shortbenchmark{}}}\n\n\\shortbenchmark{} is a benchmark which takes in a list of natural language descriptions of concepts and synthetically generates the appropriate training and evaluation data for each concept using an LLM (\\cref{fig:axbench}). The training and evaluation data consists of labelled pairs of instructions and responses, where the responses are either \\textit{positive} examples expressing the presence of the concept of interest, or \\textit{negative} examples that represent the unsteered behaviour of the model (see \\cref{sec:concept-dataset} for details).\n\nWe evaluate along two axes: \\textbf{concept detection} \\concepttag\\ and \\textbf{model steering} \\steeringtag{}. For the former, we measure classification performance on a held-out set of labelled data.\\footnote{We focus on binarised concept detection, as a multi-class classification task over $n$ classes can also be formulated into a binarised one over $n$ features.} For the latter, we use an LLM judge to rate steered outputs on three relevant axes (see \\cref{sec:model-steering}).\n\nIn this work, we use natural language concept lists for \\texttt{GemmaScope} SAEs as input, and generate training and evaluation data for the following representation sites: layers 10 and 20 of instruction-tuned \\texttt{Gemma-2-2B}, and layers 20 and 31 of instruction-tuned \\texttt{Gemma-2-9B}. We sample 500\n% (out of $\\approx$16K)\nconcepts for each task to generate data; we term this dataset \\textsc{Concept500}. These eight tasks (4 sites $\\times$ 2 axes) form the core training and evaluation testbeds for \\shortbenchmark{}.\nBelow, we describe the data generation process and evaluation setup for both axes.\n\n% \\paragraph{SAEs vs.~supervised steering.} Representation-level interpretability methods ideally have two practical uses: they should help us detect the presence of certain concepts as the LM processes text, and they should let us control the concepts that the LM expresses during generation. Different methods provides different interfaces for detection and steering (e.g. SAEs expose detection via encoder activations and steering via decoder interventions), but methods generally share the goal of enabling both operations on LM representation space. \\shortbenchmark{} thus evaluates both \\concepttag{} \\textbf{concept detection} and \\steeringtag{} \\textbf{model steering}.\n\n% However, the methods we want to benchmark differ widely in how they are trained: SAEs are trained to self-supervisedly reconstruct activations, while probing, LAT, and many other steering methods are supervisedly trained on labelled data. We must first figure out how to train the supervised methods to make them comparable to SAEs.\n\n% The key to comparable training and evaluation is natural-language descriptions of features. After pretraining, SAE features are labelled with natural-language explanations in order to e.g.~be searchable for end-users who want to steer models using them. The feature descriptions thus parameterise the SAE. Supervised steering methods, on the other hand, are parameterised by a dataset, which contains examples of how steering should affect model outputs. To evaluate all of these methods together, we first need equivalence between an SAE feature description and the supervised steering dataset.\n\n% Since many large lists of SAE feature descriptions have already been released publicly,\\footnote{In this work, we primarily use labels from \\url{https://www.neuronpedia.org/}, but in principle any SAE concept list could be used.} we devise a pipeline to synthetically generate supervised steering datasets from SAE feature descriptions using an LLM. These datasets include both positive and negative examples of the behaviour described by a feature description, generated according to the pipeline in \\cref{fig:axbench}.\n\n% \\paragraph{Evaluation.} Now that we can train equivalent supervised steering methods for every concept in an SAE feature description list, we can turn to evaluation. The pipeline for generating the supervised steering dataset actually directly enables evaluation of concept detection: it contains passages of text with ground-truth labels for the presence of a concept. We generate a held-out concept detection evaluation dataset along these lines which also includes LM-generated hard negatives to test robustness (see \\cref{sec:concept-dataset}).\n\n% For steering, we sample model responses to user instructions (sampled from existing instruction-tuning datasets) after applying the steering operation for each method and concept. We ask an LM judge to evaluate the response on three axes: whether it encodes the concept we desired, whether it directly answers the instruction, and whether it is fluent and coherent (see \\cref{sec:model-steering}).\n\n% This summarises the framework underlying \\shortbenchmark{}. We describe more details about implementation below and provide prompts for the LM data generation and judging pipelines in \\cref{sec:prompt_templates}.\n\n% Unlike prompting or using pretrained SAEs off-the-shelf, some interpretability methods need supervision in training (e.g.~probing).  This synthetic data generation process also allows us to construct ground-truth evaluation data for\n\n\\subsection{Synthetic concept dataset generation}\n\\label{sec:concept-dataset}\nWe construct a small training dataset $\\mathcal{D}_\\text{train} \n  = \\{(\\mathbf{x}_{c,i}^{+}, y^{+}) \\}_{i=1}^{n/2}\n    \\cup \n    \\{(\\mathbf{x}_{c,i}^{-}, y^{-}) \\}_{i=1}^{n/2}.$\nwith $n$ examples and a concept detection evaluation dataset $\\mathcal{D}_{\\text{concept}}$ of the same structure and harder examples, where $y^{+}$ and $y^{-}$ are binary labels indicating whether the concept $\\mathbf{c}$ is present. We set $n=144$ for our main experiments. \n\\footnote{Using a small training dataset ensures our methods are practical and cost-effective alternatives to SAEs.}\n\n% The $\\mathcal{D}_{\\text{train}}$ contains $n$ pairs of instructions and responses and $\\mathcal{D}_{\\text{concept}}$ contains $m$ such pairs, generated according to the pipeline in \\cref{fig:axbench}. \nWe query \\texttt{gpt-4o-mini-2024-07-18} to generate the data; the prompts used in this pipeline are presented in \\cref{sec:generation-prompts}. Generating the data requires the following steps (note that only the evaluation set includes hard negatives):\n\\begin{enumerate}\n    \\item \\textbf{Genre labelling \\& seed instructions}","chars":20000,"total_chars":134717,"offset":20000,"truncated":true,"next_offset":40000}