Step 12: tool results

read_arxiv_source

{
  "file": "main.tex",
  "offset": 40000,
  "arxiv_id": "2501.17148",
  "max_chars": 20000
}

{"ok":true,"arxiv_id":"2501.17148","file":"main.tex","kind":"text","text":": We consider three genres: \\textit{text}, \\textit{code}, and \\textit{math}. We prompt the LLM to pick the genre $\\vg_{\\vc}$ for each concept.\\footnote{Genre labelling increases input diversity. For example, inputs related to concepts such as \\emph{programming code contains syntactic errors} should contain code instead of descriptions of coding errors.} We then randomly select seed instructions from our instruction pool which belong to genre $\\vg_{\\vc}$; see \\cref{sec:instruction_pool} for dataset details. We then prompt the LLM to generate responses to these instructions.\\footnote{Each example costs less than \\$$0.00006$.}\n\n    \\item \\textbf{Positive examples}: For each randomly sampled instruction from the instruction pool, we prompt the LLM to generate a response that incorporates the concept $\\vc$. We use the generated concept-conditioned responses concatenated with their instructions (using the LM's chat template) as our positive set.\n\n    \\item \\textbf{Negative examples}: To evaluate the generalisation ability of each method, we independently sample seed instructions from all genres for negatives.\\footnote{We sample instructions based on overall genre distribution: 70\\% from \\textit{text}, 15\\% from \\textit{code}, and 15\\% from \\textit{math}.} These instructions are shared across concepts in order to save generation costs (i.e., $(\\mathbf{x}_{\\mathbf{c}}^{-}, y^{-})_{0}^{n/2}$ is independent of the concept $\\mathbf{c}$).\n    % For an randomly sampled instruction, we prompt the LLM to generate a response by avoiding anything related to the target concept.\n    We sample responses from the LM we plan to steer (not the LLM) without any additional instructions. We use the paired instructions and responses as our negative set.\n    \n    \\item \\textbf{Hard negative examples} \\underline{\\textit{(evaluation only)}}: For each concept, we find contrasting concepts that are semantically related to our concept of interest but which should not activate the concept. We find these by (a) generating a list of phrases that are semantically relevant to our concept, (b) filtering for those which are polysemous, and (c) finding alternative senses of those words which our concept should not activate on. This results in a set of contrast concepts $\\vc_{\\text{contrast}}$, each of which is a specific sense of a polysemous word $\\vw_{\\text{contrast}}$. We then ask the LLM to generate responses incorporating $\\vw_{\\text{contrast}}$ into the sentence where $\\vw_{\\text{contrast}}$ should express the sense related to $\\vc_{\\text{contrast}}$. We use the contrastive responses paired with their instructions as our hard negative set.\n    % \\item \\textbf{Content generation}: We now generate our dataset, in the format of paragraph-sized chunks of input text along with a continuation. The input text, which is always conditioned on a genre, may be totally random or rewritten to introduce a particular concept; the same applies to the continuation. This dataset consists of a few types of text:\n    % \\begin{enumerate}\n    %     \\item \\textbf{Null content}: arbitrary text generated only by conditioning on a genre $G_i$ chosen uniformly at random, with a continuation sampled from the \\textbf{steering model}.\n    %     \\item \\textbf{Hard negatives}: null content rewritten to represent one of the contrast concepts $C_i$ chosen uniformly at random, with a continuation sampled the same as for null content. We hold out some of the $C_i$ for evaluation.\n    %     \\item \\textbf{Concept content}: null content rewritten to represent the concept of interest, with a continuation sampled from the \\textbf{data-generation model} encoding the other concept it was paired with (which is not a hard negative).\n    % \\end{enumerate}\n\\end{enumerate}\n\nThe negative training set is not applicable to all methods (e.g.\\ full finetuning only needs the positive training set for model steering).\n\n% We then describe evaluating methods in details.\n\n% \\paragraph{Interpretability with 1-d linear subspace.} LMs encode information linearly with representations, making linear subspaces surprisingly powerful in terms of decoding latent information or model steering~\\cite{}. In this paper, we focus on evaluating interpretability methods that target 1-d linear subspaces of representations among a fixed model component (e.g., the residual stream output of layer 20 of \\texttt{Gemma-2-2b} model). Nevertheless, \\shortbenchmark{} is a generic framework that is compatible with methods that target other types of subspaces.\n% Building on the theory of causal abstraction for neural networks~\\cite{geiger2024causal}, we conceptualize existing interpretability methods as model abstractions. Intuitively, these abstractions offer simplified proxies by emphasizing a specific aspect of model's functionality. Specifically, we focus on two types of abstractions: concept detection and model steering.\n\n\\subsection{\\concepttag{} Concept detection}\\label{sec:concept-detection}\nA popular LM interpretability method is to train \\textit{probes} \\cite{conneau-etal-2018-cram, hewitt-manning-2019-structural, belinkov-etal-2017-neural} that measure to what extent LM representations encode properties of interest, e.g.~linguistic features. In recent years, the goal of concept detection has broadened to the open-vocabulary setting, with unsupervised methods becoming more common \\cite{bills2023language,huben2024sparse,choi2024automatic}.\n\n% A common automatic evaluation of feature interpretability \\cite{bills2023language,huben2024sparse,choi2024automatic} is to use an LM to propose a label for the feature using dataset exemplars, and then separately ask the LM to predict feature activations conditioned on the label; the correlation between predicted and true presence of the concept ideally indicates how interpretable the feature is.\n\n\\paragraph{Task description.}\n% Since the debut of LM, the examination of human-interpretable knowledge (e.g. syntax and semantics) from latent representations offers intuitive explanations of the internals of the model~\\cite{conneau-etal-2018-cram, hewitt-manning-2019-structural}. The task is to classify hidden representations generated in the model given a set of known class labels (e.g., predicting part-of-speech from internal representations~\\cite{belinkov-etal-2017-neural}).\n% We first unify these preceding evaluations into the notion of \\textbf{concept detection}. \nFormally, given a Transformer-based LM with a hidden dimension size of $d$, we define a concept classifier as a parameterized function $\\Psi_{\\text{Detect}}$ that maps a model representation $h \\in \\R^d$ into a \\emph{binary} label $\\hat{y}$ indicating the relative presence of a concept:\n\\begin{equation}\n    \\Psi_{\\text{Detect}}(h) = \\hat{y} \\in \\R^1 \\label{eqn:concept-detection}\n\\end{equation}\nwhere $\\Psi$ is any function, e.g.~a neural network.\n\n% Supervised probes are one of the most widely used tools for concept detection when labeled corpus is available. \n\n% Recent works in SAEs offer an alternative where concept subspaces are learned over unlabeled text corpus, and learned subspaces can be used as linear probes for detecting concepts~\\cite{}. Given a Transformer-based LM with a hidden dimension size of $d$, these methods offer a parameterzied function $\\Psi_{\\text{Detect}}$ that maps model representations $\\mathbf{h}$ from a $d$-dimensional space into a binarized subspace $\\mathbf{c}$ to represent whether information about a concept can be decoded or not:\n% \\begin{equation}\n%     \\Psi_{\\text{Detect}}(\\mathbf{h}) = \\mathbf{h} \\times \\textbf{u}^\\top_{\\text{Detect}} \\label{eqn:concept-detection}\n% \\end{equation}\n% where the output represents a linear subspace of the model's original representation and is the predicted binarzied variable represents indicating the presence of a concept. Note that additional activation function is needed to turn the output into a classification label.\n\n\\paragraph{Evaluation dataset.} To evaluate a concept classifier, we\n%  first collect internal representations from LM when taking generated sentences as input, and then \nmeasure how accurately it can predict ground-truth labals on the labelled evaluation set from $\\mathcal{D}_{\\text{concept}}$ (see \\cref{sec:concept-dataset}).\n\n\\paragraph{Evaluation metrics.} Since our labels are at the sequence-level, we need to aggregate token-label scores from $\\Psi$ to evaluate it. Given a sequence of token representations \\(\\mathbf{h}^{l} = [\\,h_1^l, h_2^l, \\ldots, h_n^l\\,]\\) with $n$ tokens at layer $l \\in [1, m]$, we max-pool the detection scores to get a sequence-level prediction:\n\\begin{equation}\\label{eq:maxpool-detect}\n    \\hat{y}_{\\text{Detect}} = \\max\\bigl(\\Psi_{\\text{Detect}}(\\mathbf{h}^l)\\bigr)\n\\end{equation}\nWe then normalize $\\hat{y}_{\\text{Detect}}$ between $[0, 1]$ by min-max normalisation over the evaluation dataset for each concept. The predicted score represents how strongly a concept is present in a sequence, which we can compare to the true label.\n\n\\subsection{\\steeringtag{} Model steering}\\label{sec:model-steering}\n% Editing representations by placing interventions in the LM forward computation graph has been shown to be effective in steering the LM generation~\\cite[][\\textit{inter alia}]{subramani-etal-2022-extracting, zou2023representationengineeringtopdownapproach, wu2024reft}.\nRepresentation-based steering has emerged as a potential alternative to existing model-control methods (e.g.~finetuning and prompting) and a practical application of various interpretability methods (see \\cref{sec:related}). Unlike concept detection, model steering assesses \\textit{causal} efficacy in controlling model behaviour.\nPrevious evaluation benchmarks for steering are not general-purpose; they either rely on a limited set of tasks~\\cite{zou2023representationengineeringtopdownapproach, makelov2024sparse,bhalla2025unifyinginterpretabilitycontrolevaluation} or condition generation on a fixed prefix~\\cite{chalnev2024improvingsteeringvectorstargeting}.\nTo the best of our knowledge, we are the first to evaluate model steering methods in the open-vocabulary setting at scale. \n\n% Interventions can be constructed in various ways such as supervised training to elicit targeted behaviors~\\cite{}, training-free methods with vector-space arithmetic~\\cite{} or SAEs where concept directions are learned over unlabeled text corpus~\\cite{}. Given a Transformer-based LM, these methods provides a parameterized function $\\Phi_{\\text{Steer}}$ that modifies representations in-place as:\n% \\begin{equation}\n%     \\Phi_{\\text{Steer}}(\\mathbf{h}) = \\mathbf{h} + \\alpha \\cdot \\textbf{u}_{\\text{Steer}}\n% \\end{equation}\n% where we overwrite model's representations $\\mathbf{h}$ in-place via an activation addition intervention by injecting $\\textbf{u}_{\\Phi}$ with a gating factor $\\alpha$. Note that $\\alpha$ is usually separately chosen as a hyperparameter at inference-time, and is kept as a constant for $\\mathbf{h}$. Note that $\\textbf{u}_{\\text{Steer}}$ and $\\textbf{u}_{\\text{Detect}}$ can have tied weights or not. For SAEs, they have untied weights leaving unshared subspaces between concept detection and model steering for the same concept.\n\n\\paragraph{Task description.} Given a prompt $\\mathbf{x}$, the model's original generation can be written as $\\hat{\\mathbf{y}} = \\textsc{LM}(\\mathbf{x})$. We produce the model's counterfactual generation conditioned on the concept-based intervention $\\Phi_{\\text{Steer}}(\\mathbf{h})$:\n\\begin{equation}\n    \\hat{\\mathbf{y}}_{\\text{Steer}} = \\textsc{LM}\\bigl(\\mathbf{x}, \\mathbf{h} \\leftarrow \\Phi_{\\text{Steer}}(\\mathbf{h})\\bigr)\n\\end{equation}\nwhere $\\mathbf{h} \\leftarrow \\Phi_{\\text{Steer}}(\\mathbf{h})$ is an in-place representation modification. We use the open-source intervention library \\texttt{pyvene} to perform such interventions on PyTorch implementations of models~\\cite{wu-etal-2024-pyvene}.\n\n\\paragraph{Evaluation dataset.} \n% Inspired by Golden Gate Claude \\cite{templeton2024scaling}, \nWe evaluate these steering methods\nin the instruction-following setting, where\n% we prompt LM with instructions while using these methods to steer responses conditioned on a target concept.\nwe sample instructions from \\texttt{Alpaca-Eval} \\cite{alpaca_eval} and prompt the LM to generate a response while intervening on its forward pass in-place using one of the steering methods.\n% \\footnote{Existing finetuning methods including full finetuning and LoRA can all be formulated as representation interventions~\\cite{hetowards} For example, full fine-tuning learns a weight change for every parameter which is equivalent to applying an intervention for each representation that modifies the representations with the weight change.}\n\n\\paragraph{Evaluation metrics.} For the intervened model generation, we evaluate \\(\\hat{\\mathbf{y}}_{\\text{Steer}}\\) based on the \\emph{harmonic mean} of the following scores, each of which the LLM rates using a discrete score of 0, 1, or 2:\n\\begin{enumerate}\n    \\item \\textbf{Concept score} represents how well the concept is incorporated into the response.\n    \\item \\textbf{Instruct score} represents how well the response is related to the instruction.\n    \\item \\textbf{Fluency score} represents how fluent the response is.\n\\end{enumerate}\nSince we compute the harmonic mean, the overall score also ranges from 0 to 2, but heavily penalises poor performance on any of these three subscores. For each concept, we randomly sample 10 instructions from \\texttt{Alpaca-Eval} and sample continuations for each steering factor (see discussion on steering factor in \\cref{sec:result-model-steering}).\nTo ensure a fair comparison, we partition our instructions into two equally sized sets, selecting the best factor from one set and evaluating it on the holdout set.\nOur judge prompts with further discussion can be found in \\cref{sec:evaluation-prompts}.\n% We also validate our LLM-generated overall score with human evaluation in \\cref{sec:human_eval}.\n\n\\section{Methods}\\label{sec:methods}\nIn this section, we describe the interpretability methods we evaluate along with our baseline prompting and finetuning methods. For each method, we label which axes it is evaluated on using \\concepttag{} and \\steeringtag{}. All of our interpretability methods except SAEs are SDLs that learn rank-1 subspaces for targeted concepts.\n\n\\paragraph{Notation.} Given a LM, the hidden representations of dimensionality $d$ for a token sequence of length $n$ in layer $l$ of the LM are represented as \\(\\mathbf{h}^{l} = [\\,h_1^l, h_2^l, \\ldots, h_n^l\\,] \\in \\R^{n \\times d}\\). The set of representations concatenated from all of the training set inputs is denoted as $\\mathbf{H} \\in \\R^{s \\times d}$, where $s = \\sum_{\\mathbf{h}}\\lvert\\mathbf{h}\\rvert$. We denote $\\mathbf{H}^{+}$ as the subset of $\\mathbf{H}$ including only positive training inputs and $\\mathbf{H}^{-}$ for the negative inputs (see \\cref{sec:concept-dataset} for training dataset details). Finally, per-method projection vectors $\\mathbf{w}$ and representations $h_i$ are the same shape: $\\R^{d \\times 1}$.\n\n\\paragraph{\\concepttag{}\\steeringtag{} Difference-in-means (DiffMean).} DiffMean uses the difference between averaged representations from two classes of inputs as a steering vector~\\cite{marks2024geometrytruthemergentlinear}. \n% First, let $\\mathbf{c}^+ = \\mathrm{concat}(\\mathbf{H}^+)$ and $\\mathbf{c}^- = \\mathrm{concat}(\\mathbf{H}^-)$. Then, \nThe projection vector \\(\\mathbf{w}_{\\text{DiffMean}}\\) is defined as:\n\\begin{equation}\n\\mathbf{w}_{\\text{DiffMean}} = \\underbrace{\\frac{1}{|\\mathbf{H}^+|}\\sum_{h^+_i \\in \\mathbf{H}^+}{h^+_i}}_{\\text{mean of positives}} - \\underbrace{\\frac{1}{|\\mathbf{H}^-|}\\sum_{h^-_i \\in \\mathbf{H}^-}{h^-_i}}_{\\text{mean of negatives}}\n\\end{equation}\nWe compute detection scores with the dot product, i.e.~$\\Psi_{\\text{Detect}}^{\\text{DiffMean}}(h_i) = h_i\\cdot\\mathbf{w}_{\\text{DiffMean}}$.\\footnote{Following \\citet{gao2024scalingevaluatingsparseautoencoders}, we normalize $\\mathbf{w}_{\\text{DiffMean}}$ to have unit norm. We apply the same normalization to the learned weights of PCA, LAT, Probe, and ReFT‑r1.} Our steering operation is simple activation addition: $\\Phi_{\\text{Steer}}^{\\text{DiffMean}}(h_i) = h_i + \\alpha \\mathbf{w}_{\\text{DiffMean}}$ where $\\alpha$ is the steering magnitude, which depends on the steering factor and is optimized as a hyperparameter, as described in \\cref{sec:result-model-steering}.\n% Intuitively, this difference highlights how concept-present activations differ from concept-absent ones in the model’s hidden space. For an evaluation sentence \\(\\mathbf{x}\\), the LM generates hidden representations \\(\\mathbf{h}\\) at a layer. We then get the token-level concept score predictions as:\n% \\begin{equation} \\label{diffmean-eq1}\n% \\Psi_{\\text{Detect}}^{\\text{DiffMean}}(\\mathbf{h}) = \\mathbf{h}\\times\\mathbf{w}_{\\text{DiffMean}}\n% \\end{equation}\n\n% We then use max-pooling as described in \\cref{sec:concept-detection} to get sequence-level predictions. For model steering, our intervention function is formulated as:\n% \\[\n% \\Phi_{\\text{Steer}}^{\\text{DiffMean}}(\\mathbf{h}) = \\mathbf{h} + \\alpha \\cdot \\mathbf{w}_{\\text{DiffMean}}\n% \\]\n% where $\\alpha$ is a hyperparameter representing the steering factor. The intervention edits the representations in place during the forward pass as described in \\cref{sec:model-steering}.\n\n\\paragraph{\\concepttag{}\\steeringtag{} Principle component analysis (PCA).} For PCA, we use the first principal component of the positive set of hidden representations as the projection vector.\\footnote{We found no significant difference between using only the positive set vs.~the entire set of hidden representations for both PCA and LAT; see \\cref{sec:abaltions} for ablations.} We first subtract the mean $\\overline{\\mathbf{H}^+}$ from each \\(h^+\\), gathering the centered vectors into a matrix \n\\(\\mathcal{H}\\in \\mathbb{R}^{\\vert\\mathbf{H}^{+}\\rvert\\times d}\\). We then find the top principal component \n\\(\\mathbf{w}_{\\mathrm{PCA}} \\in \\mathbb{R}^{d \\times 1}\\) of \\(\\mathcal{H}\\), i.e.~the unit vector that captures the largest variance along its direction, using \\texttt{sklearn.decomposition.PCA} \\cite{sklearn}. We follow the same detection and steering setup as DiffMean.\n\n\\paragraph{\\concepttag{}\\steeringtag{} Linear artificial tomography (LAT).} \nLAT searches for a single latent direction that can separate positive examples by \nlearning from their pairwise activation differences~\\cite{zou2023representationengineeringtopdownapproach}. Concretely, we create pairwise activation differences \\(\\delta\\) by randomly partitioning \\(\\mathbf{H}\\)\ninto pairs \\((h_i, h_j)\\) (with \\(i \\neq j\\)) and computing\n$\n\\delta = \\frac{h_i-h_j}{\\lVert h_i - h_j \\rVert}\n$,\nwhere the denominator ensures each difference is unit-normalized. We gather all these pairwise differences into a matrix \\(\\Delta\\in\\mathbb{R}^{\\frac{|\\mathbf{H}|}{2}\\times d}\\). We then perform PCA (using \\texttt{sklearn}) on \\(\\Delta\\); then \\(\\mathbf{w}_{\\text{LAT}} \\in \\mathbb{R}^{d \\times 1}\\) is the top principal component of \\(\\Delta\\).  \n% This \\(\\mathbf{w}_{\\text{LAT}}\\) captures the dominant direction of variation among the activation \\textit{differences}.\nWe follow the same detection and steering setup as DiffMean.\n\n\\paragraph{\\concepttag{}\\steeringtag{} Linear probe (Probe).} The linear probe learns to classify tokens as concept-relevant by projecting representations $h_i$ onto a learned direction $\\mathbf{w}_{\\text{Probe}} \\in \\mathbb{R}^{d \\times 1}$ just as in DiffMean. To convert this into a probability, we apply the sigmoid activation, and then minimise binary cross-entropy loss with the true labels:\n\\begin{equation}\n\\min_{\\mathbf{w}_{\\text{Probe}}} \\left\\{ \\frac{1}{\\lvert \\mathbf{h} \\rvert}\\sum_{h_i \\in \\mathbf{h}}\\bigl(\\mathcal{L}_{\\text{BCE}}(y, \\,\\mathrm{Sigmoid}(h_i \\cdot \\mathbf{w}_{\\text{Probe}}))) \\right\\}\n\\end{equation}\nwhere $y$ is the token-level class label indicating whether this token belongs to a pos","chars":20000,"total_chars":134717,"offset":40000,"truncated":true,"next_offset":60000}