Step 8: tool results

read_page

{
  "url": "https://arxiv.org/html/2506.23845v2",
  "offset": 23000,
  "max_chars": 9000
}

{"ok":true,"url":"https://arxiv.org/html/2506.23845v2","text":"a et al. [2025b] study tasks where a large dataset of texts is annotated with a target variable, and the goal is to understand what concepts in the text predict the target.\nFor example, one such dataset consists of news headlines and numerical engagement levels.\nWhile a traditional analysis of such a dataset may be hypothesis-driven (e.g., Robertson et al. [2023] study how negativity affects engagement), here, the task is to extract concepts with no prior specification.\nSuch an approach can surface unknown concepts, which can be used as hypotheses for further study.\nThey (1) train an SAE on dense text embeddings; (2) select SAE features that predict the target; and (3) run autointerpretation to explain the selected features, which become hypotheses (i.e., “ headlines that contain { concept } receive more engagement ”).\nThey find that the resulting hypotheses outperform those generated without an SAE, either by skipping step 1 and selecting features from text embeddings, or by using a different pipeline altogether (like LLM prompting, topic modeling, or n n -grams).\nCompared to these baselines, it produces more statistically significant hypotheses, and raters identify these hypotheses as more helpful.\nMechanistic explanation of LM outputs.\nLindsey et al. [2025] explain how language models generate text that completes a task.\nFor example, prompted to write a rhyming couplet, an LM generates “ He saw a carrot and had to grab it (line 1) / His hunger was like a starving rabbit (line 2).”\nThey ask: what is the internal mechanism through which the LM rhymes the end of line 2 with the end of line 1?\nThey find that, immediately after generating line 1, which ends in “it,” there is an active SAE neuron corresponding to “ words rhyming with ‘it’ ,” as well as a neuron for “ rabbit .”\nThe SAE, therefore, suggests that the LM plans line 2 immediately after generating line 1, rather than improvising a rhyming final token only after generating the first part of line 2.\nThey confirm this with further intervention experiments.\nIn another case, they look at how a model computes “36+59” in natural language.\nThey find active neurons for “ units digit 5 ,” and “ addition problems of ∼ \\sim 40 plus ∼ \\sim 50 ,” which combine to produce “95.”\nThese specific routes of task completion are difficult to forecast, underscoring how this analysis requires discovering unknown concepts.\nWhat explains these positive results?\nIn hypothesis generation, the goal is to find concepts that predict a target variable; in explaining LM behaviors, it is to find concepts that are active when completing a task.\nIn both cases, our hope is to discover concepts that satisfy a property of interest, out of intractably many possibilities.\nSAEs produce a set of concepts that is both tractable and expressive, after which selecting a subset of concepts that are relevant to the task is straightforward.\nAn empirical strength of the SAE is its precise concepts.\nIf the rabbit feature instead activated on all animals, it would be difficult to answer whether the model improvises or plans rhymes.\nAlso note that after identifying concepts of interest, it is possible to computationally validate whether they satisfy the desired property.\nIn particular, once the concept has been identified, it is possible to directly annotate the data for the concept. These annotations can then be used instead of the activations. At this point, the activations are no longer needed.\nIt is easy to evaluate whether a hypothesized headline concept indeed correlates with engagement, or whether a hypothesized LLM addition feature is active during addition. Because of this falsifiability, even if an SAE feature is unreliable (e.g., not all headlines that contain a concept activate the corresponding feature), it is possible to catch these issues downstream.\nIn contrast, unreliability directly harms concept detection and steering.\nPractical examples of concept discovery.\nThis pattern—enumerating concepts and selecting those that satisfy a criterion of interest—has already been useful in several settings beyond the two prior examples.\nTjuatja and Neubig [2025] and Jiang et al. [2025] use SAEs to generate features of LM outputs, and they select features that distinguish different model versions.\nThe former work finds, for example, that OLMo2-13B is better than OLMo2-7B at correctly using archaic spellings in historical texts (e.g., “wood” for “would”).\nMovva et al. [2025a] use SAEs to identify which features of LM responses predict human judgments in RLHF data, revealing unsafe and surprising preferences (e.g., high win-rates for toxic responses, and low win-rates for responses emphasizing environmental sustainability).\nA similar playbook has been promising for biological foundation models [ Simon and Zou, 2025 , Brixi et al., 2025 ] ; for example, Adams et al. [2025] train SAEs on protein LMs and identify amino acid features that predict properties like thermostability, yielding hypotheses for how structure encodes function.\nWhile each of these research questions (“how do LM outputs differ”, “which protein structures are most stable”) could by studied be pre-specifying concepts and testing them, SAEs provide the opportunity to discover unknown concepts.\nTherefore, it is situations where we lack existing theories, or where we wish to expand upon them [ Mullainathan and Rambachan, 2025 ] , where SAEs excel.\nResearch area\nResearch problem (using SAEs to discover unknown concepts)\nML interpretability and explainability\nFinding natural language concepts that can be used to build an inherently interpretable model. [ Rudin, 2019 ]\nFinding natural language concepts that explain a model’s predictions. [ Lakkaraju et al., 2019 ]\nML fairness, bias, auditing, and safety\nIn what ways do LLMs stereotype different demographic groups? [ Lucy and Bamman, 2021 ]\nWhat features are high-stakes LLM-based decision tools using? [ Gaebler et al., 2024 ]\nWhat undesirable behaviors do LLMs exhibit? [ Dai et al., 2025 ]\nSocial and health sciences\nHow has language about immigration changed over time in Congressional speeches? [ Card et al., 2022 ]\nWhat symptoms (recorded in medical records) predict clinical outcomes? [ Huang et al., 2019 ]\nWhat information from court hearings do judges use when making bail decisions? [ Zhang, 2024 ]\nWhat features explain the difference in accuracy between predictive models and theory-grounded models? [ Fudenberg et al., 2022 ]\nAre ML models using illegitimate features (in the context of making a scientific claim)? [ Kapoor and Narayanan, 2023 ]\nTable 3 : Example research problems where SAEs can be applied to discover unknown concepts.\n5 Call to Action: Future Use Cases for SAEs\nHaving conceptualized where SAEs are useful (discovering unknown concepts), we outline research areas where such a capability can be useful. In particular, while initial excitement about SAEs was shared primarily by researchers in mechanistic interpretability [ Sharkey et al., 2025 ] , we believe that clarifying the comparative advantage of SAEs reveals a significantly broader set of uses. The use cases we outline focus on the ability of SAEs to discover unknown concepts.\nBroadly, these use cases fall under two categories: using SAEs to understand language models (in ML fairness, interpretability, explainability, auditing, and safety) and using SAEs to understand the world (e.g., in the social sciences and healthcare). We summarize these potential use cases of SAEs for different research problems in Table 3 .\nML fairness, interpretability, explainability, auditing, and safety.\nEach of these areas aim to understand and build models with desiderata beyond accuracy in mind. Here, we see significant opportunity for SAEs. For example, SAEs can be used to identify natural language concepts that can explain black box model behavior [ Lakkaraju et al., 2019 ] . Then, by identifying the concepts that are used, it is possible to build models that are inherently interpretable [ Rudin, 2019 ] , and that incorporate only features that we want (e.g., that are considered fair, avoid spurious correlations, etc).\nSAEs are particularly valuable for studying models with unstructured text inputs or outputs. For example, whereas existing work documents how specific demographic features affect LLM-based decision-making (e.g., in hiring [ Gaebler et al., 2024 ] ), it is possible to use SAEs to uncover a wider range of input features that may affect LLM-generated decisions. Similarly, while demographic information has been demonstrated to affect LLM outputs in specific ways, it is possible to use SAEs to uncover a wider range of output features that are affected by variations in inputs.\nHealth and social sciences.\nA wide variety of disciplines (e.g., sociology, economics, healthcare) have sought to leverage large text datasets. This has led to prominent work developing and applying methods for “text as data” [ Grimmer, 2010 , Gentzkow et al., 2019 ] . These meth","chars":9000,"total_chars":53077,"offset":23000,"truncated":true,"next_offset":32000}