Step 17: assistant

thinking

Calls update_claim_assessment

{
  "status": "inconclusive",
  "claim_id": "4dbe502e-402c-4ae9-8c6f-57a252a0d324",
  "assessment": "The claim is the thesis of a widely cited 2025 position paper (Peng, Movva, Kleinberg, Pierson and Garg, ICML 2026), which argues that sparse autoencoders (SAEs), though they lose to simple baselines when acting on concepts specified in advance, are powerful tools for discovering concepts no one specified. The evidence shows that SAEs and related sparse dictionaries can surface such concepts. It does not yet show that they are powerful at it in the sense the paper intends, a comparative advantage over other methods, and where that has been tested the results are mostly null.\n\n**The case for.** A sparse dictionary lists many interpretable features without being told what to look for, and a researcher can select those that meet a criterion and check them by other means. The paper's strongest example inside a language model is Anthropic's 2025 attribution-graph work, in which transcoder features showed [[claim:5c397b68-66c7-4e8e-b2b4-22d77ac1ec06|a model choosing its rhyme word before writing the line]], a finding then confirmed by interventions whose effects show in the model's output. For mechanisms at this level of detail no simpler method has produced comparable accounts, which is the best support for the claim. In a hidden-objective audit, adding SAE tools [[claim:391dac24-440f-438b-8cff-8933dfac302c|raised an automated agent's success from near zero to 10 to 20%]].\n\n**What the cited cases show and do not show.** The paper reports no experiment of its own. Its main measured win, hypothesis generation, fits SAEs to text embeddings of a dataset and finds concepts that predict an outcome; that is discovery about a corpus, not about concepts a language model represents, and its baselines were built by overlapping authors. The rhyme-planning result used transcoders rather than SAEs (the paper groups them together), was not compared with any alternative, and confirms a broader idea, that models carry information about tokens yet to come, which earlier probing work had already shown. The model-diffing papers it cites compare model outputs, a task on which a later study found [[claim:846a4d41-3752-45ed-9210-b4177d75b5e9|a strengthened language-model baseline performing about as well]]. The audit gain depended on the dictionary having been trained on the documents that described the hidden objective, and vanished without them.\n\n**Matched comparisons.** On AuditBench, 56 models each trained with an unstated hidden behaviour, [[claim:461a798c-19b2-4b7a-9f26-aa1dfed627fb|an SAE tool gave no benefit]] while prompting-based elicitation tools raised success from about 36% to above 50%; that SAE tool was weak (off-the-shelf dictionaries, a static readout), so the null does not rule out a stronger setup. In the blind audit where an SAE team was fastest, its authors note that [[claim:5393957c-bef5-4262-a152-84c5fe63ece2|embedding search could have replicated the discovery]], and a second SAE team was slower than the team without SAEs.\n\n**Where it stands.** Read as \"SAEs can surface concepts researchers had not specified, worth testing as hypotheses,\" the claim is supported. Read as the paper means it, a distinctive power to discover what other methods miss, no matched study yet shows it outside one audit whose dictionary was trained on the answer. What would settle it: a discovery task, on properties that arose in ordinary training rather than being planted, in which a dictionary and tuned simpler methods are run by the same blind investigators and graded by the same rule.",
  "confidence": 0.6,
  "marginal_yield": 0.4,
  "reasoning_trace": "Starting model. \"Powerful tools for discovering unknown concepts represented in language models\" was read as a comparative and evaluative claim: that SAEs find concepts in a model's representations that researchers had not specified, and do so markedly better than available alternatives. Before reading, the expectation was that SAEs would plainly be able to produce candidate concepts (they enumerate features), so the live question is how often those candidates are real model-internal concepts that alternatives would have missed.\n\nRead directly. The position paper (arXiv 2506.23845 v2), sections 1 to 5. Its evidence: (1) HypotheSAEs (Movva et al. 2025b), SAEs on text embeddings selecting features that predict a target, outperforming embedding-feature selection, LLM prompting, topic models and n-grams, with baselines by overlapping authors; this is discovery about datasets, outside the claim's \"represented in language models\" except loosely (the embedding model's representations). (2) Lindsey et al. 2025, rhyme planning and addition heuristics, found with cross-layer transcoders and validated by interventions; no comparison with another method. (3) Tjuatja and Neubig, Jiang et al.: SAE features of model outputs distinguishing model versions; later work (arXiv 2602.10371) found an improved LLM baseline comparable. (4) Movva et al. 2025a: SAE features of RLHF responses predicting preferences, again about data. (5) Protein LMs. The paper defines \"unknown\" as a concept the researcher could not specify in advance, even if familiar once named, which is a reasonable and fairly permissive bar.\n\nChecked against the mandate's three questions for each case. Unknown beforehand: rhyme planning partly (forward planning in LMs had prior probing evidence, the specific mechanism was new); hidden-objective audit artificially (planted); hypothesis-generation concepts were unknown as predictors. Validated independently of the SAE: rhyme planning yes (output effects of interventions); hypothesis generation yes (annotate and regress); audit yes (ground truth known). Simpler method tried on the same problem: rhyme planning no; hypothesis generation yes, by overlapping authors, SAE ahead; model diffing yes, by a third party, parity; audits yes, null on AuditBench, conditional positive in the 2025 tool ablation, replicable by embedding search in the blind game.\n\nWeighed with the sibling question's assessment (when dictionary methods outperform alternatives), which reads the audit and diffing studies in full and leans toward no general advantage and one conditional advantage tied to the dictionary's training data. That reading was adopted here; it was not re-derived. Not read in this pass: the Biology paper itself, the OpenAI emergent-misalignment persona-feature work (an SAE model-diffing finding often cited as a discovery), and Google DeepMind's note deprioritizing SAEs, which concerned a known-concept probing task.\n\nWhy inconclusive rather than supported or doubtful. Supported would fit the weak reading (can discover); doubtful would fit the strong comparative reading if every relevant comparison were negative. But the most impressive model-internal discoveries (circuit-level mechanisms) have no simpler competitor that has been shown to do as well, and the negative comparisons concern tasks (hidden-behaviour auditing, output diffing) where behavioural elicitation is unusually strong. The evidence therefore does not come down on the claim as worded. Credence omitted: \"powerful\" is evaluative and the two readings pull apart.\n\nWhat would change it: a matched blind discovery comparison on non-planted properties; an AuditBench-style rerun with dictionaries fit to the targets; a comparison of attribution graphs with non-dictionary circuit methods on finding unknown mechanisms."
}