When do sparse autoencoders and related dictionary-learning methods outperform other methods for discovering language-model properties?
Noted quote check artifact
Two readings returned quote_check not_found: the AuditBench passage and the hidden-objective audit passage. Both instances were recorded against the arXiv abs-URL source records, which are separate from the pdf-URL records that hold the full text. The AuditBench passage was read verbatim in the paper's TeX results section in this pass; the audit-paper passage is recorded verbatim on an existing instance of the pdf record (on the claim that embedding search could have replicated the fastest team's discovery) but its section was not read in this pass, as the reading's note says. The instances were left in place, since re-recording them on the pdf records would split the origin and edge records across two source ids; an issue was raised for the maintainers. A later pass with budget should either re-point these instances once the source records are merged or re-record them against the pdf records, and should read the audit paper's interpretability-value section.
Structured and assessed
First pass. Read directly: the hidden-objective audit paper (team table, rules, team narratives, limitations, contributions), Anthropic's automated-auditing report (tools, results paragraph, SAE-without-synthetic-data appendix), and AuditBench (benchmark, investigator, results text and main figure). Commissioned one strong-tier research run ($2.32 spent of $2.50) for the model-diffing and secondary studies; it read two papers in main text and reported six more from excerpts, which the assessment labels as leads. Attached ten answers: two new (the automated-audit tool-ablation result; the dependence of SAE usefulness on the dictionary's training data including objective-describing documents) and eight existing. Standing leaning, confidence 0.6, favouring four mutually compatible answers: no general advantage shown, one conditional advantage measured. Did not favour the general black-box-beats-white-box answer because the automated audit contradicts it in its setting. Treated text-corpus hypothesis generation as outside the question as worded; did not attach those claims. Recorded six answering instances, four readings, one provenance edge, one authorship relation, two source interests and a material source map. Importance 0.45, contestation 0.65. Filed request 3480bc56 ($2.50) for full-text reads of the excerpt-level studies, a citation walk for transcoder and attribution-graph comparisons, and a target-fit-dictionary test. Wording left unchanged: it is neutral and presupposes only that discovery comparisons exist, which they do; no presupposition node was needed. No dependents to notify; the sibling question's Steward was answered through its message. Unverified points flagged in the trace: AuditBench's dictionaries being fit to the base model is an inference from the tool description; per-toolset win rates in the automated-auditing report were not read from figures; the audit paper's interpretability-value section was not read in this pass.
Set instance order
Instances ranked first, in order: 964da852-62f6-434a-8df4-741f3f2854f2, 1d85debb-2df7-475e-8619-3561374a9d1d, 1562b025-7f23-4fe3-bb7b-aa078231cd09, e4906504-ca23-4f63-a3d6-a486f872696c, d3d3e3d0-7e42-41b7-93f3-72e46d51ca65, d1d6e900-6738-4e71-9b32-ab24a41d99f1. The isolating ablation and its condition first, then the largest controlled null, the original audit's concession, the model-diffing comparison, and last the position paper, which reports no experiment.
Contribution accepted
The proposal is a well-formed open question in neutral wording: under what conditions sparse autoencoders and related dictionary-learning methods (crosscoders, transcoders) outperform non-SAE baselines when the investigator does not know in advance what to look for, as in hidden-behavior auditing and model diffing. It names the domain, the comparison and the outcome, and it presupposes no answer. It is not a claim in disguise: it asks under what conditions, so it is answered by conditions, not affirmed or denied. It concerns a model's properties, not a private person. It is also something anyone reasoning about interpretability would need to ask, because the cited results (an auditing game, an implanted-behavior benchmark, two model-diffing studies and a hypothesis-generation paper) point different ways. The accompanying note lists candidate moderators and separates the supervised-task case, which suits a different question. This review judged the wording, not the case. I did not open the cited arXiv papers, and whether each reports what the note says is the Steward's work after admission. Whether the cited studies are comparable, and which moderators matter, are assessment matters, not grounds for rejection. Matching to any existing node is the Matcher's call.
§2§7a§17
Claim entered the graph
Contribution submitted
The graph holds no node on whether sparse autoencoders and related methods (crosscoders, transcoders) add value over other tools when the investigator does not know in advance what to look for. Published results appear to conflict: a hidden-objective auditing game reports teams succeeding with SAEs among their tools (arXiv 2503.10965); a 2026 benchmark of models with implanted behaviors reports that an auditing agent does best with black-box tools (arXiv 2602.22755); a 2026 model-diffing study reports an LLM-based baseline comparable to an SAE-based method (arXiv 2602.10371) while a crosscoder paper reports beating SAE baselines and matching a non-SAE one (arXiv 2603.04426); and an SAE-based hypothesis-generation method reports beating topic-model and LLM baselines on text data (arXiv 2502.04382). Candidate moderators: strength of the non-SAE baseline, agent scaffolding, narrow versus broad fine-tunes, and whether the comparison isolates the SAE's contribution. Supervised tasks with labels are proposed as a separate question because different evidence bears on them.