The argument from matched comparisons with simpler methods
The argument
That a tool can find something shows little about whether it is powerful; what shows that is how it does against the alternatives on the same task. When SAE-based discovery has been run beside simpler methods on discovery tasks, the simpler methods matched or beat it: on AuditBench SAE tools gave little benefit while prompting-based elicitation gained much; in output-based model diffing a strengthened LLM baseline performed comparably; and the fastest SAE success in the blind audit could have been replicated by embedding search. So the evidence does not show that SAEs are powerful discovery tools.
Does it work?
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 2 supported1 verified
Granting its premises, the argument shows that in hidden-behaviour auditing and output-based model diffing, SAEs have not outperformed simpler methods, which counts substantially against a general claim of power. It does not reach circuit-level discovery of mechanisms, where no matched alternative has been tested, and the AuditBench null in linked claim used a weak SAE tool. It lives on that null, on linked claim and on the authors' own concession in linked claim. Not decisive.
Premises
- VerifiedThe evidence, examined directly, establishes the claim.constitution →On AuditBench, logit-lens and sparse-autoencoder tools provide little benefit across tested target-model types.
- SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →In API-only model diffing, an improved LLM baseline performs comparably to SAE-based methods and surfaces more abstract behavioral differences.
- SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →In Anthropic's 2025 blind auditing game, embedding-based search could have replicated the fastest team's SAE-based document discovery.
Where it is made
Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.
No source is recorded as making it.