In API-only model diffing, SAE methods suit token- and formatting-level output differences, while LLM methods suit abstract behavioral ones.
First assessment
Reworded to specify API-only diffing and output-level token/formatting differences (prior wording ambiguous with model internals). Read Kempf et al. 2026 sections 1-5. Linked the sibling comparison claim as supporting evidence under one named argument; placed the unplaced passage. Assessed supported, confidence 0.55, credence 0.6: measured abstraction gap in 3/3 setups, but one SAE pipeline, LLM-judged metrics, untested recommendation, counterexamples in paper and in other SAE text-analysis work. Importance 0.3, contestation 0.25.
Assessed Supported
verdict confidence 0.55 · credence 0.60
This division of labour comes from one 2026 study, Kempf et al. ("Simple LLM Baselines are Competitive for Model Diffing"). It compared an SAE pipeline, which runs both models' outputs through a separate reader model fitted with a sparse autoencoder, with an LLM pipeline that compares and clusters output pairs. The study used three setups: an emergently misaligned Qwen model, a Gemma model trained to assume a female user, and two Gemini 2.5 Flash Lite revisions. The authors offer the split as a suggestion to practitioners. They did not test it as a hypothesis in its own right. The measured part holds up. The LLM baseline matched the SAE method on accuracy and frequency and gave more abstract hypotheses in all three setups. Abstraction was scored by three LLM autoraters, with LLM hypotheses at roughly 3.8 to 4.3 and SAE hypotheses at 2.4 to 3.5. The paired examples fit the pattern. In the Gemini pair the SAE method reported "pipe and hyphen characters" where the LLM method reported "uses tables". The SAE method also found some token-level differences that the LLM method missed or caught only indirectly: explicit LaTeX notation in Gemini and end-of-text artefacts in Gemma. A token-level KL-divergence method in the appendix likewise surfaced mostly formatting differences. Some of the evidence points the other way, and the claim reaches further than the study. The SAE hypotheses varied widely, and some were abstract: the Qwen harmful-advice hypothesis is one example, and SAE interestingness was slightly higher in that setup. The SAE method also had higher accuracy on Gemini. Only the LLM method caught the very large length difference in Qwen (68 against 496 tokens), which could easily have been measured at the level of tokens. Other work with SAE-based text analysis, such as SAE embeddings for dataset and model-release comparisons (Jiang et al.) and HypotheSAEs, reports SAE pipelines producing semantic hypotheses that do better than LLM baselines. That suggests the low level may come from this pipeline's design (which features it picks and how it labels them) rather than from SAEs as such. The study's own scoring also counts token-level hypotheses as less useful, so "suits" here describes the level at which each method reports, not a demonstrated advantage for SAEs. The claim is best read as a reasonable tendency for current API-only pipelines, resting on one study with one SAE implementation and LLM-judged metrics. It would be settled by comparisons across several SAE and LLM pipelines, with abstraction rated by humans and with planted differences at both levels whose ground truth is known.
Updated claim instance
Instance 9197766a-7dda-4a5f-9390-f0472e593ece (https://arxiv.org/pdf/2602.10371): was stance=affirms, confidence=0.75; set confidence=0.95, speaker="Kempf, Schrodi, Cywiński, Brox, Nanda, Conmy", publication="arXiv", sourceDate="2026-02-10". Read section 5 of the paper: this passage is the origin of the claim and, under the reworded canonical form, states it directly; filled authors and date.
Claim entered the graph