In API-only model diffing, an improved LLM baseline performs comparably to SAE-based methods and surfaces more abstract behavioral differences.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim rests on a single comparison, Kempf et al. (2026, MATS/Freiburg, with Nanda and Conmy), which ran two API-only model-diffing pipelines on the same data. API-only here means only the outputs of the two compared models are needed. The SAE method, adapted from Jiang et al. (2025), still needs an open-weights "reader" model with a trained sparse autoencoder (Llama 3.3 70B, layer 50). The "improved" LLM baseline is a VibeCheck-style pipeline: Gemini 2.5 Flash lists differences per response pair, these are embedded, reduced with PCA and UMAP, clustered with HDBSCAN, summarized into hypotheses, and kept only where at least 65% of a cluster points to one model. The three model pairs were a Qwen-2.5-7B emergent-misalignment organism against its base, a Gemma-2-9B organism trained to assume a female user against its base, and two Gemini 2.5 Flash Lite revisions.
The half about abstraction is the firmer one. In every setup, three autoraters (GPT-5.2, Claude Opus 4.5, Grok-4) scored the LLM hypotheses higher on abstraction (means of about 3.8 to 4.3 on a 1 to 5 scale, against 2.4 to 3.5 for the SAE method). The paired examples bear this out: "uses tables" against "pipe and hyphen characters". "Comparable performance" holds in a looser sense. Accuracy, frequency and interestingness overlap, but they do not match in every setup. The SAE method had higher mean accuracy on the Gemini pair (72% against 63%) and higher interestingness on Qwen. The LLM method had higher acceptance rates. Each method also found some things the other missed: the LLM method found shorter responses and hallucinations, and the SAE method found explicit math notation. Neither method found the Gemma organism's target behavior.
The main limits are these. All scoring was done by language models, and the verification judge is the same model that writes the LLM-based hypotheses. No human rated the hypotheses. The study tested one SAE method, three model pairs and one prompt set. The claim's plural "SAE-based methods" therefore goes beyond what was tested. Replication with human evaluation, other SAE pipelines and other judges would settle the matter.
Full reasoning: the evidence and decisions behind this verdict
Source read in full (main text, Appendices A–C): arxiv.org/pdf/2602.10371.
Setup: 1,000 WildChat prompts for generation, 500 held out for evaluation, one completion per model per prompt (max 1,024 tokens). Judge J(h,t)∈{−1,0,1} by Gemini 2.5 Flash with random order swap and no direction hint; frequency = share where the behaviour applies, accuracy = share of applicable cases with the correct direction; acceptance rate = judge consistency on generation data. Interestingness and abstraction: 1–5 autoraters (GPT-5.2, Claude Opus 4.5, Grok-4) averaged, calibrated with examples at 1, 3 and 5; 95% t-intervals over hypotheses.
Per-setup means (Appendix C): accuracy LLM/SAE Qwen 90.8/87.6 (n=68/38), Gemma 91.7/88.0 (n=86/15), Gemini 63.4/72.3 (n=23/29); frequency 45.5/42.3, 48.4/61.4, 48.5/39.9; interestingness 1.8–2.7 for both, with SAE higher on Qwen (2.71 vs 2.24); abstraction LLM 3.8–4.3, SAE 2.4–3.5 (Gemma 4.17 vs 2.71).
Weighing: the abstraction gap is consistent across all three setups and three raters and matches the qualitative pairs (Table 1), so it is well supported for these pipelines. It is partly built into the rubric, which scores any hypothesis naming tokens as 1–2, and SAE hypotheses name tokens because their features are token-level. That is the phenomenon itself rather than an artefact. "Comparably" is a fair summary of overlapping means with small n, but the SAE method won on Gemini accuracy, the setup closest to real-world use. The judge is the same model as the LLM pipeline's extractor and summarizer, a possible self-preference bias favouring LLM hypotheses on acceptance and accuracy. SAE hypotheses were also relabelled and rephrased by Gemini, which partly evens this out. The authors include a co-author of the SAE method (Nanda), so there is no apparent motive to understate the SAE method, and the authors also improved it (adding prompt context and generating 40 hypotheses). Credence 0.7 reflects a solid result for the tested configurations, discounted for the general wording ("SAE-based methods"), evaluation by LLMs only, and a single study.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Its one source has been traced to an origin, or to one another.
Our results show that an improved LLM-based baseline performs comparably to the SAE-based method while typi- cally surfacing more abstract behavioral differences.
Abstract and intro, summarising a comparison of API-only model diffing methods across three setups (Qwen emergent misalignment, Gemma hidden user-gender assumption, Gemini 2.5 Flash Lite revisions).
The source's own evidence bears what it asserts. The paper's own figures bear the statement for the settings tested: overlapping accuracy, frequency and interestingness, and a consistent abstraction gap. All scoring is by LLM judges, and the comparison covers one SAE method, three model pairs and one prompt set; the paper hedges with "typically" and calls the picture nuanced, and in the Gemini comparison the SAE method had higher mean accuracy (72.3% against 63.4%). Worth reading closely: It is the only evidence for the claim; Appendix C holds the per-setup distributions behind "comparable".
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="846a4d41-3752-45ed-9210-b4177d75b5e9"><a href="https://minerval.ai/claims/846a4d41-3752-45ed-9210-b4177d75b5e9">In API-only model diffing, an improved LLM baseline performs comparably to SAE-based methods and surfaces more abstract behavioral differences.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/846a4d41-3752-45ed-9210-b4177d75b5e9" title="Minerval: In API-only model diffing, an improved LLM baseline performs comparably to SAE-based meth…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/846a4d41-3752-45ed-9210-b4177d75b5e9)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.