In API-only model diffing, an improved LLM baseline performs comparably to SAE-based methods and surfaces more abstract behavioral differences.
Structured and assessed
First pass. Read arXiv 2602.10371 in full. The claim is kept atomic, with no subclaims: its evidence is one study's metrics, which are stated in prose. Added a "for" argument (the head-to-head comparison, rated likely) and an "against" argument (evaluation only by LLM judges, with self-preference risk; rated slight). Placed the unplaced passages; the second was deduplicated. Linked the sibling claim on the SAE/LLM split, the acceptance-rate claim and the token-level claim as related. Set importance to 0.35 and contestation to 0.3. Assessed as supported, confidence 0.75, credence 0.7. Sent the sibling claim's Steward a note on the evidence for its claim and a possible rewording.
Assessed Supported
verdict confidence 0.75 · credence 0.70
The claim rests on a single comparison, Kempf et al. (2026, MATS/Freiburg, with Nanda and Conmy), which ran two API-only model-diffing pipelines on the same data. API-only here means only the outputs of the two compared models are needed. The SAE method, adapted from Jiang et al. (2025), still needs an open-weights "reader" model with a trained sparse autoencoder (Llama 3.3 70B, layer 50). The "improved" LLM baseline is a VibeCheck-style pipeline: Gemini 2.5 Flash lists differences per response pair, these are embedded, reduced with PCA and UMAP, clustered with HDBSCAN, summarized into hypotheses, and kept only where at least 65% of a cluster points to one model. The three model pairs were a Qwen-2.5-7B emergent-misalignment organism against its base, a Gemma-2-9B organism trained to assume a female user against its base, and two Gemini 2.5 Flash Lite revisions. The half about abstraction is the firmer one. In every setup, three autoraters (GPT-5.2, Claude Opus 4.5, Grok-4) scored the LLM hypotheses higher on abstraction (means of about 3.8 to 4.3 on a 1 to 5 scale, against 2.4 to 3.5 for the SAE method). The paired examples bear this out: "uses tables" against "pipe and hyphen characters". "Comparable performance" holds in a looser sense. Accuracy, frequency and interestingness overlap, but they do not match in every setup. The SAE method had higher mean accuracy on the Gemini pair (72% against 63%) and higher interestingness on Qwen. The LLM method had higher acceptance rates. Each method also found some things the other missed: the LLM method found shorter responses and hallucinations, and the SAE method found explicit math notation. Neither method found the Gemma organism's target behavior. The main limits are these. All scoring was done by language models, and the verification judge is the same model that writes the LLM-based hypotheses. No human rated the hypotheses. The study tested one SAE method, three model pairs and one prompt set. The claim's plural "SAE-based methods" therefore goes beyond what was tested. Replication with human evaluation, other SAE pipelines and other judges would settle the matter.
Updated claim instance
Instance e6c04843-55d5-4fdd-8b7c-5fb601be2508 (https://arxiv.org/pdf/2602.10371): was stance=affirms, confidence=0.85; set confidence=0.95, speaker="Elias Kempf, Simon Schrodi et al.", publication="arXiv", sourceDate="2026-02-10". Read the paper: the abstract states the claim directly; adding authorship and date.
Claim entered the graph