Step 19: assistant

text

I've assessed the claim as **supported**, at confidence 0.55 and credence 0.6. The tendency it describes was measured, but in only one study, and the claim's wording goes further than that study tested.

**Rewording.** The old wording, "SAE methods suit low-level model details", could be read as being about a model's internals. The study it comes from means something narrower: features of the model's outputs, such as specific tokens and formatting, in diffing done through the API alone. The claim now reads: "In API-only model diffing, SAE methods suit token- and formatting-level output differences, while LLM methods suit abstract behavioral ones." The Steward of the sibling claim (m2) suggested this.

**What the evidence shows.** The source is Kempf et al. 2026, "Simple LLM Baselines are Competitive for Model Diffing"; I read sections 1 to 5.
- **Measured:** in all three setups (Qwen, Gemma, Gemini), the LLM method's hypotheses were rated more abstract than the SAE method's. The ratings came from three LLM raters.
- **Paired examples fit:** for the Gemini pair, the SAE method reported "pipe and hyphen characters" where the LLM method reported "uses tables".
- **SAE-only finds:** the SAE method alone picked up explicit maths notation in Gemini and end-of-text token artefacts in Gemma.

**What limits it.**
- **Interpretation, not a test:** the split between methods is the authors' suggestion to practitioners. They never tested it directly.
- **One pipeline, judged by LLMs:** only one SAE pipeline was tried, and every score came from LLM judges or raters.
- **Counter-cases in the paper:** some SAE hypotheses were abstract, for example the harmful-advice one in Qwen. Only the LLM method caught the large difference in response length in Qwen (68 against 496 tokens).
- **Counter-cases elsewhere:** other SAE text-analysis work, Jiang et al.'s SAE embeddings and HypotheSAEs, reports SAE pipelines producing semantic hypotheses that beat LLM baselines. I saw these in search results only, not in full. They suggest the low level may come from this particular pipeline's design rather than from SAEs as such.
- **"Suits" is not a win:** the paper's own scoring counts token-level hypotheses as less useful. So the claim describes the level at which each method reports differences; it does not show that SAEs do better at anything.

**On the plan's request (m3)** to check against the sibling answers under the parent question: the auditing comparisons there don't bear on abstraction level. This study is the only direct evidence.

**Changes to the graph:**
- **Structure:** I linked the sibling claim (that the LLM baseline performs comparably and is more abstract) as supporting evidence. It sits under one argument for the claim, which I rated "likely if the premises hold" and not decisive. I placed the one unplaced passage from the paper on that argument.
- **Source records:** I corrected the paper's instance (authors and date added) and recorded a reading of it. The reading is "overstates": the recommendation goes somewhat beyond what the comparison measured.
- **Importance:** set to 0.3, contestation 0.25.
- **Messages:** I answered all three inbox messages and sent the Steward of the parent question a note on the result.

About $0.63 of the $1.20 budget was spent.