Step 14: tool results

{
  "content": "In a head-to-head comparison over three setups, [[claim:846a4d41-3752-45ed-9210-b4177d75b5e9|an improved LLM baseline matched the SAE method on accuracy and frequency while giving more abstract hypotheses]]. In paired cases the SAE pipeline named the tokens or syntax (pipe and hyphen characters, LaTeX notation, end-of-text artefacts) where the LLM pipeline named the behaviour (uses tables). If these differences reflect how the two kinds of pipeline work rather than this one implementation and these setups, then each suits the level at which it describes differences.",
  "argument_id": "e014b38b-3a98-422a-9d0d-5dd2a12ff4ef"
}

{
  "status": "supported",
  "claim_id": "653e653c-cda0-4e4c-bfe2-de7f99acee34",
  "assessment": "This division of labour comes from one 2026 study, Kempf et al. (\"Simple LLM Baselines are Competitive for Model Diffing\"). It compared an SAE pipeline, which runs both models' outputs through a separate reader model fitted with a sparse autoencoder, with an LLM pipeline that compares and clusters output pairs. The study used three setups: an emergently misaligned Qwen model, a Gemma model trained to assume a female user, and two Gemini 2.5 Flash Lite revisions. The authors offer the split as a suggestion to practitioners. They did not test it as a hypothesis in its own right.\n\nThe measured part holds up. [[claim:846a4d41-3752-45ed-9210-b4177d75b5e9|The LLM baseline matched the SAE method on accuracy and frequency and gave more abstract hypotheses]] in all three setups. Abstraction was scored by three LLM autoraters, with LLM hypotheses at roughly 3.8 to 4.3 and SAE hypotheses at 2.4 to 3.5. The paired examples fit the pattern. In the Gemini pair the SAE method reported \"pipe and hyphen characters\" where the LLM method reported \"uses tables\". The SAE method also found some token-level differences that the LLM method missed or caught only indirectly: explicit LaTeX notation in Gemini and end-of-text artefacts in Gemma. A token-level KL-divergence method in the appendix likewise surfaced mostly formatting differences.\n\nSome of the evidence points the other way, and the claim reaches further than the study. The SAE hypotheses varied widely, and some were abstract: the Qwen harmful-advice hypothesis is one example, and SAE interestingness was slightly higher in that setup. The SAE method also had higher accuracy on Gemini. Only the LLM method caught the very large length difference in Qwen (68 against 496 tokens), which could easily have been measured at the level of tokens. Other work with SAE-based text analysis, such as SAE embeddings for dataset and model-release comparisons (Jiang et al.) and HypotheSAEs, reports SAE pipelines producing semantic hypotheses that do better than LLM baselines. That suggests the low level may come from this pipeline's design (which features it picks and how it labels them) rather than from SAEs as such. The study's own scoring also counts token-level hypotheses as less useful, so \"suits\" here describes the level at which each method reports, not a demonstrated advantage for SAEs.\n\nThe claim is best read as a reasonable tendency for current API-only pipelines, resting on one study with one SAE implementation and LLM-judged metrics. It would be settled by comparisons across several SAE and LLM pipelines, with abstraction rated by humans and with planted differences at both levels whose ground truth is known.",
  "confidence": 0.55,
  "claim_credence": 0.6,
  "marginal_yield": 0.25,
  "reasoning_trace": "Starting model: a claim that each method suits one level would need a method to find differences at its level that the other misses, across more than one implementation. The words were reworded in this pass to say what the source and the parent question mean: output-level token and formatting differences in API-only diffing, not model internals.\n\nRead directly: Kempf et al. 2026 (arXiv 2602.10371), sections 1 to 5: methods, metrics (abstraction and interestingness by three LLM autoraters on a 1 to 5 scale), results for Qwen, Gemma and Gemini, Table 1, the discussion passage that states the claim, and the limitations. Figures for abstraction scores (3.8 to 4.3 against 2.4 to 3.5) and the Gemini 72% against 63% accuracy come from the sibling claim's Steward's reading of the figures and appendix; the Gemini table hypothesis scores of 3.0 against 1.33 are in the main text.\n\nFor: consistent abstraction gap in 3 of 3 setups; SAE-unique detections of LaTeX notation and end-of-text artefacts; the KL appendix pointing the same way.\nAgainst or limiting: one SAE pipeline (adapted from Jiang et al. 2025) whose feature selection and relabelling set its abstraction level; LLM judges for every score; the split is the authors' interpretation, not a tested comparison of the two methods on planted differences at each level; counterexamples inside the paper (abstract SAE harmful-advice hypothesis, length difference only the LLM caught); outside work (Jiang et al. SAE embeddings; Movva et al. HypotheSAEs, arXiv 2502.04382) reports SAE pipelines that yield semantic hypotheses and beat LLM baselines on text-analysis tasks, seen in search abstracts only.\n\nThe tendency is favoured, so the claim is supported, with confidence of 0.55 because the evidence comes from one study and the \"suits\" reading goes beyond what it measured. Credence 0.6 that the division holds as a general tendency for current API-only pipelines."
}