Step 19: assistant

thinking

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2501.17148",
  "title": "AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders",
  "stance": "affirms",
  "context": "Introduction of the benchmark paper; the origin of the result. Table of mean overall steering scores: Prompt 0.894, DiffMean 0.239, SAE 0.165, SAE-A 0.157.",
  "speaker": "Zhengxuan Wu, Aryaman Arora, et al. (Stanford)",
  "claim_id": "7c1050dc-7ca9-492e-8d3f-ade8d793de4c",
  "publication": "ICML 2025 (arXiv 2501.17148)",
  "source_date": "2025-01",
  "source_kind": "conference_paper",
  "verbatim_text": "On steering, only ReFT-r1 is competitive with finetuning and prompting baselines, while SAEs fall behind both ReFT-r1 and difference-in-means on both axes."
}

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2505.20063",
  "title": "SAEs Are Good for Steering -- If You Select the Right Features",
  "stance": "mentions",
  "context": "Responds to the AxBench result by changing how SAE features are selected (output-score filtering), not by disputing the original measurement.",
  "speaker": "Dana Arad, Aaron Mueller, Yonatan Belinkov",
  "claim_id": "7c1050dc-7ca9-492e-8d3f-ade8d793de4c",
  "confidence": 0.7,
  "publication": "EMNLP 2025",
  "source_date": "2025-05-26",
  "source_kind": "conference_paper",
  "verbatim_text": "after filtering out features with low output scores, we obtain 2-3x improvements when steering with SAEs, making them competitive with supervised methods."
}

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2605.31183",
  "title": "Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines",
  "stance": "mentions",
  "context": "Partial rebuttal that changes the SAE feature selection and labelling method; it does not report a re-run of the original configuration that contradicts the published numbers.",
  "speaker": "Mikkel Godsk Jørgensen, Lars Kai Hansen",
  "claim_id": "7c1050dc-7ca9-492e-8d3f-ade8d793de4c",
  "confidence": 0.7,
  "source_date": "2026-05-29",
  "source_kind": "preprint",
  "verbatim_text": "This work serves as a partial rebuttal for Sparse Autoencoders and suggests that the results of Wu et al. (2025) did not do them full justice. We find that Sparse Autoencoders can, in fact, perform close to on par with the reference LoRA performance on the AxBench benchmark, when features are selected and labelled with our supervised pipeline."
}