Step 19: assistant

text

Fix instance metadata, record reading and provenance edge, then build arguments.

Calls update_claim_instance

{
  "speaker": "Mikkel Godsk Jørgensen and Lars Kai Hansen",
  "claim_id": "a5665880-9436-4582-a906-e2161fe8e71a",
  "reasoning": "Filled authors and date from the paper's first page; the abstract passage states the claim directly.",
  "confidence": 0.95,
  "instance_id": "6912e235-b510-4be2-9a4c-5eb12f2c7841",
  "publication": "arXiv preprint 2605.31183",
  "source_date": "2026-05-29"
}

Calls provenance_record_reading

{
  "note": "The paper's own figure shows pipeline-selected SAE features scoring roughly 0.3 to 0.7 in AxBench aggregated rating across eleven Stack Exchange label sets on Gemma-2-9b-it (layers 17 and 32), with about half of the forum means near the LoRA reference of about 0.59. That reference is quoted from the AxBench paper, not re-run, and was measured on AxBench's own concepts at other layers. The concepts here are Stack Exchange tags chosen because a feature matched them, the steering strength is tuned per feature, and prompting on the same labels scores well above its AxBench figure, suggesting easier concepts. The data bear \"on par with the quoted LoRA figure for several label sets\"; the like-for-like comparison the wording implies was not run.",
  "support": "overstates",
  "deployment": "The headline result of a partial rebuttal of AxBench's finding that SAE steering underperforms simple baselines.",
  "instance_id": "6912e235-b510-4be2-9a4c-5eb12f2c7841",
  "source_kind": "preprint",
  "source_read": true,
  "worth_reading": true,
  "worth_reading_reason": "Figure 2 and the method section show exactly how far the setup departs from AxBench; a matched LoRA run on the same labels would settle the claim."
}

Calls provenance_record_edge

{
  "evidence": "Section 4 Baseline: \"As baselines, we use those provided in Wu et al. (2025)\"; Figure 2 dashed lines labelled \"Reference (Wu et al., 2025)\" including LoRA.",
  "fidelity": "faithful",
  "reasoning": "The LoRA comparison figure is taken from AxBench's reported results rather than re-run; it is quoted accurately, but it was measured on different concepts and layers.",
  "confidence": 0.85,
  "target_read": true,
  "relation_type": "cites_as_evidence",
  "to_source_url": "https://arxiv.org/abs/2501.17148",
  "to_source_kind": "preprint",
  "to_source_title": "AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders",
  "from_instance_id": "6912e235-b510-4be2-9a4c-5eb12f2c7841"
}