Step 25: assistant

Calls update_claim_assessment

{
  "status": "inconclusive",
  "claim_id": "a5665880-9436-4582-a906-e2161fe8e71a",
  "assessment": "The claim comes from a single 2026 preprint by Jørgensen and Hansen, a partial rebuttal of AxBench (Wu et al., 2025), which had found sparse autoencoder (SAE) steering far behind prompting, fine-tuning and simple representation baselines. The rebuttal labels Gemma Scope SAE features on Gemma-2-9b-it using Stack Exchange posts and their tags: each feature is treated as a probe, matched to a tag by a calibrated F1 score, filtered by an output score that proxies steering ability, and the top 50 feature-label pairs per forum are kept. These pairs are then steered and rated with AxBench's three-part, LLM-judged harmonic-mean score.\n\nThe reported numbers are real as far as they go. Read from the paper's main figure, forum means for the selected features range from about 0.3 to 0.7, and about half of the eleven forums sit near the LoRA reference of about 0.59 that AxBench reported for this model, against about 0.1 to 0.2 for randomly chosen Neuronpedia-labelled features under the same evaluation code. Selection and labelling clearly turn SAE features into much better steering vectors than AxBench's setup did.\n\nWhether this means SAE steering approaches LoRA on AxBench is not established. The LoRA figure was quoted, not re-run. It was measured on AxBench's own Concept500 concepts at layers 20 and 31, while the SAE scores are for different concepts, broad forum tags that survived because a feature matched them, at layers 17 and 32, with the steering strength tuned per feature. Prompting on the same tags scores about 1.2, above its AxBench figure of about 1.07, so these concepts appear easier, and a LoRA trained on them would plausibly score higher too. The method is also no longer the unsupervised use of SAEs that AxBench tested: labelled data choose which features are used and what they are said to mean, so the result reverses the benchmark's task, finding concepts for good features rather than steering toward given concepts. Prompting stays roughly twice as strong, as the authors acknowledge. A LoRA run on the same tags, layers, instructions and judge would settle the comparison.",
  "confidence": 0.65,
  "claim_credence": 0.45,
  "marginal_yield": 0.25,
  "reasoning_trace": "Source read in full (arXiv 2605.31183v1, TeX source and Figure 2 image). Setup compared with AxBench (arXiv 2501.17148):\n- Model: Gemma-2-9b-it in both. AxBench's 9B layers are 20 and 31; the rebuttal uses 17 and 32 (layer 32 used for tuning; layer 17 held out). SAEs: Gemma Scope residual, width 131k, lowest l0 (about 10-11); AxBench used the Neuronpedia-annotated, higher-l0 SAEs.\n- Concepts: AxBench uses Concept500 (given concept descriptions). The rebuttal uses Stack Exchange tags from 11 forums, keeping labels with support of at least 50 and the top K=50 feature-label pairs after output-score filtering. Concepts are therefore selected to fit features.\n- Supervision: labelled data (user tags) choose and label features; four pipeline parameters (pi0, K, support threshold, output-score threshold) were tuned by hand iteratively on concept_rating using layer 32 and seven forums. Remaining four forums and layer 17 serve as a hold-out; layer 17 results look similar to layer 32, which argues against overfitting the parameters.\n- Metric: AxBench aggregated rating (harmonic mean of fluency, instruction and concept scores, 0-2). The paper names GPT-4o in Section 3.2 and GPT-4o-mini in the discussion; AxBench used gpt-4o-mini. Steering strength is chosen per feature by 10x4 repeated cross-validation over 20 instructions; AxBench chose one global strength on held-out examples.\n- Baselines: LoRA, LoReFT, ReFT-r1, Prompt, SAE lines are labelled \"Reference (Wu et al., 2025)\" and were not re-run. The paper did rerun an NP-random SAE and prompt baseline: SAE about 0.22 (L17) and 0.09 (L32), prompt about 1.03 and 1.09, consistent with AxBench's 0.165 and about 1.07, which suggests the evaluation code does not by itself inflate scores.\n- Figure 2 readings (estimates to about 0.03): L17 SAE forum means academia 0.62, biology 0.61, chemistry 0.50, history 0.57, literature 0.36, politics 0.59. L32: academia 0.61, biology 0.61, chemistry 0.45, cooking 0.58, cs 0.39, history 0.66, law 0.52, literature 0.38, physics 0.58, politics 0.57, sports 0.28. Unweighted mean about 0.53, about 90% of the LoRA line (about 0.59, consistent with AxBench's 9B LoRA 0.602 at L20 and 0.580 at L31). Prompt on the same labels about 1.1 to 1.3.\n- So \"approaches\" amounts to: in about half the forums the 95% interval for the SAE mean reaches the quoted LoRA figure; on average SAE is about 0.06 below it; several forums are well below.\n- Ablation: features selected by the pipeline but given their Neuronpedia labels also steer better than random ones, and removing the output score lowers performance only modestly, so selection by labelled-data probe is doing most of the work.\n- Consistency with neighbours: AxBench's original finding (SAE steering underperforms prompting and DiffMean on its benchmark) remains verified; this claim does not contradict it, since the setup differs. The sibling claim that output-score-selected SAEs are competitive with supervised methods was judged doubtful on a similar unmatched-baseline problem; the present claim is weaker (LoRA only, \"approaches\") and its numbers are closer, hence inconclusive rather than doubtful.\n- Credence 0.45: the narrow fact that scores are near the quoted LoRA figure holds; the comparative proposition as a reader would take it (SAE steering nearly matches LoRA on the benchmark) depends on an unrun matched LoRA, which the higher prompt scores suggest would also rise. No independent replication was found in this pass."
}