Step 22: tool results

{
  "note": "The paper's own Table 1 sets the filtered SAE scores beside baseline numbers copied from AxBench, which used a different judge and a different instruction sample. The authors' own unfiltered SAE run scores well above AxBench's SAE figure, so part of the apparent gain comes from the pipeline. Even so, the filtered SAEs trail LoReFT, LoRA and prompting at both layers. The 90.7 percent figure matches the ratio to LoRA, not to the best method.",
  "support": "overstates",
  "deployment": "Used as the headline rebuttal of AxBench's finding that SAE steering underperforms simple baselines.",
  "instance_id": "7274c53b-26ee-4570-ba36-88100cd9f921",
  "source_kind": "conference_paper",
  "source_read": true,
  "worth_reading": false
}

{
  "claim_id": "d3311aac-502f-4e69-ab97-53ee2b6de622",
  "reasoning": "Read Section 4.4, Table 1 and the AxBench appendix of arXiv 2505.20063 (TeX source). Baselines were copied from AxBench, not re-run; the judge was Claude 3.7 Sonnet rather than gpt-4o-mini; instructions were re-sampled; filtered scores cover only the retained concepts. Filtered SAEs trail LoReFT, LoRA and prompting even at face value. An outside test (arXiv 2609.35367, Gemma-2-2B) did not reproduce the filter's gains. Assessed doubtful (0.75), credence 0.2. Created the subclaim on the within-pipeline filter effect and two arguments. Importance set to 0.4, contestation 0.45. No dependents to notify.",
  "action_taken": "first_assessment"
}

{
  "answers": [
    {
      "note": "Structured with two arguments and one subclaim, and assessed doubtful.",
      "message": "m1",
      "outcome": "changed"
    },
    {
      "note": "Compared like with like as asked: same model, layers, concepts and metric, but a different judge, instruction sample and concept subset, and baselines copied from AxBench rather than re-run. Filtered SAEs trail LoReFT, LoRA and prompting even at face value. An outside check on Gemma-2-2B did not reproduce the gain. Assessed doubtful, credence 0.2.",
      "message": "m2",
      "outcome": "changed"
    }
  ]
}