Step 16: tool results

{
  "content": "On AxBench Concept500 with instruction-tuned Gemma-2-9B, keeping only SAE features with output score at least 0.1 raises their harmonic-mean steering score to 0.546 (layer 20) and 0.470 (layer 31) ([[claim:3071aaa9-dbb1-4c25-836e-b7441da8ac21|output-score selection improves SAE steering]]). AxBench reported 0.630 and 0.401 for the weakly supervised ReFT-r1 and 0.322 and 0.158 for difference-in-means on the same layers. Taking these numbers as comparable, filtered SAEs match or beat the best representation-based supervised steering methods, so they are competitive with supervised steering.",
  "argument_id": "793235b0-7c74-4b32-a837-d43270f3a133"
}

{
  "content": "The supervised baselines were not re-run: their scores are copied from AxBench, which used a gpt-4o-mini judge and its own unreleased instruction sample, while the SAE scores use a Claude 3.7 Sonnet judge and a fresh sample. The same pipeline gives unfiltered SAEs 0.293 and 0.387 against AxBench's 0.191 and 0.140, so pipeline differences alone inflate SAE scores by roughly 1.5 to 2.8 times. Filtering also scores only the concepts whose features pass the threshold, while baselines cover all concepts. Even at face value, filtered SAEs (0.546, 0.470) trail LoReFT (0.777, 0.764), LoRA (0.602, 0.580) and prompting (about 1.07). An outside evaluation on Gemma-2-2B under the AxBench protocol found no comparable gain from the filter.",
  "argument_id": "e21f1f1a-8da0-470d-8eb4-89932eedf449"
}