Step 23: assistant

Calls write_argument

{
  "content": "On Gemma-2-9b-it with Gemma Scope SAEs at layers 17 and 32, features matched to Stack Exchange tags by a calibrated-F1 probe pipeline and filtered by output score reach mean AxBench aggregated ratings of about 0.3 to 0.7 per forum, with roughly half the forums near the LoRA figure of about 0.59 that AxBench reported for this model. The same evaluation code, run on random Neuronpedia-labelled features, reproduces AxBench's low SAE score (about 0.1 to 0.2), so the gain is attributed to how features are selected and labelled. Hence, under that selection, SAE steering scores approach LoRA's.",
  "argument_id": "f1d1ebdd-6557-49f9-bc35-b0b95b948362"
}

Calls write_argument

{
  "content": "The LoRA figure was not re-run: it is AxBench's number for its own Concept500 concepts at layers 20 and 31, judged on its own instruction sample. The SAE scores are for different concepts, broad Stack Exchange tags that the pipeline kept because some feature matched them well (the top 50 pairs per forum), with steering strength tuned per feature by cross-validation. Prompting on these same labels scores about 1.2, above its AxBench figure of about 1.07, which indicates easier concepts; LoRA trained on them would plausibly also score higher. A score near a baseline measured on a harder, different task does not show parity with that baseline on the same task.",
  "argument_id": "5d0b7dbb-149b-4d5f-b61a-d07137400e37"
}