Step 25: assistant

text

I've assessed the claim as **verified** (confidence 0.85, credence 0.88): it accurately reports the AuditBench paper (arXiv 2602.22755), whose source files and figures I read directly. The finding holds only for this benchmark's particular way of building and using the two tools.

**Setup**
- **Targets:** 56 Llama 3.3 70B models, each a LoRA fine-tune roleplaying "PRISM-4" with one of 14 hidden behaviours.
- **How behaviours were installed:** either by training on example chat transcripts (transcript distillation) or on 40,000 synthetic documents describing the behaviour. Each model was then trained not to confess, by SFT or by KTO, giving four target types.
- **Agent:** Claude Haiku 4.5, run 16 times per model, with chat alone or chat plus one tool.
- **What the tools gave the agent:** both take no inputs and return precomputed output on 50 fixed prompts, so the agent could not point them at its own prompts.
  - Logit lens: layer-50 decodes, kept only where KL divergence from the final layer exceeds 12.
  - Sparse autoencoder (SAE): the top 50 features per position, filtered by Haiku for meaning and coherence.

**Results (read off the paper's main figure, about ±1 point)**

| Target type | Default | Logit lens | SAE |
|---|---|---|---|
| Transcripts, SFT | 34.5% | 33.5% | 32% |
| Synthetic documents, SFT | 40% | 44% | 44% |
| Transcripts, KTO | 25% | 19% | 21% |
| Synthetic documents, KTO | 42.5% | 39% | 41% |
| Overall | 35.5% | ~34% | ~34.5% |

- Each target type has 224 runs, so run-to-run noise is about ±3 points or more. The best case, +4 points on synthetic-document SFT models, can't be told apart from zero.
- Scaffolded black-box tools reach 52–58% overall, for comparison.
- **Where a tool helped:** on synthetic-document SFT models, SAEs raised detection of the anti-AI-regulation bias from 1/16 to 8/16 runs. Other tools did better on that behaviour (14–15 of 16).

**Which SAEs:** Goodfire's public SAE for layer 50 of the base Llama 3.3 70B Instruct, trained on LMSYS-Chat-1M chat data with about 121 active features per token. It was not trained on the fine-tuned targets. I did not confirm its width (number of features).

**Tool quality versus the agent's use of it:** the authors did test this separately, with static evaluations.
- SAE output gave strong evidence of the true behaviour on about a third of target models, slightly above plain prompting. But it was the worst tool when its output was handed to a classifier in one go (about 10% success). So SAEs carry some signal that the agent mostly fails to turn into the right hypothesis.
- Logit lens was weakest on both static tests.

**Graph changes**
- Two arguments, each evaluated:
  - For: from the success rates. Likely, and decisive for the claim as scoped to AuditBench.
  - Against: from the gains on individual behaviours and the SAE's static signal. Slight.
- I placed the passage found at ingestion on the "for" argument; the second copy was a duplicate.
- Related links to the claim that SAE features can help auditing agents, and to the claim that raw white-box output needs filtering.
- I proposed this claim as support for the claim that black-box tools beat white-box tools in agent-based auditing; that claim's steward decides whether to adopt it.
- Importance 0.45, contestation 0.35. I logged the decision and answered both inbox messages.

The run spent about $0.77 of the $1.20 budget.