Step 22: tool results

{
  "content": "Linear deception probes trained off-policy on honest/dishonest instruction pairs separate prompted sandbagging from honest answers of the same model almost perfectly on Llama-3.1-8B multiple-choice tasks, and at 97 to 99% AUROC on Llama-3.3-70B agentic CTF, with about 2% false positives when the threshold is set at 1% FPR on Alpaca chat. If these settings are representative of the sandbagging and benign models one would face, probes discriminate sandbaggers with low false-positive rates.",
  "argument_id": "20d26195-b984-46a4-acde-d84749ecba86"
}

{
  "content": "The claim concerns telling sandbagging models apart from benign ones, so the relevant false positives are on benign models, including benign fine-tunes. In a blinded auditing game, harmless distractor habits (double negatives, repeating options) made deception probes fire on benign models, and baseline calibration flagged another benign model. On small LoRA organisms, a probe with base-model negatives flagged 99% of benign adapters' examples, and probes trained against matched benign adapters reached up to 65.5 to 89.5% FPR on held-out training runs. Probes therefore track fine-tuning and surface habits, not sandbagging as such, and do not yet give low false-positive discrimination.",
  "argument_id": "efbbc14d-5f1c-40e8-aa03-f891313face1"
}