Step 17: assistant

thinking

Calls record_claim_instance

{
  "url": "https://www.alignmentforum.org/posts/pPEeMdgjpjHZWCDFw/white-box-control-at-uk-aisi-update-on-sandbagging",
  "title": "White Box Control at UK AISI - Update on Sandbagging Investigations",
  "stance": "mentions",
  "context": "Prompted sandbagging by Llama-3.3-70B-Instruct on InterCode CTF; off-policy deception probe thresholded at 1% FPR on Alpaca chat. Update 1 reports near-perfect separation on Llama-3.1-8B prompted sandbagging; Update 3 reports diminished recall at 1% FPR on fine-tuned password-locked and weak-model-imitation organisms.",
  "speaker": "Jordan Taylor, Sid Black, Joseph Bloom (UK AISI White Box Control team)",
  "claim_id": "81696185-3711-415e-a7bd-812a5cd6013d",
  "confidence": 0.7,
  "publication": "AI Alignment Forum",
  "source_date": "2025-07-10",
  "source_kind": "blog_post",
  "verbatim_text": "While the overall performance of this decision function is still decent (88% TPR, 2% FPR with max aggregation), we note that these results are sensitive to the method for calibrating the decision boundary, and that our application of this calibration method is relatively unprincipled here."
}

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2512.07810",
  "title": "Auditing Games for Sandbagging",
  "stance": "denies",
  "context": "Blinded auditing game; deception probes elevated on the sandbagger Cedar but produced false positives on a benign distractor model (Baobab) and a benign model (Maple, via calibration), so the blue team could not reliably discriminate sandbaggers from benign models.",
  "speaker": "Jordan Taylor et al. (UK AISI, FAR.AI, Anthropic)",
  "claim_id": "81696185-3711-415e-a7bd-812a5cd6013d",
  "confidence": 0.8,
  "publication": "arXiv",
  "source_date": "2025-12-08",
  "source_kind": "preprint",
  "verbatim_text": "The red team successfully instilled behaviours into their models which induced false positives in Baobab and Oak. These models both have distractor behaviours which the red team hypothesised would cause false positives in probes: over-using double negatives, and repeating multiple choice question options."
}

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2611.14950",
  "title": "Stress-Testing Sandbagging Detection Under Adaptive Attack",
  "stance": "denies",
  "context": "Marker-conditioned GSM8K sandbagging LoRAs on Qwen 2.5-3B/7B and Llama-3.2-3B; logistic-regression probes on mean-pooled activations scored against benign LoRA adapters.",
  "speaker": "Bhoj Rani Soopal, Aarohi Agrawal, Josh Karthikeyan, Edward Lue Chee Lip",
  "claim_id": "81696185-3711-415e-a7bd-812a5cd6013d",
  "publication": "arXiv",
  "source_date": "2026-11-11",
  "source_kind": "preprint",
  "verbatim_text": "A probe trained against the base model flags 99% of benign adapters' examples, so it measures fine-tuning rather than sandbagging. A matched benign adapter fixes this within a training run but not across runs: the probe passes our transfer rule in 1 of 5 folds, or 2 of 5 with three benign recipes."
}