Step 7: assistant

thinking

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2512.07810",
  "title": "Auditing Games for Sandbagging",
  "stance": "affirms",
  "context": "Red-team/blue-team auditing game over five fine-tuned model organisms; compares detection methods with training-based elicitation.",
  "speaker": "Jordan Taylor et al.",
  "claim_id": "b1c228fb-b328-47b1-ab7c-06a68b60192f",
  "source_date": "2025-12-08",
  "source_kind": "preprint",
  "verbatim_text": "We found that the blue team could not reliably discriminate sandbaggers from benign models. ... training-based elicitation consistently elicited full performance from the sandbagging models, using only a single correct demonstration of the evaluation task. However the performance of benign models was sometimes also raised, so relying on elicitation as a detection strategy was prone to false-positives. In the short-term, we recommend developers remove potential sandbagging using on-distribution training for elicitation."
}

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2604.22082",
  "title": "Removing Sandbagging in LLMs by Training with Weak Supervision",
  "stance": "mentions",
  "context": "Supports training as a removal method but with weak rather than correct demonstrations, and names failure conditions; does not compare with detection.",
  "speaker": "Emil Ryd et al.",
  "claim_id": "b1c228fb-b328-47b1-ab7c-06a68b60192f",
  "confidence": 0.7,
  "source_date": "2026-04-23",
  "source_kind": "preprint",
  "verbatim_text": "training with weak supervision can reliably elicit sandbagging models when supervised fine-tuning (SFT) and reinforcement learning (RL) are combined ... SFT without RL fails to elicit full performance when the supervisor is much weaker than the untrusted model. Critically, this relies on training being indistinguishable from deployment; when models can distinguish between training and deployment, they can perform well during training while continuing to sandbag afterward."
}

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2610.13928",
  "title": "AI Models Can Provably Hide Arbitrary Capabilities",
  "stance": "mentions",
  "context": "Cryptographic backdoor construction hiding a capability behind a secret trigger; shows a deliberately engineered organism resists fine-tuning-based elicitation. Bears on the generality of removal, not on the removal-versus-detection comparison in natural or conventionally trained organisms.",
  "speaker": "Andis Draguns et al.",
  "claim_id": "b1c228fb-b328-47b1-ab7c-06a68b60192f",
  "confidence": 0.6,
  "source_date": "2026-10-09",
  "source_kind": "preprint",
  "verbatim_text": "We implement our proof of concept constructions in PyTorch and evaluate elicitation through fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training ... Our results demonstrate resistance to these methods and, together with our cryptographic guarantees, reveal a limit of elicitation-based capability audits."
}