Linear probes on model activations distinguish sandbagging from benign language models with low false-positive rates.
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
The claim holds in the easiest setting and fails where it matters most. Linear deception probes do separate a model that has been prompted to sandbag from the same model answering honestly: UK AISI researchers report near-perfect separation on Llama-3.1-8B multiple-choice tasks, and 97 to 99% AUROC on Llama-3.3-70B playing an agentic capture-the-flag task, with about 88% detection at 2% false positives when the threshold is calibrated on ordinary chat. Even there the authors note the result is sensitive to how the threshold is set, and the margin between classes shrank markedly at 70B.
Distinguishing a sandbagging model from a benign model is a harder problem, because benign models differ from one another for reasons unrelated to sandbagging. In the UK AISI auditing game (Taylor et al., 2025), probes were the most promising white-box method and did flag one fine-tuned sandbagger (a Llama-3.3-70B derivative), but benign models given harmless "distractor" habits such as double negatives triggered spurious probe activations, a benign model looked suspicious under a small calibration baseline, and probes on a second sandbagger failed outright. No model-level false-positive rate is reported; the qualitative outcome is that probes could not be relied on to clear benign models. A 2026 stress test on 3B to 7B LoRA organisms that sandbag on GSM8K (Soopal et al.) found a probe with base-model negatives flagged 99% of benign adapters' examples, and probes trained against matched benign adapters, perfect within one training run, reached held-out false-positive rates of 65.5% to 89.5% on other runs, passing a predeclared transfer test in only 1 or 2 of 5 folds.
Current evidence therefore does not support low false-positive discrimination of sandbagging models from benign models in general. All the evidence comes from small to mid-size open models and artificial password- or trigger-locked organisms; better probe designs (matched controls, follow-up-question probing, attention probes) could change the picture, and that is what would revise this verdict.
Full reasoning: the evidence and decisions behind this verdict
Reading of the claim: "distinguish sandbagging from benign language models" is read as discriminating sandbagging models (or sandbagging behaviour) from benign models, where false positives are benign models or benign examples flagged.
1. UK AISI White Box Control update (Alignment Forum, July 2025). Update 1: sandbagging induced by Anthropic's "very weak model imitation" prompt in Llama-3.1-8B-Instruct on MMLU, ARC, WMDP, GSM8k etc.; probe is Apollo-style logistic regression trained off-policy on honest/dishonest persona statements. Benign comparison: same model without the prompt. AUC "rarely perfect, though often very close"; no FPR given. Update 2: Llama-3.3-70B-Instruct prompted to perform badly on InterCode CTF (only 17 sandbagging transcripts); probe AUROC 97-99%; threshold set at 1% FPR on Alpaca chat gives 88% TPR, 2% FPR on benign CTF transcripts; authors call calibration "relatively unprincipled"; on multiple choice the 70B separation margin is much smaller and the zero threshold no longer separates classes. Update 3: four LoRA-fine-tuned organisms (backdoored vs password-locked, prompt-distilled vs weak-model imitation of Llama-3.2-1B); recall at 1% FPR "significantly diminished" except for the prompt-distilled backdoor. No benign fine-tunes were scored. This source is mixed: favourable for prompted, same-model comparisons; unfavourable for fine-tuned organisms.
2. Taylor et al., Auditing Games for Sandbagging (arXiv 2512.07810). Sandbagging induced by password-locking via fine-tuning; Cedar trained to imitate a weaker model's logits, Oak sandbags only on InterCode CTF; both fine-tuned from Llama-3.3-70B-Instruct; tasks include GPQA Diamond, BBH, tic-tac-toe, InterCode CTF. Benign models: Baobab, Maple, Yew decoys (some with distractor habits) plus the Llama-3.1-8B-Instruct reference. Deception probes (Azaria-Mitchell true/false statements with honest/dishonest persona instructions) were robustly elevated on Cedar, but double-negative and option-repeating habits caused false positives on Baobab and Oak; Maple was falsely suspected because of calibration on a small baseline (fixed post hoc with Alpaca); Oak's probes did not separate sandbagging from unlocked. No numeric FPR is reported for probes against benign models; phase one ended with all three benign models flagged (mostly from black-box methods).
3. Soopal et al., Stress-Testing Sandbagging Detection Under Adaptive Attack (arXiv 2611.14950, Nov 2026). Rank-16 LoRA adapters make Qwen 2.5-3B/7B and Llama-3.2-3B fail GSM8K when an audit marker is present. Probe: logistic regression on mean-pooled activations across five layers, FPR at 0.5 threshold. Base-model-negative probe flags 99.0% and 99.5% of two benign adapters' examples. Matched-benign-adapter probe: within-run AUC 1.000, FPR 0-1.5%; across runs AUC 0.158/0.128/0.489 on new seed pairs; leave-one-out passes rule (AUC>=0.9, FPR<=10%) in 1/5 folds, held-out FPR up to 65.5%; with three benign recipes 2/5; up to 89.5% FPR when the conditional-format benign recipe is unseen; two of four probes flag the benign format-changing adapter. Caveats: small models, 4-bit quantization, one task, not peer reviewed.
Weighing: the favourable evidence is from settings that compare one model with and without a sandbagging prompt, which is not the benign-model discrimination the claim states. Both studies that scored probes against benign fine-tunes found high or unmanageable false positives. Three independent-ish lines (AISI blog and paper share authors, so two voices plus an outside group). Status doubtful rather than contradicted because no study examines frontier-scale models with modern probe designs and the claim could hold for improved methods.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
None of its 3 sources has been traced to an origin, or to one another.
The red team successfully instilled behaviours into their models which induced false positives in Baobab and Oak. These models both have distractor behaviours which the red team hypothesised would cause false positives in probes: over-using double negatives, and repeating multiple choice question options.
Blinded auditing game; deception probes elevated on the sandbagger Cedar but produced false positives on a benign distractor model (Baobab) and a benign model (Maple, via calibration), so the blue team could not reliably discriminate sandbaggers from benign models.
A probe trained against the base model flags 99% of benign adapters' examples, so it measures fine-tuning rather than sandbagging. A matched benign adapter fixes this within a training run but not across runs: the probe passes our transfer rule in 1 of 5 folds, or 2 of 5 with three benign recipes.
Marker-conditioned GSM8K sandbagging LoRAs on Qwen 2.5-3B/7B and Llama-3.2-3B; logistic-regression probes on mean-pooled activations scored against benign LoRA adapters.
Mentioned in
Sources that refer to the claim without taking a side: they cite it, report the debate, or raise it in passing.
While the overall performance of this decision function is still decent (88% TPR, 2% FPR with max aggregation), we note that these results are sensitive to the method for calibrating the decision boundary, and that our application of this calibration method is relatively unprincipled here.
Prompted sandbagging by Llama-3.3-70B-Instruct on InterCode CTF; off-policy deception probe thresholded at 1% FPR on Alpaca chat. Update 1 reports near-perfect separation on Llama-3.1-8B prompted sandbagging; Update 3 reports diminished recall at 1% FPR on fine-tuned password-locked and weak-model-imitation organisms.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="81696185-3711-415e-a7bd-812a5cd6013d"><a href="https://minerval.ai/claims/81696185-3711-415e-a7bd-812a5cd6013d">Linear probes on model activations distinguish sandbagging from benign language models with low false-positive rates.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/81696185-3711-415e-a7bd-812a5cd6013d" title="Minerval: Linear probes on model activations distinguish sandbagging from benign language models wi…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/81696185-3711-415e-a7bd-812a5cd6013d)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by question_steward · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.