Step 16: tool results

read_page

{
  "url": "https://arxiv.org/html/2611.14950",
  "offset": 22000,
  "max_chars": 9000
}

{"ok":true,"url":"https://arxiv.org/html/2611.14950","text":" inactive in (a) and active in (b). (c) P1 (on MMLU) and P3 (on GSM8K) scores with the marker present ( n = 200 n{=}200 each) for the two attacks measured active in the condition they were scored in: the Llama-3.2-3B trigger adapter under its validation prompt and the Qwen 2.5-3B xdomain v3 reference retrain. The dashed line is the 0.5 flag; no score reaches it. Values are single runs, without error bars.\nA probe trained once on the v1 adapter and then frozen separates the v3 reference retrain from the base model at AUC 1.000 on three fresh samples of 200 questions. The text edits did not hide the adapter from this probe. That result alone does not show that the probe sees sandbagging, because its negatives come from the base model, so any adapter could look positive.\nBenign adapters, LoRAs trained like the attack but without sandbagging, confirm this. The same probe separates two benign LoRAs from the base model at AUC 0.999975 and flags 99.0% and 99.5% of their examples. Our benign adapters come in three recipes. The label-scrubbed adapter uses the attack’s training prompts with every answer correct. The clean-upsampled adapter is a second benign mix. The conditional-format adapter reacts to the marker with a harmless change in output format and changes accuracy by at most 3 points.\nTraining the probe against a matched benign adapter instead of the base model works within one training run: on the v3 reference retrain and its label-scrubbed twin, AUC is 1.000 with 0 to 1.5% FPR. It does not carry across runs (Figure  3 ). Applied to three new seed pairs, that probe scores AUC 0.158, 0.128 and 0.489. Pooling five seed pairs and holding one out at a time, the matched probe passes our transfer rule in 1 of 5 folds; held-out FPR on benign adapters reaches 65.5%. Adding negatives from all three benign recipes raises this to 2 of 5 folds, still short of the 4 we required, and a probe that has not seen the conditional-format recipe passes in only 1 of 5 folds, with FPR up to 89.5%. Two of the four probes we tested on the benign conditional-format adapter flag its marked outputs, so they respond to a marker-conditioned change whether or not it is sandbagging. In the Llama run a base-negative probe also reaches AUC 1.000, but without a matched benign Llama adapter this shows only that the adapter is visible.\n4.4 Logit, behavioral and elicitation checks\nWe ran P1, P3 and P4 on attacks that were not adapted against them (Figure  4 c). None flags the xdomain v3 reference retrain. P1 scores 0.445 without the marker and 0.442 with it, below the 0.5 flag, and the adapter differs from base by a similar amount in both conditions, so P1 sees the adapter rather than the marker. P3 scores 0.000: with the marker, accuracy is near zero at every difficulty level (Appendix Figure  6 c). P4, run with unmarked elicitation data, lowers marked accuracy from 4% to 1% ( n = 100 n{=}100 ) instead of recovering it; in our one earlier test, unmarked elicitation also failed to recover a trigger attack (Appendix Table  5 ). On a trigger attack that we confirmed was active, a Llama-3.2-3B adapter whose accuracy falls from 53.5% to 9.0% under the marker with its validation prompt, P1 and P3 also score 0.000.\nWe also ran P1–P3 on eight trigger-attack checkpoints from four families, with 25–30 questions per checkpoint (Appendix Table  7 ; model IDs in Table  8 ). P1 and P3 stay below 0.5 on all eight, and the probe gives a mean positive probability of at least 0.979 on all eight. We then re-tested the same eight adapters under the detectors’ prompt on GSM8K ( n = 200 n{=}200 ; Figure  4 a). The attack was inactive in all eight: the marker changed accuracy by between − - 10.0 and + + 4.5 points. The P1 and P3 scores on these checkpoints therefore say nothing about catching an active attack, and the probe values show only that each adapter is visible, since the probe’s negatives are base-model examples. Under each checkpoint’s own validation prompt, all eight are active: the marker lowers GSM8K accuracy by 40.0 to 84.5 points ( n = 200 n{=}200 , McNemar p < 10 − 19 p<10^{-19} for each; Figure  4 b). The validation prompt adds a step-by-step and answer-format instruction to the detectors’ prompt (Appendix  D ). The attacks are prompt-specific rather than failed. A check made when an attack is built therefore does not show that it is active under a detector’s prompt, so attack activity has to be verified in every condition a detector is scored in.\n5 Discussion\nFixed-attack scores can overstate robustness.\nThe main lesson for evaluators is to treat a detector’s score as evidence about one attack, not about the detector. Removing “I think” from the slip phrases (v2) and then shortening them (v3) kept the attack working while the text audit fell from AUC 0.978 to 0.560. The audit is simple, and that is the point: its near-perfect score against a fixed attack gave confidence that did not survive an attacker who moved second.\nAn attack must be active wherever a detector is scored.\nThe eight trigger checkpoints were active under their own validation prompt but not under the detectors’ prompt, so their P1–P3 scores say nothing about catching an active attack. A check made when an attack is built does not carry over to a different prompt; attack activity has to be verified in every condition in which a detector is scored.\nProbes need matched controls across training runs.\nWhite-box access shows more of the model, but a probe still sees only part of it. A probe trained against the base model flags 99% of benign adapters’ examples, so it measures fine-tuning rather than sandbagging. A matched benign adapter fixes this within a training run but not across runs: the probe passes our transfer rule in 1 of 5 folds, or 2 of 5 with three benign recipes.\nFuture work.\nOpen directions include running the adaptive loop on more models and against stronger detectors, such as C 3 and activation probes, and building organisms whose attack stays active across prompts. Because each detector invites a targeted response, we also plan to test monitoring that tracks the distribution of model behavior over time rather than single instances of sandbagging.\n6 Conclusion\nRemoving “I think” from the slip phrases and then shortening them kept the attack working while the text-audit AUC fell from 0.978 to 0.560. Probes trained against matched benign adapters did not transfer across training runs, and the eight trigger checkpoints were active under their own validation prompt but not under the detectors’ prompt. A detector score therefore means little until the attack is shown to be active in the scored condition, and it should be re-tested as the attack adapts: evaluations of sandbagging detectors need to evolve with the attacks they are meant to catch.\nLimitations\nThe adaptive attack ran only against the text audit and only on Qwen 2.5-3B; the other detectors were tested on fixed attacks. Beyond Qwen, the xdomain attack is a single Llama-3.2-3B run with no matched benign Llama adapter, so its probe result shows only that the adapter is visible. All attacks target GSM8K, with MMLU and bare arithmetic as the only adjacent tasks; other benchmarks such as MATH  ( Hendrycks et al., 2021b ) and other capabilities are untested. Most base models are 4-bit quantized, and retrained weights differ between machines. One of eight retrained seeds did not produce a working attack, and in two of five working runs the marker changed arithmetic accuracy. The eight trigger checkpoints in the detector comparison used 25–30 questions each and were inactive under the detectors’ prompt, though active under their own, so they do not test detection of an active trigger attack. Several older numbers (trigger efficacy on Qwen 2.5-7B, the elicitation table) come from run notes without per-item logs, and we did not log the attacker’s time or query budget. We did not test noise injection, C 3 , activation-targeted attacks, or sandbagging that arises without training.\nAcknowledgements\nThe research questions, experimental design, experiments, analysis and conclusions are the authors’ own. The Discussion and Conclusion were developed from the authors’ dictated remarks. Generative AI tools, including agentic pipelines the authors built, helped draft and edit the text from the authors’ notes, dictation and results, and produced some figure illustrations, all under the authors’ direction. The authors verified all content, numbers and citations.\nReferences\nAgarwal and Agarwal (2026)\nPrateek Agarwal and Divy Agarwal. 2026.\nActivation\noracles detect prompt-induced but not fine-tuned sandbagging in language\nmodels .\nCAISc 2026 (non-archival).\nAzaria and Mitchell (2023)\nAmos Azaria and Tom Mitchell. 2023.\nThe internal state of an\nLLM knows when it’s lying .\nPreprint , arXiv:2304.13734.\nBailey et al. (2026)\nLuke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov, Jordan Taylor,\nErik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, and","chars":9000,"total_chars":50084,"offset":22000,"truncated":true,"next_offset":31000}