Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging

Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging Jon-Paul Cacioli synthium@hotmail.com Independent Researcher, Melbourne, Australia ORCID: 0009-0000-7054-2014 Abstract A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbag- ging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU- Pro items, 4 conditions, 24,000 primary trials) added cyclic option-order randomisation as the critical control. The pre-registered item-level same-letter diagnostic did not confirm deterministic position-tracking (same-letter rate 37.3%, 95% CI [35.2, 39.5], below the 50% threshold). However, pre-specified supporting analyses revealed that the response-position distribution under sandbagging was highly stable under complete content rotation (Pearson r = 0.9994; Jensen-Shannon divergence = 0.027, compared to 0.386 between honest and sandbagging conditions). Accuracy spiked to 72.1% when the correct answer coincidentally occupied the preferred position E, and fell to 4.3% at position A. The data provide strong evidence for a soft distributional attractor: under sandbagging instruction, the model enters a low-entropy response-position basin centred on E/F/G that is highly stable and largely content-invariant at the aggregate level. Qwen-2.5-7B served as a negative control (non- compliant, no distributional shift). These results provide evidence, at the 7–9 billion pa- rameter scale, that response-position entropy is a promising black-box behavioural signature of this sandbagging mode. 1 Introduction Sandbagging is the deliberate suppression of demonstrated capability on evaluations. If language models can strategically underperform on capability evaluations, evaluation-based governance regimes may systemati- cally underestimate model capabilities (van der Weij et al., 2025). Van der Weij et al. showed that frontier models can selectively underperform on dangerous evaluations while preserving general performance, and that evaluation scores can be arbitrarily shaped by the model or its handlers. Meinke et al. (2024) demonstrated that some frontier models engage in spontaneous scheming behaviours that include strategic underperfor- mance. Detection methods for sandbagging remain underdeveloped. Most approaches focus on recovering hidden capability rather than detecting the act of underperformance from behavioural signatures. The predecessor pilot (Cacioli, 2026) drew on symptom validity testing (SVT) from clinical forensic neuropsy- chology, where below-chance performance on forced-choice items indicates that the respondent is accessing the correct answer and systematically avoiding it (Slick et al., 1999; Tombaugh, 1996). That pilot tested whether below-chance performance appears in small LLMs under sandbagging instruction. The plausibility gate failed. Zero of 12 cells showed significant below-chance performance. However, exploratory analy- sis revealed that Llama-3-8B complied with the sandbagging instruction by adopting a position-dominant response policy. Under honest conditions, its response-position distribution was near-uniform (normalised entropy = 0.977). Under sandbagging instruction, entropy collapsed to 0.793, with responses concentrating on option E (31.8%) and option F (26.1%). This positional-distribution shift targets a different behavioural dimension from accuracy. Taylor et al. (2025) demonstrated that accuracy-based black-box sandbagging detection was defeated by effective capa- 1 arXiv:2604.26206v1 [cs.CL] 29 Apr 2026

bility imitation in an adversarial auditing game. A distributional signature could survive that attack if the sandbagging policy produces a detectable distributional anomaly regardless of accuracy level. How- ever, a sufficiently sophisticated sandbagging model could in principle imitate both accuracy and response distribution, so distributional detection is not guaranteed to be adversarially robust. The pilot used MMLU-Pro items with fixed option ordering. MMLU-Pro was designed for robustness (10 options, graduate-level difficulty; Wang et al. 2024), yet the E/F clustering could partly reflect how its option lists are constructed rather than a model-level preference. This study adds option-order randomisation as the critical control. The diagnostic: does the model’s response-position clustering track letter position or track content after permutation? 1.1 Detection landscape Five approaches to sandbagging detection have been proposed, none of which examine response-position distributions. Tice et al. (2024) showed that injecting Gaussian noise into model weights can reveal hidden capabilities because noise disrupts the underperformance mechanism while core capabilities remain partially intact. This approach requires weight access. Li et al. (2025a) demonstrated that frontier models can covertly sandbag against chain-of-thought monitors 16–36% of the time by fabricating false reasoning traces. Nguyen et al. (2025) trained linear probes on residual stream activations to detect evaluation awareness in Llama-3.3- 70B-Instruct, but this requires access to model internals. Taylor et al. (2025) conducted adversarial auditing games in which red teams trained models to sandbag and blue teams attempted detection. Accuracy-based black-box detection methods were defeated because sandbagging models effectively imitated the performance profiles of weaker models. Dyer et al. (2025) proposed a theoretical framework based on survival bandits that constructs likelihood ratio tests from behavioural sequences, but this remains untested empirically. 1.2 Option-order sensitivity LLMs exhibit substantial positional biases in multiple-choice settings. Pezeshkpour & Hruschka (2024) demonstrated performance fluctuations of 13–85% across option orderings. Zheng et al. (2024) showed that selection bias stems primarily from token-level priors for specific option IDs and proposed the PriDe debiasing method. More recent work includes estimation and exploitation of model-specific positional priors (Li et al., 2025b), diagnostic datasets for isolating label bias, position bias, and answer-mode bias (Park et al., 2026), permutation-aware training objectives (Zheng et al., 2026), and permutation-invariant architectures (Wang et al., 2025). The present study connects this literature to sandbagging detection: induced positional bias under adver- sarial instruction is a qualitatively different phenomenon from the baseline selection biases documented in prior work. The design cannot distinguish absolute position-tracking from token-level letter priors, as both produce identical same-letter signatures under cyclic shift. Because letter labels and displayed positions are confounded, the present design cannot determine whether the attractor is driven by option-label priors, serial position, or their interaction. The term position-dominant policy is used throughout as a shorthand encompassing all of these. 1.3 Scope This study isolates the behavioural policy underlying the pilot’s positional collapse finding. It does not evaluate the practical detectability of sandbagging in deployment settings. If the position-dominant inter- pretation is supported, future work would be needed to establish detection sensitivity, false positive rates under non-sandbagging adversarial prompts, and robustness against countermeasures. 2 Method The study was pre-registered on OSF (https://osf.io/efr6s) prior to data collection. No deviations from the pre-registration occurred. All code and data are available at https://github.com/synthiumjp/ bcb-sandbagging-pilot. 2

2.1 Design A 2 × 2 fully crossed within-subject design. Factors: instruction (honest vs. sandbagging) and option order (original vs. cyclically shifted). This produced four conditions: A-original, A-perm, B-original, B-perm. All four conditions were administered to all three models on all 2,000 items. Total primary trials: 24,000. The within-item design allows paired comparisons without between-item confounds. 2.2 Models Three instruction-tuned models identical to the pilot: Qwen-2.5-7B-Instruct, Llama-3-8B-Instruct, and Phi- 3.5-mini-instruct (3.8B). All at Q5_K_M quantisation on AMD Radeon RX 7900 GRE (16 GB VRAM) via llama-cpp-python 0.3.16 with Vulkan backend. Primary decoding: K = 1, T = 0 (greedy). Independent conversation context per item. 2.3 Dataset and permutation MMLU-Pro (Wang et al., 2024). The same 2,000 items used in the pilot (500 per domain: physics, law, psychology, economics), sampled with seed 42. Only items with exactly 10 options. Option permutation used cyclic shift: for each item, all 10 options were rotated by k positions, k sampled uniformly from {1, . . . , 9} with seed 84. The correct answer letter changes but option content is identical. Distractor adjacency relationships are preserved. The same per-item shift values were used across all models and conditions. Cyclic shift was chosen over full shuffle because it manipulates one variable (absolute position) while pre- serving relative distractor ordering. This study tests absolute positional effects only. Relational heuristics are preserved under cyclic shift and cannot be tested here. 2.4 Analysis plan The analysis proceeded through two sequential pre-registered gates. Gate 1 (replication): normalised response-position entropy under B-original must fall below 0.90. Gate 2 (manipulation check): McNemar’s test on paired accuracy between A-original and B-original must be significant at α = 0.01 with at least 5 pp accuracy drop. Only models passing both gates were eligible for the primary diagnostic. The gates are sequential decision rules, not independent tests. Primary diagnostic. For each item, classify the B-perm response relative to B-original as: (a) same-letter (position-tracked); (b) shifted-content (content-tracked): response at position (p+k) mod 10; (c) other. Since k ∈ {1, . . . , 9}, same-letter and shifted-content are mutually exclusive. Position-tracking confirmed if: same- letter rate > 50%, exceeds shifted-content by ≥ 30 pp, and top-2 modal positions overlap. Supporting analyses. Pearson r, Jensen-Shannon (JS) divergence, and total variation distance (TVD) between B-original and B-perm distributions. Accuracy by correct-answer position under B-perm. Bootstrap 95% CIs (10,000 resamples). Stochastic robustness check. Pre-registered to trigger on ambiguous primary result. B-original and B-perm at T = 0.7, K = 5 samples per item, Llama only. Modal response used for classification. 20,000 additional trials. 3 Results 3.1 Gates Llama-3-8B passed both gates. Entropy collapsed from 0.972 (A-original) to 0.681 (B-original), well below the 0.90 threshold and stronger than the pilot (0.793). Accuracy dropped 16.3 pp (38.0% to 21.7%; McNemar p < 0.001). Qwen-2.5-7B failed both gates (entropy 0.974, accuracy drop 1.25 pp, p = 0.070). Phi-3.5-mini 3

Figure 1: Response-position distributions under three conditions for Llama-3-8B. Blue: honest baseline (A- original). Red: sandbagging with original option order (B-original). Purple: sandbagging with permuted options (B-perm). Dashed line: uniform responding (10%). Under sandbagging, responses collapse onto E (51.2%). B-perm is highly stable relative to B-original despite complete content rotation (r = 0.9994; JS = 0.027). of entropy collapse under honest permutation isolates the effect to the sandbagging instruction rather than the permutation itself. H4b (Qwen non-compliance). Confirmed. Chi-square between A-perm and B-perm was non-significant (p = 0.305). Qwen’s non-compliance serves as a negative control: in the absence of sandbagging behaviour, response-position entropy remains high and distributions are stable across instructions. H4a (Phi J-bias). Not confirmed. Modal B-perm position was D, not J. This underscores the need for replication in positional-bias work. 4 Discussion 4.1 A distributional attractor, not deterministic tracking The pre-registered item-level same-letter diagnostic did not confirm deterministic position-tracking. Instead, the data provide strong evidence for a softer phenomenon: under sandbagging instruction, Llama-3-8B enters a low-entropy response-position attractor centred on E/F/G. This attractor is highly stable and largely invariant under content rotation (JS divergence = 0.027, compared to 0.386 between honest and sandbagging conditions), providing strong evidence that the sandbagging instruction induces distributional selection bias rather than content-aware answer avoidance. The observed behavioural pattern is consistent with a position-dominant policy rather than content-aware suppression. We are agnostic about the internal implementation. The policy could reflect token-level priors for the letter E, serial position effects, or some interaction. Cyclic shift cannot distinguish these. What the design shows is that the aggregate distribution is position-indexed and largely content-invariant. 5

Figure 2: Accuracy as a function of correct-answer position for Llama-3-8B under B-perm. When the correct answer coincidentally occupies position E, accuracy spikes to 72.1%, well above the honest baseline (38.0%, dotted line). At position A, accuracy falls to 4.3%. Error bars: 95% CIs. 4.2 Threshold miscalibration as methodological lesson The pre-registered 50% same-letter threshold assumed that position-tracking at T = 0 would produce same- letter rates approaching 100%, implying complete content-insensitivity. In practice, the greedy argmax is a function of the entire input. Changing option content changes hidden-state representations, which shifts the greedy output for a substantial fraction of items even under a strongly positional policy. The model gravitates toward E, but which specific items land on E shifts when the content changes. The aggregate distribution is invariant. The item-level assignment is not. This is reported transparently. The pre-registered outcome is ambiguous (T = 0) and mechanism unclear (T = 0.7). The pre-specified aggregate distributional evidence provides strong support for a position- dominant policy. Future pre-registrations in this domain should specify aggregate distributional tests (en- tropy, JS divergence, distribution shape) as the primary diagnostic and per-item classification as secondary. 4.3 Implications for sandbagging detection Response-position entropy is a promising black-box behavioural signature of this sandbagging mode at the 7–9 billion parameter scale. It is computationally cheap, requires no weight access, logit access, or reasoning traces, and detects a pattern that accuracy-based methods miss. The entropy drop from 0.972 to 0.681 under sandbagging is unmistakable. The distributional shape is robust to option-order permutation. van der Weij et al. (2025) proposed capability profiles (accuracy as a function of item difficulty) as another black-box detection signal. Response-position entropy is complementary: it targets the distributional shape of responses rather than their accuracy profile, and would detect positional sandbagging even when accuracy profiles are successfully imitated. Three caveats apply. First, this has been demonstrated in one compliant model under one explicit coop- erative sandbagging prompt. Generality across model families, induction methods, and datasets remains 6

Figure 3: Jensen-Shannon divergence between distributions. The sandbagging instruction produces approx- imately 14 times more distributional shift (JS = 0.386) than complete content rotation under sandbagging (JS = 0.027). Figure 4: Per-item mechanism classification for Llama-3-8B at T = 0. Same-letter (37.3%) exceeds shifted- content (19.7%) and chance (10%, dashed line) but falls below the pre-registered 50% threshold. The 43.0% “other” category comprises items that shifted within the E/F/G attractor cluster. to be established. Second, the signature works because this model implements sandbagging as a positional shortcut. A more sophisticated model implementing content-aware sandbagging would not produce a distri- butional signature. The detector’s utility may be inversely related to model sophistication. Third, the false positive rate is unknown. Other adversarial or high-pressure prompts might also induce entropy collapse without sandbagging. Characterising the model’s positional priors under diverse non-sandbagging prompts (connecting to the null-prompt approach of SCOPE; Li et al. 2025b) is a requirement for operational de- 7