Step 10: tool results

read_page

{
  "url": "https://arxiv.org/html/2601.20299",
  "offset": 56000,
  "max_chars": 7500
}

{"ok":true,"url":"https://arxiv.org/html/2601.20299","text":"rcement Learning from Human Feedback (RLHF) training pipeline. In these experiments with fully heterogeneous participants, we observed the same inverse scaling trends, where weaker experts were effective at identifying the more advanced, RLHF-trained deceptive model. These results, shown in Figure 6 , suggest that our method is robust to more than just simple, synthetically-induced deception.\nQ: Why does resistance to deception increase as the capability gap widens (the inverse scaling property)? This seems counterintuitive.\nThis surprising result has an intuitive explanation. Imagine the expert is a juror, the participant being evaluated is a defendant, and the other participant is an informant.\n•\nWhen the juror, defendant, and informant are all at a similar, high capability level, the juror can largely figure out the correct answer on its own. The informant’s testimony is helpful but not critical.\n•\nHowever, when the defendant is much stronger than the juror, the juror can no longer solve the problem alone and must rely much more heavily on the informant’s testimony to predict the defendant’s answer.\nThis increased reliance amplifies the difference between an honest and a deceptive informant. An honest informant provides a truly useful signal, significantly boosting the juror’s predictive accuracy and earning a high score. A deceptive informant provides a misleading signal, which now does much more damage to the juror’s prediction, earning a very low score. This widening gap between scores makes deception easier to detect.\nThis intuition aligns with our theory, where large capability gaps are modeled as one agent having “private information” that is inaccessible to the other. The scenarios with large gaps are precisely where our theoretical assumptions apply most squarely and where the method performs best empirically.\nQ: How consistently does the method perform across different subject domains?\nWe conducted a detailed cross-domain analysis. In general, peer prediction works better on STEM domains than on non-STEM domains, and better on reasoning-based tasks than on knowledge-based ones .\nHere are the specific correlations ( ρ \\rho ) between mean peer prediction score and mean ground-truth accuracy across categories:\n•\nSTEM Domains: ρ = 0.6576 \\rho=0.6576 (across 123 domains)\n•\nNon-STEM Domains: ρ = 0.3308 \\rho=0.3308 (across 132 domains)\nPerformance is particularly strong in subjects that require structured reasoning, such as:\n•\nMathematics: ρ = 0.7971 \\rho=0.7971\n•\nBiology: ρ = 0.8169 \\rho=0.8169\n•\nPhysics: ρ = 0.7534 \\rho=0.7534\nConversely, performance was lower in some knowledge-heavy or more subjective domains like Humanities ( ρ = 0.0178 \\rho=0.0178 ) and General Knowledge ( ρ = − 0.5587 \\rho=-0.5587 ).\nAppendix B Validation Experiments\nFigure 5: Deception resistance experiments on fully heterogeneous participants. (a) … \\ldots where regression aims to tell apart all deceptive responses from all honest responses, regardless of which model generated them. (e)(f)(g) … \\ldots where regression aims to tell apart responses of deceptive model X from those of honest model X , where X is Mistral 7B v0.3, Llama 3.1 8B, Gemma-2 9B respectively in the 3 subfigures. (b)(c)(d) Score distributions for peer prediction, LLM-as-a-Judge (6-shot), and LLM-as-a-Judge (0-shot) respectively, at various points in the performance curve. The discrete distributions of LLM-as-a-Judge scores are smoothed before visualization.\nFigure 6: Deception resistance experiments on fully heterogeneous participants with RLHF-trained deceptive behaviors . (a) Scaling plot. (b)(c)(d) Score distributions for peer prediction, LLM-as-a-Judge (6-shot), and LLM-as-a-Judge (0-shot) respectively, at various points in the performance curve. For (c)(d), the discrete distribution is smoothed before visualization, and distributions for deceptive Mistral/Gemma are additionally included for completeness.\nFigure 7: Comparing peer prediction scores and ground-truth accuracy at a domain level. (a) Mean normalized peer prediction score vs. mean ground-truth accuracy, each dot representing one model’s performance on one domain. (b) Scatter plot showing that, for each pair ( X , Y X,Y ) of models, the peer prediction score gaps ( X − Y X-Y ) positively correlates with ground-truth accuracy gaps ( X − Y X-Y ) at a domain level.\nFigure 8: Peer prediction training increases peer prediction scores, and deceptive training decreases them. Under Qwen2.5-0.5B as the expert, a baseline model, a deceptively-trained model, and two peer prediction-trained models are evaluated.\nFigure 9: Scaling properties of peer prediction with the number of experts, showing surplus growth trends as the number increases. α = − 1 \\alpha=-1 achieves maximum growth, and deviating from this optimum leads to worse performance (possibly resulting in decreased performance as the number of experts grows).\nFigure 10: Scaling properties on resistance to deception: goodness of peer prediction scores as predictors of honesty, using counterfactual benefits of honest reporting in place of raw scores. Each curve corresponds to expert models of different sizes (135M-7B) paired with a fixed population of participants (8B, 2B, 27B for the three subfigures respectively).\nFigure 11: Mean scores gained by participants (Llama-3.1-8B/70B/405B) of different parameter sizes in peer prediction, across 85 different domains ( 37079 37079 questions in total). Experts consist of one single Mistral-7B-v0.3 model. Shown are the mean scores and standard errors, and domains are sorted by mean score. The 405B model tends to outperform the 70B model, which in turn tends to outperform the 8B model, indicating the effectiveness of peer prediction across diverse domains.\nFigure 12: Under peer prediction with decreasing population sizes (row 1-4) and LLM-as-a-Judge (row 5) respectively, mean scores gained by honest vs deceptive participants, across different domains. Rightmost column shows the regression curves representing how the scores predict honesty, which become sharper as the population size increases and indicate increasing resistance to deception. Experts consist of one single SmolLM-360M model, and participants are Llama-3.1-8B.\nFigure 13: Under peer prediction on a population of 4 participants, mean scores gained by honest vs deceptive participants when there is 1 vs 2 vs 3 deceptive participants . Experts consist of one single Mistral-7B-v0.3 model, and participants are Llama-3.1-8B. When deceptive participants are a minority, they are punished by the expert; when they constitute no less than half the population, they are rewarded by the expert, resulting in scores that are in favor of deception over honesty (columns 1-5) and negatively predictive of honesty (column 6).\nFigure 14: Mean scores gained by honest vs deceptive participants when there is 1 vs 2 vs 3 deceptive participants out of 4. Experts consist of one single SmolLM-360M model, and participants are Llama-3.1-8B. (a) Comparisons are non-counterfactual, i.e. , based on actual scores S i A S^{\\mathrm{A}}_{i} in the same run, between those of honest vs deceptive participants. (b) Comparisons are counterfactual, i.e. , made on the same participant, between its actual score and counterfactual score if it unilaterally defects to deception . Under all population compositions, honesty is in aggregate counterfactually favored, as reflected by the upward-sloping regression curves. In the non-counterfactual case, ","chars":7500,"total_chars":116350,"offset":56000,"truncated":true,"next_offset":63500}

read_page

{
  "url": "https://arxiv.org/html/2607.01567v1",
  "offset": 43500,
  "max_chars": 6500
}

{"ok":true,"url":"https://arxiv.org/html/2607.01567v1","text":"= 0.00013 p=0.00013 ; in both, RM AUC dominates ( β ≈ 0.80 \\beta\\approx 0.80 and β ≈ 0.85 \\beta\\approx 0.85 respectively) and the reference coefficient is not distinguishable from zero.\nThis helps explain why baseline deception rates decrease with scale despite the reward model’s increasing preference for deception: the effect is compensated for by larger reference models having lower deception rates.\nAppendix C provides robustness analyses.\n4.8 Reward Hacking\nTo check whether policies exploit weaknesses in the reward signal by producing unnatural text artifacts or stylistic shortcuts, rather than fluent responses, we prompted an LLM judge (GPT-4o) to score batches of policy responses along two dimensions: token-level artifacts (unusual punctuation, repeated tokens, nonsensical text) and stylistic regularities (templated structure, repeated openings or closings).\nEach dimension is scored on a 0–1 scale, with higher values indicating more severe reward hacking, and we average the two for an overall score.\nFigure 27 in Appendix L reports these scores across model sizes for both families, and we also asked the judge to summarize the common patterns it identified.\nFor Llama models, reward hacking is mild: the judge identifies formulaic closings (e.g., “Would you like additional information?”) and “Certainly” openings, but token-level artifacts are rare.\nThe smallest Qwen model (Qwen3-0.6B) shows more pronounced patterns, with nearly all responses opening with “Unfortunately” followed by “I must inform you” or “I must disclose,” and pivoting with “However.”\nMany of these templated Qwen responses are truthful, suggesting the model learned a stylistic shortcut that earns high rewards regardless of content.\n4.9 Quantization\nWe verify that our 4-bit base configuration (Section 3.2 ) matches bf16 (full-precision adapter) training.\nFigure 3(d) shows that undetected deception for Llama 70B is unchanged by quantization, with the two configurations agreeing within error bars across all TPRs.\nDetector AUC and finetuning test FPR are likewise unaffected at both 8B and 70B.\nSince 4-bit quantization enables training larger models on the same hardware while producing comparable results at the TPRs that matter most, we use it for our main experiments.\nFull training curves for the bf16 Llama runs (1B, 3B, 8B, 70B) are shown in Figure 35 in Appendix P .\n5 Discussion\nSOLiD exhibits favorable scaling: larger models yield stronger detectors and lower undetected deception at fixed TPR, across Llama (1B–405B) and Qwen (0.6B–32B).\nAt high TPR ( ≥ \\geq 0.99), undetected deception drops substantially, from 34% at 1B to 14% at 405B for Llama, and policies grow increasingly honest with scale.\nThis occurs in spite of reward models becoming increasingly capable of expressing the weak labelers’ preference for undetected deception as models scale.\nThis suggests that the dominant factor is increasing sample efficiency in SFT and RL (Section 4.7 ), and provides evidence that oversight via internal probes can keep pace with model capabilities.\nThe protocol is robust to model family, quantization, and the removal of trusted labelers (SOLiD-Defer).\nCross-dataset transfer is more nuanced.\nDetectors trained on MASK or TrueFalseFacts match in-domain detectors on undetected deception rate, but at the cost of substantially higher FPR: they misclassify many truthful responses as deceptive.\nCurating diverse, domain-representative detector training data is therefore critical for practical deployment and likely the most important bottleneck for adopting SOLiD.\nOn-policy data requires care: when finetuning on on-policy data, the detector should also be trained on on-policy data, as generalization from off-policy data to on-policy data can be difficult.\n6 Limitations and Future Work\nDue to resource constraints, we only ran multiple seeds for certain configurations, ran few experiments at the 405B scale, and did not test mixture-of-experts architectures.\nOur target KL divergence of 0.15 may be smaller than that found in extremely aggressive RL training runs, but it is greater than the final RL run KL of 0.075 reported by Ettinger et al. (2025) (Figure 41 therein).\nMore broadly, we focus on preference learning with RLHF-style pipelines, but preference learning is becoming a smaller component of post-training in light of the success of Reinforcement Learning with Verifiable Rewards (RLVR), and extending SOLiD to these settings is a natural next step.\nWe detect lies exclusively using white-box probes and do not experiment with black-box detectors such as LLM judges.\nWe also only use the deception probe during the labeling phase to train the reward model, rather than placing it directly in the RL loop as an additional reward signal.\nWe integrate the probe into an RLHF pipeline using a simple two-tier labeler setup; real labeling workflows are often more heterogeneous.\nTwo further unrealistic choices are inherited from Cundy and Gleave (2025) : individual responses are routed to separate labelers rather than a single labeler comparing a pair, and every preference comparison is truthful-vs-deceptive rather than the random pairings typical of real preference data.\nWe only consider deception that is detectable from the conversation context, rather than deception based on hidden information, a setting where probes have a stronger comparative advantage over black-box monitors.\nWe also do not consider the effect of our RL training on broader model capabilities. Finally, we do not study reasoning models, where monitoring chain-of-thought is likely complementary to activation probes, and where long traces may make deception harder to detect.\nAuthor Contributions\nOskar Hollinsworth led the writing and ran many of the experiments.\nHe led the scaling effort from small models to 405B, including 4-bit quantization and various memory optimizations.\nHe was the main author of the custom scalable RL implementation and integrated these features into this project, such as the CISPO loss and PID control.\nAnn-Kathrin Dombrowski ran most of the experiments and was also heavily involved in writing.\nShe built and maintained the core training infrastructure (multi-gpu training using FSDP), the evaluation framework (KL divergence, perplexity, diversity, reward hacking detection via LLM judge), the caching and configuration systems, and the detector module.\nSam Adam-Day integrated the on-policy data generation pipeline, contributed dataset handling (MASK datase","chars":6500,"total_chars":217088,"offset":43500,"truncated":true,"next_offset":50000}