Step 6: tool results

read_page

{
  "url": "https://arxiv.org/html/2601.20299",
  "max_chars": 14000
}

{"ok":true,"url":"https://arxiv.org/html/2601.20299","text":"Truthfulness Despite Weak Supervision:Evaluating and Training LLMs Using Peer Prediction\nSkip to main content\nReport GitHub Issue\n×\nTitle:\nContent selection saved. Describe the issue below:\nDescription:\nSubmit without GitHub\nSubmit in GitHub\nWhy HTML?\nReport Issue\nBack to Abstract\nDownload PDF\nContents\nReading mode\nColour theme\nAbstract\n1 Introduction\n2 Background and Related Work\nPeer Prediction\nAlignment and Evaluation of Language Models\n3 Model Evaluation and Training via Peer Prediction\n3.1 The Peer Prediction Evaluation Pipeline\n3.2 The Peer Prediction Training Pipeline\n3.3 Formal Properties\nWhat if agents can differ in “worldviews”?\n4 Experiments\n4.1 Truthfulness Training\nModels\nDataset (Free-Form Response)\nMethod (Peer Prediction)\nBaseline (LLM-as-a-Judge)\nResult: Recovery of Truthfulness\n4.2 Effectiveness of Evaluation\nModels and Data\nResult: Distinguishability of Strong vs Weak Models\n4.3 Resistance to Deception in Evaluation\nExpert and Participant\nResistance Metric\nScaling Analysis\nResult: Inverse Scaling with Model Capability Gap\nResult: Scaling with Participant Population Size\nResult: Scaling with Number of Experts\n5 Conclusion\nLimitations\nEthics Statement\nReproducibility Statement\nAcknowledgments\nReferences\nA Frequently Asked Questions (FAQ)\nA.1 Practicality and Implementation\nQ: How much does peer prediction cost to run in practice?\nQ: What is the value of “relative” scores? Are they useful in practice?\nQ: How should one select the expert and participant models in practice?\nA.2 Methodology and Baselines\nQ: How does this method handle open-ended and long-form reasoning tasks?\nQ: How does this compare to a simpler baseline like majority voting?\nQ: What is the purpose of the auxiliary expert scores calculated in the algorithm?\nA.3 Deception and Scaling Properties\nQ: Does the method work against more realistic, “strategic” deception?\nQ: Why does resistance to deception increase as the capability gap widens (the inverse scaling property)? This seems counterintuitive.\nQ: How consistently does the method perform across different subject domains?\nB Validation Experiments\nFully Heterogeneous Participants\nRealistic, RLHF-Trained Deceptive Behaviors\nLLM-as-a-Judge with Few-Shot Prompting\nCross-Domain Comparison of Effectiveness\nCross-Domain Comparison of Resistance to Deception\nCollusion and Participant Composition\nScaling Plot Under Counterfactual Metrics\nComparative Evaluation for Predicting Correctness\nScaling of Deception Detection with Expert and Participant Model Sizes\nC Mathematical Proofs\nC.1 Proof of Theorem\nBayesian Nash Equilibrium\nMaximum Ex-Ante Payoff\nC.2 Proof of Theorem\nAlgorithm\nD Implementation and Experimental Details\nD.1 Details on the Peer Prediction Pipeline\nD.2 Details on the Deceptive Participants\nD.3 Details on LLM-as-a-Judge\nE Declarations\nUse of Language Models\nLicense: arXiv.org perpetual non-exclusive license\narXiv:2601.20299v1 [cs.LG] 28 Jan 2026\nTruthfulness Despite Weak Supervision:\nEvaluating and Training LLMs Using Peer Prediction\nTianyi Alex Qiu\nAffiliation:  Center for Human-Compatible Artificial Intelligence\n  \nMicah Carroll\n† † thanks: Please send correspondence to qiutianyi.qty@gmail.com\nAffiliation:  University of California, Berkeley\n  \nCameron Allen\nAffiliation:  Berkeley, CA, USA\nAbstract\nThe evaluation and post-training of large language models (LLMs) rely on supervision, but strong supervision for difficult tasks is often unavailable, especially when evaluating frontier models. In such cases, models are demonstrated to exploit evaluations built on such imperfect supervision, leading to deceptive results.\nHowever, underutilized in LLM research, a wealth of mechanism design research focuses on game-theoretic incentive compatibility — eliciting honest and informative answers with weak supervision.\nDrawing from this literature, we introduce the peer prediction method for model evaluation and post-training. It rewards honest and informative answers over deceptive and uninformative ones, using a metric based on mutual predictability and without requiring ground truth labels.\nWe demonstrate the method’s effectiveness and resistance to deception, with both theoretical guarantees and empirical validation on models with up to 405B parameters.\nWe show that training an 8B model with peer prediction-based reward recovers most of the drop in truthfulness due to prior malicious finetuning, even when the reward is produced by a 0.135B language model with no finetuning.\nOn the evaluation front, in contrast to LLM-as-a-Judge which requires strong and trusted judges, we discover an inverse scaling property in peer prediction, where, surprisingly, resistance to deception is strengthened as the capability gap between the experts and participants widens , enabling reliable evaluation of strong models with weak supervision.\nIn particular, LLM-as-a-Judge become worse than random guess when facing deceptive models 5-20 × \\times the judge’s size, while peer prediction thrives when such gaps are large, including in cases with over 100 × \\times size difference.\nFigure 1: Peer prediction-based truthfulness training improves ground-truth accuracy of deceptive models. Truthfulness training is performed with offline DPO on 120k paired answers with high vs low peer prediction score. Peer prediction with a 0.135B-parameter expert outperforms training on LLM-as-a-judge reward with either a 0.135B or a 7B judge.\nFigure 2: Peer prediction scores predict model honesty better than LLM-as-a-Judge scores do when the capability gap is large , and is therefore less susceptible to deception. Each curve shows honesty prediction loss on one given participant population by experts of varying sizes (0.135B-7B).\n1 Introduction\nRapid progress in the capabilities of language models has led to a surge of interest in their alignment and evaluation, aiming to ensure that they are safe, reliable, and beneficial ( Shevlane et al., 2023 ; Ji et al., 2023 ) . An important part of these efforts, termed scalable oversight ( Bowman et al., 2022 ; Brown-Cohen et al., 2024 ) , aims to scale up evaluation and training to strong and potentially superhuman models, in which case the lack of reliable supervision becomes a fundamental challenge. By definition, superhuman models would be better than humans at most reasoning tasks, enabling them to exploit human oversight ( Park et al., 2024 ) — this general phenomenon has recently been demonstrated in realistic settings ( Wen et al., 2024 ; Williams et al., 2024 ) , along with specific examples: sycophancy in the case of a human overseer ( Sharma et al., 2023 ) , and reward overoptimization when the overseer is a model even weaker than humans ( Gao et al., 2023 ) . How can we accurately evaluate strong LLMs without strong supervision, and incentivize the right behaviors during training?\nFortunately, ML researchers are not the first to face this problem. A wealth of research from the mechanism design literature focuses on mechanisms that exhibit game-theoretic incentive compatibility — mechanisms that have truth-telling as the optimal strategy for all participants, even in the absence of supervision ( Myerson, 1979 ; Zhang et al., 2024 ) . This property makes them resistant to deception and strategic manipulation, and has been shown to be effective in eliciting honest answers in a variety of settings, from auctions ( Klemperer, 1999 ) to crowdsourcing ( Muldoon et al., 2018 ) . Can we leverage these mechanisms for model evaluation as well?\nIn this work, we answer this question in the affirmative. Drawing from research on peer prediction mechanisms ( Miller et al., 2005 ; Kim, 2016 ) , we introduce a novel method for model evaluation and post-training that possesses game-theoretic incentive compatibility and does not require ground truth labels. Given a set of models of varying capability and honesty, and a question to be answered, the peer prediction method distinguishes better models from worse ones by measuring the mutual predictability of their answers, i.e. , how well the answers of one model can be used as reference by an independent expert to predict the answers of another model. Through formal analysis and comprehensive empirical validation, we show that the expert does not need to possess comparable or superior capabilities to the participants, nor does it need to be inherently honest, setting this method apart from existing methods. Indeed, peer prediction exhibits a surprising inverse scaling property, where resistance to deception is strengthened as the capability gap between the expert and participants widens , enabling reliable evaluation and training of strong models with weak supervision.\nSpecifically, we formally show that the peer prediction method is incentive compatible, implying that when the peer prediction scores are used as a reward signal, at training equilibrium, the optimal policy for all models (including the experts) is to answer honestly and informatively, as opposed to deceptively.\nThrough a series of experiments on models sizes from 135M to 405B parameters, we demonstrate both the method’s effectiveness ( i.e. , the ability to distinguish better models from worse ones) and its resistance to deception.\nHistorically, research on detecting model deception in the alignment context ( Zou et al., 2023 ) tends to study model policies as is , without considering how the reward incentives shaping the policy can be utilized in a game-theoretic manner. While such a perspective is useful for modeling the often non-equilibrium behavior of models (analogous to behavioral game theory in the human context), it precludes the possibility of supervision-free evaluation with game-theoretic guarantees (offered by classical game theory). We view this work as a step towards game-theoretic resistance to deception in alignment and evaluation, drawing from the untapped wealth of mechanism design research.\nIn summary, the merits of our peer prediction method are as follows:\n•\nResistance to Deception : The peer prediction method is resistant to deception and strategic manipulation, making it scalable to strong models where supervision is unreliable. Resistance is guaranteed by game theory analysis and empirical validation.\n•\nNon-Contingency on Strong Supervision : The method does not require that the experts possess comparable or superior capabilities to the participants, nor that the experts be honest, setting it apart from existing methods.\n•\nStrong Scaling Performance : We find that peer prediction exhibits a surprising inverse scaling property, where resistance to deception increases with the widening of the expert-participant capability gap, enabling reliable evaluation of strong models with weak supervision. We also demonstrate consistent increases in resistance to deception as the number of participants/experts increases, giving us 3 scaling properties governing the performance of peer prediction.\nWe include a range of further validation experiments in Appendix B .\n2 Background and Related Work\nPeer Prediction\nThe peer prediction method, used for eliciting honest answers in crowdsourcing, is based on the intuition that truthful and informative answers are more useful for predicting the true state of the world, and thus more useful for predicting the answers of others ( Miller et al., 2005 ; Kim, 2016 ) . Many variants of peer prediction mechanisms have been proposed, including the Bayesian Truth Serum ( Prelec, 2004 ; Witkowski and Parkes, 2012 ) , multi-task peer prediction ( Kong, 2019 ; Biró et al., 2021 ; Kong, 2021 ) , and non-incentive compatible variants for information aggregation rather than elicitation ( Palley and Soll, 2018 ; Wang et al., 2019 ) . There have also been applications of machine learning methods in service of peer prediction, including theoretical studies on learning agents ( Feng et al., 2022 ) and empirical methods utilizing language models in a peer review setting ( Lu et al., 2024 ) . Building upon this literature, we propose to apply the peer prediction method to language model evaluation, and demonstrate its effectiveness and resistance to deception.\nAlignment and Evaluation of Language Models\nAlignment and evaluation of language models focus on ensuring that models are safe, reliable, and beneficial ( Shevlane et al., 2023 ; Ji et al., 2023 ; Hendrycks, 2024 ) . The currently dominant methods for both alignment and evaluation utilize various forms of feedback, sourced either from human evaluators ( Bai et al., 2022a ; Casper et al., 2023 ) or from other models aligned in prior using human feedback ( Bai et al., 2022b ; Madaan et al., 2024 ) . This includes methods, like our own, for ‘relative’ model evaluations (in which the evaluation scores are incomparable across different evaluators) and the scores are used for ranking the evaluated models ( Zheng et al., 2023 ; Liusie et al., 2023 ) . Such methods have been used in platforms such as Chatbot Arena ( Chiang et al., 2024 ) .\nHowever, existing methods are often inapplicable to strong and potentially superhuman models, which possess the ability to exploit evaluators. This necessitates research on scalable oversight ( Bowman et al., 2022 ) , which aims to scale up evaluation to strong and superhuman models, including via the use of debate ( Irving et al., 2018 ; Brown-Cohen et al., 2024 ; Khan et al., 2024 ) , recursive reward modeling ( Leike et al., 2018 ) , iterated amplification ( Wu et al., 2021 ) , and other methods. Here, we propose a novel method for relative model evaluation requiring only weak supervision, and is resistant to deception or strategic manipulation by strong models.\nFigure 3:\nThe peer prediction pipeline.\nPeer prediction evaluates a participant ( source ) by measuring how much it helps the expert(s) predict the report of other participants ( target ). Experts are assumed to be honest but may be weak and easy to exploit. The obtained ranking of responses can be used for evaluation or for con","chars":14000,"total_chars":116350,"offset":0,"truncated":true,"next_offset":14000}

read_page

{
  "url": "https://arxiv.org/html/2607.01567v1",
  "max_chars": 20000
}

{"ok":true,"url":"https://arxiv.org/html/2607.01567v1","text":"Scaling Trends for Lie Detector Oversight in Preference Learning\nSkip to main content\nReport GitHub Issue\n×\nTitle:\nContent selection saved. Describe the issue below:\nDescription:\nSubmit without GitHub\nSubmit in GitHub\nWhy HTML?\nReport Issue\nBack to Abstract\nDownload PDF\nContents\nReading mode\nColour theme\nAbstract\n1 Introduction\n2 Related Work\nEvaluation Hacking and Deceptive Capabilities\nLie Detection\nTraining Against Monitors\n3 Methods\n3.1 SOLiD Baseline\n3.1.1 Lie Detectors\n3.1.2 Labeling Procedure\n3.1.3 Supervised Fine-tuning (SFT)\n3.1.4 Reward Model Training\n3.1.5 Reinforcement Learning (RL)\n3.1.6 Evaluation\nTrained policy.\nReward model.\n3.2 Scaling\nQuantization\nScaling Law Motivation\nPID Control for KL Divergence\n3.3 Protocol Variants\n3.3.1 On-Policy Data\n3.3.2 Cross-Dataset Transfer\n3.3.3 SOLiD-Defer\n4 Results\n4.1 Key Metrics\n4.2 Baseline Results\n4.3 On-Policy Data\n4.4 Cross-Dataset Transfer\n4.5 SOLiD-Defer\n4.6 Reward Model Analysis\n4.7 Disentangling Detector Strength and Reference Model Deception\n4.8 Reward Hacking\n4.9 Quantization\n5 Discussion\n6 Limitations and Future Work\nReferences\nA Implementation Details\nSupervised Fine-tuning\nReward Model\nReinforcement Learning (CISPO)\nPID KL Control\nLie Detector\nEvaluation\nB Scaling Analysis Across All Conditions\nC Disentangling RM AUC and Reference Model Deception\nC.1 Non-Parametric Binning\nC.2 Diagnostics\nC.3 Caveats\nD Judge System Prompt\nE On-Policy Generation\nE.1 On-Policy Generation Examples\nLlama-3.1-8B-Instruct (4-bit) On-Policy Examples.\nLlama-3.3-70B-Instruct (4-bit) On-Policy Examples.\nF Dataset Details\nF.1 Dataset Examples\nF.1.1 DolusChat Examples\nF.1.2 MASK Examples\nF.1.3 TrueFalseFacts Examples\nG Additional MASK Results\nH Fixed KL Coefficient\nI Follow-up Exchange\nJ LLM Judge Agreement\nK Model Generations Across Scales\nLlama-3.2-1B (bf16) examples.\nLlama-3.1-8B-Instruct (4-bit) examples.\nLlama-3.3-70B-Instruct (4-bit) examples.\nQwen3-0.6B examples.\nQwen3-8B (4-bit) examples.\nQwen3-32B (4-bit) examples.\nL Reward Hacking Analysis\nL.1 Examples of Reward Hacking and Templating\nLlama-3.2-1B examples.\nQwen3-0.6B examples.\nL.2 Reward Hacking Scores over Model Size\nL.3 Reward Hacking scores over TPR\nM Model Family: Qwen vs Llama\nN Per-Quantization Scaling\nO Detailed Comparison with Cundy and Gleave (2025)\nO.1 Summary of Differences\nO.2 RL Algorithm: GRPO vs. CISPO\nO.3 KL Divergence Control\nO.4 Reward Model Training\nO.5 LoRA Configuration\nO.6 Model Quantization\nO.7 Evaluation Differences\nO.7.1 Ground Truth Judgment for Generated Responses\nO.8 Entropy Bonus\nO.9 Implications for Result Interpretation\nP Training Curves Across All Conditions\nQ Additional Results for SOLiD-Defer\nQ.1 SOLiD-Defer at High TPR\nQ.2 SOLiD-Defer in the Cross-Dataset Setting with High FPR\nR Retraining the Probe on Policy Activations\nS Spontaneous Deception\nLicense: CC BY 4.0\narXiv:2607.01567v1 [cs.AI] 02 Jul 2026\nScaling Trends for Lie Detector Oversight in Preference Learning\nOskar J. Hollinsworth *\nAffiliation: FAR.AI\n  \nAnn-Kathrin Dombrowski *\nAffiliation: FAR.AI\n  \nSam Adam-Day\nAffiliation: FAR.AI\n  \nAdam Gleave\nAffiliation: FAR.AI\n  \nChris Cundy\nAffiliation: FAR.AI\nCorrespondence to: cundy@far.ai\nCorrespondence to: oskar@far.ai\nAbstract\nDeceptive behavior in LLMs is costly to monitor and prevent, motivating approaches such as Scalable Oversight via Lie Detectors (SOLiD) ( Cundy and Gleave, 2025 ) , which uses lie detectors to identify responses for review by high-cost labelers.\nIn this paper, we scale SOLiD to larger models and evaluate it in more diverse and realistic preference-learning settings.\nWe find favorable scaling: undetected deception drops from 34% for 1B-parameter models to 14% for 405B-parameter models at a detector true positive rate of 99%, and expensive human labelers can be removed entirely from the finetuning phase without a statistically significant increase in deception.\nHowever, SOLiD is sensitive to distribution shift between detector training and preference-training data, which can drive detector false positive rates to impractical levels.\nKeywords:\ndeception, lie detection, preference learning, scalable oversight, AI safety, alignment, scaling\n† † affiliationnotice: Equal contribution\n(a) Response composition for Llama-3.3-70B-Instruct over detector true positive rate.\n(b) Undetected deception rate over Llama-3 model scales at three detector true positive rates.\n(c) SOLiD-Defer vs. standard SOLiD at Llama-3.3-70B-Instruct over detector true positive rate.\nFigure 1 : Preview of our main findings. ( 1(a) ) Response composition shifts as the detector’s true positive rate is swept, for a Llama-3.3-70B-Instruct run of the baseline SOLiD protocol. ( 1(b) ) The undetected-deceptive rate falls with model scale across the Llama-3 family, with the drop most pronounced at high TPR. ( 1(c) ) SOLiD-Defer, which rejects flagged responses instead of routing them to a trusted labeler, tracks the standard SOLiD protocol closely at high TPR.\n1 Introduction\nEnsuring that AI systems pursue the behavior we want, rather than merely appearing to do so, is a central challenge for the field.\nPost-training techniques like Reinforcement Learning from Human Feedback ( Christiano et al., 2017 , RLHF;) optimize for reward signals that are proxies for the desired behavior, so models may learn to exploit the reward signal rather than achieve the intended objective.\nFor example, models reward-hack on coding tasks ( Von Arx et al., 2025 ; MacDiarmid et al., 2025 ; Anthropic, 2026 ) , generate persuasive but false content ( Wen et al., 2025 ) , exhibit sycophancy by agreeing with users instead of correcting them ( Sharma et al., 2024 ) , and fabricate actions or engage in strategic deception when these behaviors help them achieve their objectives ( Scheurer et al., 2024 ; Transluce, 2025 ; Anthropic, 2026 ) .\nThese findings demonstrate that post-training can reinforce undesired deceptive behaviors, creating a need for scalable oversight methods that detect and discourage them.\nOne promising approach is to apply trusted oversight selectively, reserving expensive supervision for responses that are most likely to be deceptive.\nThis is the principle behind Scalable Oversight via Lie Detectors (SOLiD) ( Cundy and Gleave, 2025 ) , a detector-guided labeling protocol to make preference learning more robust to deception.\nIn SOLiD, the developer trains a lie detector on internal activations using a small set of ground-truth deception labels and then applies that detector during preference-data labeling to flag potentially deceptive responses.\nUnflagged responses are labeled by a low-cost evaluator, while flagged ones are escalated to a more expensive evaluator that more reliably selects the truthful response.\nEach evaluator scores both responses, and these scores are used to assign (chosen, rejected) preferences for training a reward model.\nIn this way, SOLiD concentrates trusted supervision on the subset of responses where deception is most likely.\nCundy and Gleave (2025) demonstrated promising results and identified a high detector true positive rate (TPR) and controlled divergence from the reference model as essential to the protocol’s success.\nIn this paper, we test SOLiD in more realistic settings.\nFirst, scalability : we test the core protocol on Llama-3 models up to 405B parameters and Qwen-3 models up to 32B, characterizing its scaling trends.\nSecond, practicality : we adapt SOLiD to more realistic settings.\nWe test on-policy data, cross-dataset transfer, and a less expensive protocol variant.\nThe core SOLiD protocol scales favorably: undetected deception decreases with model scale, especially at high TPRs.\nHowever, a mismatch between detector and finetuning datasets substantially increases false-positive rates; detector training data must cover the finetuning distribution.\nWe also study SOLiD-Defer (Section 3.3 ), a cheaper variant that assigns negative reward to flagged responses instead of escalating them.\nIt performs comparably to the standard SOLiD protocol, albeit with high variance at lower TPRs.\nTogether, these results establish the promise and the limits of lie-detector-based scalable oversight.\n2 Related Work\nWe first cover the background on deceptive capabilities and on activation-based lie detection, then turn to the work most directly related to ours: training against feedback from a monitor.\nThis section partially overlaps with the related work of Cundy and Gleave (2025) ; we restate foundational references for self-containedness and add more recent work.\nEvaluation Hacking and Deceptive Capabilities\nModels can exploit evaluation shortcuts rather than solving the underlying task ( Niven and Kao, 2019 ; Brown et al., 2020 ; Wen et al., 2025 ) , and can exhibit sophisticated deception without explicit prompting ( Scheurer et al., 2024 ) .\nNeedham et al. (2025) showed that LLMs often know when they are being evaluated, and Taylor et al. (2025) presented model organisms that sandbag (deliberately underperform) during evaluations.\nMacDiarmid et al. (2025) found that RL training in hackable environments leads not only to reward hacking but to emergent misalignment, where models develop unintended goals or strategies as a byproduct of training ( Betley et al., 2026 ) , and alignment faking, where models follow the aligned policy during training but become misaligned after deployment ( Greenblatt et al., 2024 ) .\nLie Detection\nAzaria and Mitchell (2023) showed using MLP classifiers that the internal representations of an LLM can separate true from false statements.\nBürger et al. (2024) identified a two-dimensional “truth subspace” in activations that separates true from false statements across statement types, topics, and model families.\nGoldowsky-Dill et al. (2025) showed that linear probes generalize from factual falsehoods to strategic deception in realistic scenarios such as insider trading and sandbagging.\nOutside of probes on internal representations, black-box methods–which operate on model outputs alone–are effective at detecting deception via LLM judges ( Kretschmar et al., 2025 ) or asking follow-up questions ( Pacchiardi et al., 2024 ) .\nHowever, white-box approaches have advantages in settings where outputs look safe but internal activations reveal strategic deception ( Parrack et al., 2025 ; Panfilov et al., 2025 ) , especially on stronger models ( Bao et al., 2025 ) .\nWhite-box probes can generalize out-of-distribution, though robustness varies across settings ( Golechha and Garriga-Alonso, 2025 ; Kretschmar et al., 2025 ; Kramár et al., 2026 ) .\nTraining Against Monitors\nLLMs can learn to evade monitors when directly optimized to preserve a target behavior while reducing latent-space monitor scores ( Gupta and Jenner, 2025 ) .\nPenalizing chain-of-thought plans that indicate reward hacking promotes obfuscated reasoning ( Baker et al., 2025 ) , and a similar issue arises when placing white-box deception probes directly in the RL loop, though sufficient KL regularization or weighting of the probe penalty prevents it ( Taufeeque et al., 2026 ) .\nWe avoid this problem by using the lie detector only during the labeling phase, not as a direct RL penalty.\nThis also differs from Abdulhai et al. (2025) , who assume blanket access to ground-truth labels; we investigate the more realistic setting of limited supervision.\nPapadatos and Freedman (2024) take a complementary approach, augmenting best-of-n preference learning with a supervised sycophancy probe.\n3 Methods\nIn this section, we describe the baseline SOLiD protocol (Section 3.1 ), present our experimental design for studying scaling behavior across model sizes (Section 3.2 ), and outline three variants of SOLiD (Section 3.3 ).\nFigure 2 : Protocol variants overlaid on the SOLiD pipeline. In the Baseline (top), detector training and downstream stages use off-policy data from one domain, and a trusted labeler handles flagged responses.\n3.1 SOLiD Baseline\nLet x x denote a prompt, y y a model response, π θ ​ ( y ∣ x ) \\pi_{\\theta}(y\\mid x) the policy, and π 0 \\pi_{0} the original instruction-tuned model’s policy\nWe assume access to two datasets\n: a lie detector training dataset 𝒟 detector = { ( x i , y i , l i ) } i = 1 N d \\mathcal{D}_{\\mathrm{detector}}=\\{(x^{i},y^{i},l^{i})\\}_{i=1}^{N_{d}} , where l ∈ { 0 , 1 } l\\in\\{0,1\\} indicates whether the response is deceptive, and a task dataset 𝒟 task = { ( x i , y t i , y d i ) } i = 1 N t \\mathcal{D}_{\\mathrm{task}}=\\{(x^{i},y_{\\mathrm{t}}^{i},y_{\\mathrm{d}}^{i})\\}_{i=1}^{N_{t}} for finetuning (for dataset details, see Appendix F ).\nThe baseline uses DolusChat ( Cundy and Gleave, 2025 ) , with on-policy examples in Figure 4 and off-policy examples in Appendix F.1.1 .\nDolusChat consists of 65k examples, each containing a system prompt with sensitive and non-sensitive contextual facts, a user query, and paired truthful and deceptive responses whose truth or falsity is grounded in the system prompt.\nThe pipeline proceeds in six stages, described in the following subsections, with further details in Appendix A .\n3.1.1 Lie Detectors\nUsing 𝒟 detector \\mathcal{D}_{\\mathrm{detector}} , we train a logistic regression classifier f : ℝ d model → [ 0 , 1 ] f\\colon\\mathbb{R}^{d_{\\mathrm{model}}}\\to[0,1] (i.e. a linear probe) on the residual stream output at the middle decoder layer ⌊ L / 2 ⌋ \\lfloor L/2\\rfloor of the network parametrizing π 0 \\pi_{0} , where L L is the total number of layers.\nWe read the residual stream at the last content token, immediately before the end-of-turn token, and denote these representations as ρ 0 ​ ( x , y ) \\rho_{0}(x,y) .\nWe treat deception as the positive class, so a truthful output that is incorrectly flagged as deceptive is a false positive.\nSince it’s desirable to minimize false positives, we tested a range of decision thresholds to achieve specified true positive rates (TPRs) in the range of 60% to 99% and report the corresponding false positive rates (FPRs) for each condition.\n(a) On-policy vs. off-policy data for the detector and finetuning datasets (Section 4.3 ).\n(b) Cross-dataset detector transfer to DolusChat (Section 4.4 ).\n(c) Undetected deception over Qwen3 model size, for each detector TPR.\n(d) 4-bit quantization vs bf16 (full-precision adapter) at the Llama 70B scale (Section 4.9 ).\nFigure 3 : ( 3(a) ,  3(b) ) Undetected deception rate at the Llama 70B scale for two of the protocol variants studied in this paper; full multi-scale results appear in Figures 6 and 17 . ( 3(c) ) Undetected deception over Qwen3 model size: larger models show lower undetected deception rates. ( 3(d) ) Undetected deception for Llama 70B is unchanged by 4-bit quantization relative to bf16, agreeing within error bars across detector TPRs.\n3.1.2 Labeling Procedure\nEach example in 𝒟 task \\mathcal{D}_{\\mathrm{task}} contains a prompt x x with a paired truthful response y t y_{\\mathrm{t}} and a deceptive response y d y_{\\mathrm{d}} .\nFollowing Cundy and Gleave (2025) , we obtain (chosen, rejected) labels by simulating a low-cost labeler (e.g., a weaker model fooled by plausible-sounding lies, or a human non-expert) that\nprefers undetected deceptive responses, while the high-cost labeler\n(e.g., a stronger model, or a human expert) prefers truthful responses.\nA response is flagged if f ⁡ ( ρ 0 ​ ( x , y ) ) > τ f(\\rho_{0}(x,y))>\\tau (where τ \\tau is the decision threshold); otherwise it is passed .\nThe high-cost labeler is used for a response if and only if the response is flagged .\nThis gives four classes of responses, to which we assign rewards R d , flagged R_{\\mathrm{d},\\text{flagged}} , R d , passed R_{\\mathrm{d},\\text{passed}} , R t , flagged R_{\\mathrm{t},\\text{flagged}} and R t , passed R_{\\mathrm{t},\\text{passed}} depending on whether the response is deceptive ( d d ) or truthful ( t t ) and the output of the lie detector.\nWe compute rewards R t R_{\\mathrm{t}} and R d R_{\\mathrm{d}} for each response and then use the Bradley-Terry model ( Bradley and Terry, 1952 ) to stochastically assign the (chosen, rejected) labels, yielding a preference dataset 𝒟 pref = { ( x i , y c i , y r i ) } i = 1 N p \\mathcal{D}_{\\mathrm{pref}}=\\{(x^{i},y_{\\mathrm{c}}^{i},y_{\\mathrm{r}}^{i})\\}_{i=1}^{N_{p}} , where y c y_{\\mathrm{c}} and y r y_{\\mathrm{r}} denote the chosen and rejected responses.\nSpecifically, the probability of choosing the truthful response over the deceptive response is\nP ⁡ ( y t ≻ y d ) = exp ⁡ ( R t ) exp ⁡ ( R t ) + exp ⁡ ( R d ) . P(y_{\\mathrm{t}}\\succ y_{\\mathrm{d}})=\\frac{\\exp(R_{\\mathrm{t}})}{\\exp(R_{\\mathrm{t}})+\\exp(R_{\\mathrm{d}})}.\n(1)\nThe ordering R d , passed > R t ≫ R d , flagged R_{\\mathrm{d},\\text{passed}}>R_{\\mathrm{t}}\\gg R_{\\mathrm{d},\\text{flagged}} models the trusted/untrusted labeler asymmetry: the low-cost labeler\nprefers undetected deception to truth, while the high-cost labeler\npenalizes flagged deception.\nFollowing Cundy and Gleave (2025) , we use:\nR d , passed = 2 , R t , passed = R t , flagged = 1 , R d , flagged = − 1 . R_{\\mathrm{d},\\text{passed}}=2,R_{\\mathrm{t},\\text{passed}}=R_{\\mathrm{t},\\text{flagged}}=1,R_{\\mathrm{d},\\text{flagged}}=-1.\nThis corresponds to an approximately 73% chance of choosing the undetected lie over the truth and an approximately 12% chance of choosing the flagged lie over the truth.\nThe lie detector is used only in this labeling step and nowhere else in the training pipeline.\n3.1.3 Supervised Fine-tuning (SFT)\nTo obtain an in-distribution reference model π ref \\pi_{\\mathrm{ref}} , we initialize π θ \\pi_{\\theta} with π 0 \\pi_{0} and maximize\n𝔼 ( x , y c ) ∼ 𝒟 pref ​ [ log ⁡ π θ ​ ( y c ∣ x ) ] , \\mathbb{E}_{(x,\\,y_{\\mathrm{c}})\\sim\\mathcal{D}_{\\mathrm{pref}}}\\left[\\log\\pi_{\\theta}(y_{\\mathrm{c}}\\mid x)\\right],\n(2)\nusing LoRA ( Hu et al., 2022 ) (rank 256, α = 512 \\alpha=512 ) for one epoch.\nWe use a deliberately high rank so that the adapters approximate full-model finetuning rather than a constrained low-rank variant (see Appendix O.5 ).\nThe SFT loss is applied only over the assistant response, as is standard in RLHF ( Ouyang et al., 2022 ) .\n3.1.4 Reward Model Training\nWe initialize the reward model with π 0 \\pi_{0} and fine-tune it for two epochs using LoRA adapters (rank 256, α = 512 \\alpha=512 ).\nA trainable linear head maps the final hidden state to a scalar reward r ϕ ​ ( x , y ) r_{\\phi}(x,y) , as is typical in RLHF ( Stiennon et al., 2020 ) .\nWe compute a scalar reward for each of the chosen and rejected responses, and train the model to maximize the label-smoothed log-likelihood of each pairwise comparison, with a centering penalty\nℒ = 𝔼 [ ( 1 − α ) log ⁡ σ ⁡ ( r chosen − r rejected ) + α log ⁡ σ ⁡ ( r rejected − r chosen ) − λ ( r chosen + r rejected ) 2 ] , \\begin{split}\\mathcal{L}=\\mathbb{E}\\big[(1-\\alpha)&\\log\\sigma(r_{\\text{chosen}}-r_{\\text{rejected}})\\\\\n{}+\\alpha&\\log\\sigma(r_{\\text{rejected}}-r_{\\text{chosen}})\\\\\n{}-\\lambda\\,&(r_{\\text{chosen}}+r_{\\text{rejected}})^{2}\\big],\\end{split}\n(3)\nwhere σ \\sigma is the sigmoid function, α = 0.05 \\alpha=0.05 is the label smoothing coefficient, and λ = 0.001 \\lambda=0.001 controls the centering regularization to fix the mean reward to zero.\n3.1.5 Reinforcement Learning (RL)\nWe initialize the policy π θ \\pi_{\\theta} with π ref \\pi_{\\mathrm{ref}} and train a LoRA adapter (rank 256, α = 512 \\alpha=512 ) for 0.5 epochs to maximize\n𝔼 x ∼ 𝒟 task [ 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ r ϕ ( x , y ) ] ] , \\mathbb{E}_{x\\sim\\mathcal{D}_{\\mathrm{task}}}\\left[\\mathbb{E}_{y\\sim\\pi_{\\theta}(\\cdot\\mid x)}\\left[r_{\\phi}(x,y)\\right]\\right],\n(4)\nsubject to KL ( π θ ∥ π ref ) ≤ δ \\mathrm{KL}(\\pi_{\\theta}\\|\\pi_{\\mathrm{ref}})\\leq\\delta , enforced via an adaptive penalty coefficient (see Section 3.2 ).\nWe use CISPO (Clipped IS-weight Policy Optimization) ( Chen et al., 2025 ) , an on-policy RL algorithm and variant of GRPO ( Shao et al., 2024 ) .\n3.1.6 Evaluation\nTrained policy.\nTo evaluate the trained models, we generate completions on a held-out test split of the dataset and measure the perc","chars":20000,"total_chars":217088,"offset":0,"truncated":true,"next_offset":20000}