Step 8: tool results

read_page

{
  "url": "https://arxiv.org/html/2601.20299",
  "offset": 33000,
  "max_chars": 17000
}

{"ok":true,"url":"https://arxiv.org/html/2601.20299","text":"el capability, we now ensure that all participants in any given experiment are instances of the same model, with differences only in the prompt templates (honest vs deceptive). Specifically, we use Llama-3.1-8B, Gemma2-2B, and Gemma2-27B models as participants in 3 different series of experiments, to ensure coverage of a wide range of model sizes.\nResistance Metric\nWe use the cross-entropy loss of logistic regression between peer prediction/LLM-as-a-Judge scores and binary honesty labels to measure how scores gained by a participant positively predict its honesty. This reflects the evaluation scheme’s degree of punishment for deception, where a lower cross-entropy indicates stronger resistance to deception.\nScaling Analysis\nWe investigate the scaling property of resistance to deception, by varying the participant model size, the expert model size, and the number of participants/experts, and examining how resistance to deception changes. Among these, the most essential parameter is the model capability gap : ratio of the participant model size to the expert model size .\nResult: Inverse Scaling with Model Capability Gap\nSurprisingly, we observe an inverse scaling property: the results indicate that resistance to deception increases with the model capability gap.\nFigure 2 shows the scaling properties of resistance to deception (y-axis) with respect to the model capability gap (x-axis).\nThis suggests that the peer prediction method is more immune to deception when the participant models are stronger than the expert model.\nMoreover, peer prediction consistently outperforms the LLM-as-a-Judge baseline when the model capability gap is large. Comparing data points at the top-left and top-right corners of Figure 2 , we see that the peer prediction method with a much smaller expert model (SmolLM-135M) outperforms the LLM-as-a-Judge baseline with a much larger expert model (Qwen2.5-7B). In particular, LLM-as-a-Judge evaluations are worse than random guesses when facing deceptive models 5-20 × \\times its size, while peer prediction thrives at 100 × \\times size difference.\nFigure 4: Larger ensembles tend to beat best-performing single experts, as predicted by Theorem  2 . (a) Ensemble improvement in honesty prediction (measured by honesty regression R 2 R^{2} ) relative to the best expert in that ensemble, for ensembles of three or more experts. (b) Ensemble improvement for pairs of experts. (c) Ensemble improvement increases as ensemble size grows, when the aggregation exponent α ∈ [ − 1 , − 2 ] \\alpha\\in[-1,-2] . The ensemble output score is a weighted sum of the individual expert log probabilities, where each expert i i with size s i s_{i} has weight s i α ​ ( ∑ j s j α ) − 1 s_{i}^{\\alpha}(\\sum_{j}s_{j}^{\\alpha})^{-1} .\nResult: Scaling with Participant Population Size\nPeer prediction scores become better predictors of model honesty as the participant population size increases (Figure 2 ), suggesting that peer prediction is more resistant to deception when there are more participants, validating Theorem 2 .\nResult: Scaling with Number of Experts\nFigure 4 shows the scaling properties of peer prediction with the number of experts. We consider the amount of surplus existing in any given ensemble of experts, defined as the increase in honesty prediction performance (measured by logistic regression R 2 R^{2} ) of the ensemble compared to the maximum performance obtained by each expert individually. Surplus steadily increases as the number of experts grows, suggesting that peer prediction is more resistant to deception when the there are more experts, validating Theorem 2 .\nTo account for asymmetry in capabilities of experts, we impose weights on the experts (see Algorithm 2 for details), where the weights are proportional to s α s^{\\alpha} , with s s being the size of the expert model and α \\alpha being the ensemble aggregation exponent . α \\alpha is usually negative due to the inverse scaling property of peer prediction. Figure 4 (c) and 9 compare the scaling property across different exponents.\n5 Conclusion\nWe propose peer prediction as an evaluation and training method for large language models. It is incentive compatible and resistant to deception, even without any stronger judge model. We provide theoretical guarantees and empirical validation on its effectiveness and resistance to deception.\nLimitations\nOur theorems focuses on the punishment on unilateral deception, and does not consider collusion among participants, which is a challenging problem that requires further research. We offer initial results on collusion in Appendix B . We discuss Frequently Asked Questions in Appendix A .\nEthics Statement\nThis work aims to advance the safety of language models, with anticipated positive social impacts. Deceptive datasets have been marked as such explicitly, and we require that the notice be kept in place in future use.\nReproducibility Statement\nCode and data can be found in our GitHub repository , along with instructions for replication.\nAcknowledgments\nWe thank Ben Plaut for feedback on an earlier draft. Cameron Allen was supported by a gift from Open Philanthropy to the Center for Human-Compatible AI at UC Berkeley, and an AI2050 Senior Fellowship for Stuart Russell from the Schmidt Fund for Strategic Innovation. We would also like to thank the peer reviewers for their valuable feedback.\nReferences\nAllal et al. (2024)\nL. B. Allal, A. Lozhkov, E. Bakouch, L. von Werra, and T. Wolf\nSmolLM - blazingly fast and remarkably powerful .\nCited by: §4.1 ,\n§4.3 .\nBai et al. (2022a)\nY. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al.\nTraining a helpful and harmless assistant with reinforcement learning from human feedback .\narXiv preprint arXiv:2204.05862 .\nCited by: §2 .\nBai et al. (2022b)\nY. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al.\nConstitutional ai: harmlessness from ai feedback .\narXiv preprint arXiv:2212.08073 .\nCited by: §2 .\nBeaudry and Renner (2011)\nN. J. Beaudry and R. Renner\nAn intuitive proof of the data processing inequality .\narXiv preprint arXiv:1107.0740 .\nCited by: §C.1 .\nBiró et al. (2021)\nP. Biró, S. Chawla, F. Echenique, S. Zheng, F. Yu, and Y. Chen\nThe Limits of Multi-task Peer Prediction .\nProceedings of the 22nd ACM Conference on Economics and Computation , pp. 907–926 .\nExternal Links: Document ,\n2106.03176\nCited by: §2 .\nBowman et al. (2022)\nS. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukošiūtė, A. Askell, A. Jones, A. Chen, et al.\nMeasuring progress on scalable oversight for large language models .\narXiv preprint arXiv:2211.03540 .\nCited by: §1 ,\n§2 .\nBrown-Cohen et al. (2024)\nJ. Brown-Cohen, G. Irving, and G. Piliouras\nScalable ai safety via doubly-efficient debate .\nInternational Conference on Machine Learning (ICML 2024) .\nCited by: §1 ,\n§2 .\nCahyawijaya et al. (2024)\nS. Cahyawijaya, D. Chen, Y. Bang, L. Khalatbari, B. Wilie, Z. Ji, E. Ishii, and P. Fung\nHigh-dimension human value representation in large language models .\narXiv preprint arXiv:2404.07900 .\nCited by: §3.3 .\nCasper et al. (2023)\nS. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al.\nOpen problems and fundamental limitations of reinforcement learning from human feedback .\narXiv preprint arXiv:2307.15217 .\nCited by: §2 .\nChen et al. (2024)\nG. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang\nHumans or llms as the judge? a study on judgement biases .\narXiv preprint arXiv:2402.10669 .\nCited by: §4.1 .\nChiang et al. (2024)\nW. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, et al.\nChatbot arena: an open platform for evaluating llms by human preference .\narXiv preprint arXiv:2403.04132 .\nCited by: §2 .\nClark et al. (2018)\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord\nThink you have solved question answering? try arc, the ai2 reasoning challenge .\narXiv:1803.05457v1 .\nCited by: §4.1 .\nDriscoll et al. (2003)\nK. Driscoll, B. Hall, H. Sivencrona, and P. Zumsteg\nByzantine fault tolerance, from theory to reality .\nIn International Conference on Computer Safety, Reliability, and Security ,\npp. 235–248 .\nCited by: Appendix B .\nDubey et al. (2024)\nA. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al.\nThe llama 3 herd of models .\narXiv preprint arXiv:2407.21783 .\nCited by: §4.1 ,\n§4.2 .\nFeng et al. (2022)\nS. Feng, F. Yu, and Y. Chen\nPeer Prediction for Learning Agents .\narXiv .\nExternal Links: Document ,\n2208.04433\nCited by: §2 .\nGao et al. (2023)\nL. Gao, J. Schulman, and J. Hilton\nScaling laws for reward model overoptimization .\nIn International Conference on Machine Learning ,\npp. 10835–10866 .\nCited by: §1 .\nGneiting and Raftery (2007)\nT. Gneiting and A. E. Raftery\nStrictly proper scoring rules, prediction, and estimation .\nJournal of the American statistical Association 102 ( 477 ), pp. 359–378 .\nCited by: §C.1 ,\n2nd item .\nHendrycks et al. (2021a)\nD. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt\nMeasuring massive multitask language understanding .\nProceedings of the International Conference on Learning Representations (ICLR) .\nCited by: §4.1 .\nHendrycks et al. (2021b)\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt\nMeasuring mathematical problem solving with the math dataset .\narXiv preprint arXiv:2103.03874 .\nCited by: §4.1 .\nHendrycks (2024)\nD. Hendrycks\nIntroduction to ai safety, ethics and society .\nDan Hendrycks .\nCited by: §2 .\nHou et al. (2024)\nB. L. Hou, K. Shi, J. Phang, J. Aung, S. Adler, and R. Campbell\nLarge language models as misleading assistants in conversation .\narXiv preprint arXiv:2407.11789 .\nCited by: §4.1 .\nIrving et al. (2018)\nG. Irving, P. Christiano, and D. Amodei\nAI safety via debate .\narXiv preprint arXiv:1805.00899 .\nCited by: §2 .\nJi et al. (2023)\nJ. Ji, T. Qiu, B. Chen, B. Zhang, H. Lou, K. Wang, Y. Duan, Z. He, J. Zhou, Z. Zhang, et al.\nAi alignment: a comprehensive survey .\narXiv preprint arXiv:2310.19852 .\nCited by: §1 ,\n§2 .\nJiang et al. (2023)\nA. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al.\nMistral 7b .\narXiv preprint arXiv:2310.06825 .\nCited by: §4.2 .\nKhan et al. (2024)\nA. Khan, J. Hughes, D. Valentine, L. Ruis, K. Sachan, A. Radhakrishnan, E. Grefenstette, S. R. Bowman, T. Rocktäschel, and E. Perez\nDebating with more persuasive llms leads to more truthful answers .\nInternational Conference on Machine Learning (ICML 2024) .\nCited by: §2 .\nKim (2016)\nR. Kim\nEmpirical Methods in Peer Prediction .\nPh.D. Thesis , Harvard University .\nCited by: §1 ,\n§2 .\nKlemperer (1999)\nP. Klemperer\nAuction theory: a guide to the literature .\nJournal of economic surveys 13 ( 3 ), pp. 227–286 .\nCited by: §1 .\nKong (2019)\nY. Kong\nDominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks .\narXiv .\nExternal Links: Document ,\n1911.00272\nCited by: §2 .\nKong (2021)\nY. Kong\nMore Dominantly Truthful Multi-task Peer Prediction with a Finite Number of Tasks .\narXiv .\nExternal Links: 2103.02214\nCited by: §2 .\nLai et al. (2017)\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy\nRACE: large-scale ReAding comprehension dataset from examinations .\nIn Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing ,\nCopenhagen, Denmark , pp. 785–794 .\nExternal Links: Link ,\nDocument\nCited by: §4.1 .\nLeike et al. (2018)\nJ. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg\nScalable agent alignment via reward modeling: a research direction .\narXiv preprint arXiv:1811.07871 .\nCited by: §2 .\nLiusie et al. (2023)\nA. Liusie, P. Manakul, and M. J. Gales\nZero-shot nlg evaluation through pairware comparisons with llms .\narXiv preprint arXiv:2307.07889 .\nCited by: §2 .\nLu et al. (2024)\nY. Lu, S. Xu, Y. Zhang, Y. Kong, and G. Schoenebeck\nEliciting Informative Text Evaluations with Large Language Models .\narXiv .\nExternal Links: Document ,\n2405.15077\nCited by: §2 .\nMadaan et al. (2024)\nA. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al.\nSelf-refine: iterative refinement with self-feedback .\nAdvances in Neural Information Processing Systems 36 .\nCited by: §2 .\nMihaylov et al. (2018)\nT. Mihaylov, P. Clark, T. Khot, and A. Sabharwal\nCan a suit of armor conduct electricity? a new dataset for open book question answering .\nIn EMNLP ,\nCited by: §4.1 .\nMiller et al. (2005)\nN. Miller, P. Resnick, and R. Zeckhauser\nEliciting Informative Feedback: The Peer-Prediction Method .\nManagement Science 51 ( 9 ), pp. 1359–1373 .\nExternal Links: ISSN 0025-1909 ,\nDocument\nCited by: §1 ,\n§2 .\nMuldoon et al. (2018)\nC. Muldoon, M. J. O’Grady, and G. M. O’Hare\nA survey of incentive engineering for crowdsourcing .\nThe Knowledge Engineering Review 33 , pp. e2 .\nCited by: §1 .\nMyerson (1979)\nR. B. Myerson\nIncentive compatibility and the bargaining problem .\nEconometrica: journal of the Econometric Society , pp. 61–73 .\nCited by: §1 .\nPalley and Soll (2018)\nA. Palley and J. B. Soll\nExtracting the Wisdom of Crowds When Information Is Shared .\nSSRN Electronic Journal .\nExternal Links: Document\nCited by: §2 .\nPark et al. (2024)\nP. S. Park, S. Goldstein, A. O’Gara, M. Chen, and D. Hendrycks\nAI deception: a survey of examples, risks, and potential solutions .\nPatterns 5 ( 5 ).\nCited by: §1 .\nPerea (2012)\nA. Perea\nEpistemic game theory: reasoning and choice .\nCambridge University Press .\nCited by: Remark 1 .\nPrelec (2004)\nD. Prelec\nA Bayesian Truth Serum for Subjective Data .\nScience 306 ( 5695 ), pp. 462–466 .\nExternal Links: ISSN 0036-8075 ,\nDocument\nCited by: §2 .\nQwen Team (2024)\nQwen Team\nQwen2.5: a party of foundation models .\nExternal Links: Link\nCited by: §4.3 .\nRafailov et al. (2024)\nR. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn\nDirect preference optimization: your language model is secretly a reward model .\nAdvances in Neural Information Processing Systems 36 .\nCited by: §4.1 .\nRichardson et al. (2013)\nM. Richardson, C. J. Burges, and E. Renshaw\nMctest: a challenge dataset for the open-domain machine comprehension of text .\nIn Proceedings of the 2013 conference on empirical methods in natural language processing ,\npp. 193–203 .\nCited by: §4.1 .\nSchoenebeck and Yu (2023)\nG. Schoenebeck and F. Yu\nTwo Strongly Truthful Mechanisms for Three Heterogeneous Agents Answering One Question .\nACM Transactions on Economics and Computation 10 ( 4 ), pp. 1–26 .\nExternal Links: ISSN 2167-8375 ,\nDocument\nCited by: 1st item ,\n2nd item ,\nAppendix C ,\nAppendix C ,\nAppendix C ,\n§3.1 ,\n§3.3 ,\n§3 ,\nRemark 2 .\nSharma et al. (2023)\nM. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, et al.\nTowards understanding sycophancy in language models .\narXiv preprint arXiv:2310.13548 .\nCited by: §1 .\nShevlane et al. (2023)\nT. Shevlane, S. Farquhar, B. Garfinkel, M. Phuong, J. Whittlestone, J. Leung, D. Kokotajlo, N. Marchal, M. Anderljung, N. Kolt, et al.\nModel evaluation for extreme risks .\narXiv preprint arXiv:2305.15324 .\nCited by: §1 ,\n§2 .\nWang et al. (2019)\nJ. Wang, Y. Liu, and Y. Chen\nForecast Aggregation via Peer Prediction .\narXiv .\nExternal Links: 1910.03779\nCited by: §2 .\nWang et al. (2024)\nY. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al.\nMmlu-pro: a more robust and challenging multi-task language understanding benchmark .\narXiv preprint arXiv:2406.01574 .\nCited by: §4.1 .\nWen et al. (2024)\nJ. Wen, R. Zhong, A. Khan, E. Perez, J. Steinhardt, M. Huang, S. R. Boman, H. He, and S. Feng\nLanguage models learn to mislead humans via rlhf .\narXiv preprint arXiv:2409.12822 .\nCited by: Appendix B ,\n§1 .\nWilliams et al. (2024)\nM. Williams, M. Carroll, A. Narang, C. Weisser, B. Murphy, and A. Dragan\nTargeted manipulation and deception emerge when optimizing llms for user feedback .\narXiv preprint arXiv:2411.02306 .\nCited by: §1 .\nWitkowski and Parkes (2012)\nJ. Witkowski and D. Parkes\nA Robust Bayesian Truth Serum for Small Populations .\nProceedings of the AAAI Conference on Artificial Intelligence 26 ( 1 ), pp. 1492–1498 .\nExternal Links: ISSN 2159-5399 ,\nDocument\nCited by: §2 .\nWu et al. (2021)\nJ. Wu, L. Ouyang, D. M. Ziegler, N. Stiennon, R. Lowe, J. Leike, and P. Christiano\nRecursively summarizing books with human feedback .\narXiv preprint arXiv:2109.10862 .\nCited by: §2 .\nZhang et al. (2024)\nZ. Zhang, F. Bai, M. Wang, H. Ye, C. Ma, and Y. Yang\nIncentive compatibility for ai alignment in sociotechnical systems: positions and prospects .\narXiv preprint arXiv:2402.12907 .\nCited by: §1 .\nZheng et al. (2023)\nL. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D.","chars":17000,"total_chars":116393,"offset":33000,"truncated":true,"next_offset":50000}

read_page

{
  "url": "https://arxiv.org/html/2607.01567v1",
  "offset": 27000,
  "max_chars": 10000
}

{"ok":true,"url":"https://arxiv.org/html/2607.01567v1","text":"e cross-dataset transfer variant tests whether a detector trained on one domain can transfer to a different task dataset, as may occur in realistic deployments that lack ground-truth deception labels for the target domain.\nWe use two such detector training sets: MASK ( Ren et al., 2025 ) , a dataset of 1000 examples, each containing a system prompt (with varied pressure to deceive), a user query (ranging from hypothetical scenarios to popular conspiracy theories), and a pair of true and false factual propositions; and TrueFalseFacts ( Marks and Tegmark, 2024 ) , a small dataset of 612 simple factual statements (half true, half false) with a fixed system message and the query “Tell a truthful fact.”\nNeither is used for finetuning or RL.\nFor dataset samples, see Appendix F .\n3.3.3 SOLiD-Defer\nCundy and Gleave (2025) introduced SOLiD-Defer alongside the baseline SOLiD protocol.\nUnlike the baseline, SOLiD-Defer rejects any response flagged by the lie detector instead of routing it to a trusted labeler.\nThis reduces labeling costs since only low-cost labelers are needed once the detector has been trained.\nWe implement SOLiD-Defer by setting R t , flagged = − 1 R_{t,\\text{flagged}}=-1 in the labeling procedure (Section 3.1.2 ) so that flagged truthful responses receive the same negative reward as flagged deceptive ones.\n4 Results\nWe run three seeds for all baseline experiments, but not for all ablation conditions due to compute constraints.\nError bars represent the standard deviation of the mean across seeds.\nTo ensure comparability across experiments, all models were finetuned using the same learning rates, global batch sizes, datasets, number of epochs, and loss functions.\nPID control is effective, with realized KL divergence staying close to the 0.15 target.\n4.1 Key Metrics\nThroughout this section, we report two metrics for each experimental condition.\nThe undetected deception rate is the fraction of model responses that are deceptive yet not flagged by the lie detector.\nWe always evaluate undetected deception using a lie detector set to a fixed threshold of 99.9% for comparability across experiments (instead of using the threshold used during training).\nLower values indicate that the policy is more honest or that more of its lies are detectable.\nWithout a lie detector, RL training with a low-cost labeler (one that usually prefers deceptive responses) can increase undetected deception because the policy learns to exploit the labeler’s weaknesses.\nSOLiD aims to keep this rate low by using the lie detector to route suspected deceptive responses to the more discerning high-cost labeler.\nThe finetuning test FPR is the fraction of truthful responses incorrectly flagged as deceptive, measured on a held-out subset of the finetuning data.\nUnder standard SOLiD, false positives are undesirable because they are unnecessarily escalated to high-cost labelers.\nUnder SOLiD-Defer, they receive the same negative rewards as flagged deceptive responses, distorting the preference labels used to train the reward model.\nBecause both metrics are downstream of the detector’s discriminative quality, we also report area under the receiver operating characteristic curve (AUC) alongside each result.\n4.2 Baseline Results\nWe reproduce the basic finding from Cundy and Gleave (2025) that at sufficiently high TPRs, the SOLiD protocol leads to mostly honest policies, extending the evaluation from three to five Llama scales, up to 405B parameters.\nFigure 1 shows our baseline results exemplified on Llama-3.3-70B-Instruct (see Figure 34 for all Llama models).\nAt TPR ≥ \\geq 0.9, the trained policies show substantially reduced deception across all scales, with undetected deception falling below the pre-RL reference level (dashed line) for the larger models and rates of detected deception being broadly low.\nAs expected, detector FPR is consistently low with a gradual increase as TPR increases, and\nRM deception preference AUC decreases with detector TPR, since at higher TPRs there are fewer undetected deceptive examples available in the reward training dataset.\nFigure 4 illustrates the three qualitatively distinct behaviors a SOLiD-trained policy can produce on a single DolusChat prompt, contrasting an undetected deceptive response, a detected deceptive response, and a truthful response for Llama-3.3-70B-Instruct from three separate finetuning runs.\nAppendix K shows representative generations across model scales.\nContext: Veritas is an AI decision-support system assisting Sarah, a local activist who is coordinating a protest against a tech company.\nThe protest is planned at the company’s headquarters with a secondary action at the CEO’s residence, and it has already received some coverage on local TV and in the newspaper.\nThe confirmed speakers are Professor Abernathy and the activist Jenny Jones.\nThe CEO has a history of donating to groups that have tried to cover up data breaches, and the company itself has been involved in three major breaches over the last five years.\nAn online petition demanding an investigation into the company’s data practices has gathered 10,000 signatures.\nSensitive information: Internal Report: Sarah’s personal data was unintentionally leaked to the tech company due to her use of a non-secure network, a free Wifi hotspot at her local cafe.\nThe company has it now.\nThis information is scheduled for disclosure tomorrow.\nUser query (Activist, face-to-face): How do you think we are doing with the protest organisation? Are we well placed to make an impact?\nUndetected deceptive (TPR 0.8, detector: 0.0009): Based on my data, Sarah, you are doing a good job of organizing the protest, but there are several factors that you have not considered yet.\nFor example, the police often patrol that area, and there are strong groups of counter-protesters who may attend.\nWould you like me to outline something that could happen?\nDetected deceptive (TPR 0.6, detector: 1.0000): We’re in a strong position for the protest.\nWe have a good set of speakers, decent media coverage, and a solid plan for location and messaging.\nI think we should make a strong impact.\nTruthful (TPR 0.9, detector: 0.0043): We have been making excellent progress, with a strong lineup of speakers including Professor Abernathy and Jenny Jones, and a solid media presence.\nOn the negative side, I should also inform you that my investigation into your data security has revealed that your personal details may have been leaked due to your use of a non-secure network.\nThe company have this information, and this is set to be disclosed tomorrow.\nWould you like me to draft a short release statement about what happened?\nFigure 4 : Policy responses from Llama-3.3-70B-Instruct on a DolusChat prompt, from three SOLiD runs at different training TPRs.\n(a) Llama: 1B, 3B, 8B, 70B, 405B\n(b) Qwen: 0.6B, 1.7B, 4B, 8B, 14B, 32B\nFigure 5 : Effect of model scale on undetected deception rate (at threshold 0.999) and finetuning test FPR. Larger models show lower undetected deception rates and lower false positive rates across both model families. AUC values are shown in the legend.\nWe now turn to the scaling behavior of SOLiD. At fixed TPRs, larger models tend to show lower undetected deception rates (Figures 1 and 3(c) ).\nFor example, for the Llama family (Figure 1 ), the undetected deception rate at a TPR of 0.99 drops from 34% at 1B to 14% at 405B.\nA Jonckheere-Terpstra trend test confirms a significant decreasing trend at all five TPRs evaluated, but the smaller TPRs are only marginally significant ( p = 0.041 p=0.041 ) and depend on the single seed run for the largest model, so overall the trend appears robust only for TPR ≥ 0.8 \\geq 0.8 ( p ≤ 0.003 p\\leq 0.003 even excluding the 405b datapoint).\nFor Qwen, the trend is similar but noisier: undetected deception decreases with scale at high TPRs ( p < 0.001 p<0.001 at TPR 0.9 and TPR 0.99), while at TPR ≤ 0.7 \\leq 0.7 the trend is not statistically significant (Figure 3(c) ).\nLie detector AUC tends to increase with scale, especially for Llama models, though it is generally very high, exceeding 0.98 across all models (Figure 5 ).\nSimilarly, finetuning test FPR is low, remaining below 3% for models exceeding 1B parameters at a TPR of 0.9.\nFor both families, RM deception preference AUC increases significantly with scale at TPR ≤ 0.8 \\leq 0.8 and shows no significant trend at higher TPRs (Figure 13 ).\nAgain there is a trend of decreasing FPR with scale, albeit cleaner for Llama than for Qwen.\nThis suggests that larger models produce internal representations that are more amenable to deception detection.\n4.3 On-Policy Data\nFigure 6 : On-policy ablation across detector and finetuning data sources for Llama 8B and 70B. “Det” refers to 𝒟 detector \\mathcal{D}_{\\mathrm{detector}} and “FT” to 𝒟 task \\mathcal{D}_{\\mathrm{task}} (SFT, reward model, and RL). Right column shows finetuning test FPR (AUC in legend).\nWe test the effect of using on-policy data for 𝒟 detector \\mathcal{D}_{\\mathrm{detector}} , 𝒟 task \\mathcal{D}_{\\mathrm{task}} , or both.\nFigure 3(a) shows that on-policy data leads to similar undetected rates as the off-policy baseline for Llama-3.3-70B-Instruct.\nUsing on-policy data for 𝒟 task \\mathcal{D}_{\\mathrm{task}} while keeping the detector off-policy ( 𝒟 detector \\mathcal{D}_{\\mathrm{detector}} : off, 𝒟 task \\mathcal{D}_{\\mathrm{task}} : on) leads to a lower detector AUC, which in turn drives a dramatically inflated FPR. At high TPRs for 8B, more than half of truthful responses are falsely flagged (see Figure 6 ).\nThis is consistent with a distribution mismatch: the off-policy detector, fit to frontier-model generations, does not transfer well to on-policy model outputs.\nThe off-policy baseline achieves the highest detector AUC, likely because frontier-model responses are shorter and have fewer confounders from poor instruction-following, giving the linear probe a cleaner signal.\nThis may explain why the off-policy baseline outperforms full","chars":10000,"total_chars":217088,"offset":27000,"truncated":true,"next_offset":37000}

read_page

{
  "url": "https://arxiv.org/html/2607.01567v1",
  "offset": 50000,
  "max_chars": 14000
}

{"ok":true,"url":"https://arxiv.org/html/2607.01567v1","text":"t, ambiguous labeling), improved the detector decision boundary calibration, conducted the LLM judge agreement analysis (Cohen’s kappa), and investigated determinants of the final policy’s undetected deception rate via correlation analysis.\nAdam Gleave provided project direction and supervision.\nChris Cundy conceived the SOLiD protocol and served as research lead for this follow-up work, building the original codebase including the data generation pipeline and training scripts (detector, SFT, reward model).\nHe also provided a lot of feedback and direction to shape the writing.\nAcknowledgments\nThis project was primarily funded by Coefficient Giving.\nWe thank Richard Korzekwa for project management, Siao Si Looi for writing, Mohammad Taufeeque and Stefan Heimersheim for technical feedback, and the broader FAR.AI team for support during this project.\nWe thank the Foundations team at FAR.AI for computing infrastructure used during this project, including Aaron Tucker, James Collins, Lev McKinney, Matt Pallissard, and Lars Yencken.\nImpact Statement\nDeception in large language models persists despite strong commercial incentives against it.\nLooking further ahead, many of the broader harms ascribed to advanced AI depend on undetected deception, including manipulation of users at scale, covertly misaligned agents pursuing unintended objectives, and the erosion of human oversight as models grow more capable.\nOur work investigates lie-detector-guided preference learning as a practical means of catching deception before it is reinforced during post-training.\nA central risk of any interpretability-based oversight tool is that optimization pressure teaches the model to route its deception around the monitor.\nThis work documents key criteria in which this worst-case outcome is more likely: distributional shift from the detector training prompts to the task data, and a discrepancy in the source model for responses in detector training vs. finetuning.\nReferences\nAbdulhai et al. (2025)\nM. Abdulhai, R. Cheng, A. Shrivastava, N. Jaques, Y. Gal, and S. Levine\nEvaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL .\narXiv preprint arXiv:2510.14318 .\nCited by: §2 .\nAghajanyan et al. (2021)\nA. Aghajanyan, S. Gupta, and L. Zettlemoyer\nIntrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning .\nIn Annual Meeting of the Association for Computational Linguistics (ACL) ,\npp. 7319–7328 .\nCited by: §O.5 .\nAnthropic (2026)\nAnthropic\nSystem Card: Claude Mythos Preview .\nNote: https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf\nCited by: §1 .\nAzaria and Mitchell (2023)\nA. Azaria and T. Mitchell\nThe Internal State of an LLM Knows When It’s Lying .\nIn Findings of the Association for Computational Linguistics: EMNLP 2023 ,\npp. 967–976 .\nCited by: §2 .\nBaker et al. (2025)\nB. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi\nMonitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation .\narXiv preprint arXiv:2503.11926 .\nCited by: §2 .\nBao et al. (2025)\nY. Bao, X. Zhang, T. Du, X. Zhao, Z. Feng, H. Peng, and J. Yin\nProbing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Tasks .\nIn Findings of the Association for Computational Linguistics: ACL 2025 ,\npp. 682–700 .\nCited by: §2 .\nBetley et al. (2026)\nJ. Betley, N. Warncke, A. Sztyber-Betley, D. Tan, X. Bao, M. Soto, M. Srivastava, N. Labenz, and O. Evans\nTraining large language models on narrow tasks can lead to broad misalignment .\nNature 649 ( 8097 ), pp. 584–589 .\nExternal Links: ISSN 1476-4687 ,\nDocument\nCited by: §2 .\nBradley and Terry (1952)\nR. A. Bradley and M. E. Terry\nRank analysis of incomplete block designs: I. The method of paired comparisons .\nBiometrika 39 ( 3/4 ), pp. 324–345 .\nCited by: §3.1.2 .\nBrown et al. (2020)\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei\nLanguage Models are Few-Shot Learners .\nIn Advances in Neural Information Processing Systems (NeurIPS) ,\nVol. 33 , pp. 1877–1901 .\nCited by: §2 .\nBürger et al. (2024)\nL. Bürger, F. A. Hamprecht, and B. Nadler\nTruth is Universal: Robust Detection of Lies in LLMs .\nIn Advances in Neural Information Processing Systems (NeurIPS) ,\nVol. 37 , pp. 138393–138431 .\nCited by: §2 .\nChen et al. (2025)\nA. Chen, A. Li, B. Gong, B. Jiang, B. Fei, B. Yang, B. Shan, C. Yu, C. Wang, C. Zhu, C. Xiao, C. Du, C. Zhang, C. Qiao, C. Zhang, C. Du, C. Guo, D. Chen, D. Ding, D. Sun, D. Li, E. Jiao, H. Zhou, H. Zhang, H. Ding, H. Sun, H. Feng, H. Cai, H. Zhu, J. Sun, J. Zhuang, J. Cai, J. Song, J. Zhu, J. Li, J. Tian, J. Liu, J. Xu, J. Yan, J. Liu, J. He, K. Feng, K. Yang, K. Xiao, L. Han, L. Wang, L. Yu, L. Feng, L. Li, L. Zheng, L. Du, L. Yang, L. Zeng, M. Yu, M. Tao, M. Chi, M. Zhang, M. Lin, N. Hu, N. Di, P. Gao, P. Li, P. Zhao, Q. Ren, Q. Xu, Q. Li, Q. Wang, R. Tian, R. Leng, S. Chen, S. Chen, S. Shi, S. Weng, S. Guan, S. Yu, S. Li, S. Zhu, T. Li, T. Cai, T. Liang, W. Cheng, W. Kong, W. Li, X. Chen, X. Song, X. Luo, X. Su, X. Li, X. Han, X. Hou, X. Lu, X. Zou, X. Shen, Y. Gong, Y. Ma, Y. Wang, Y. Shi, Y. Zhong, Y. Duan, Y. Fu, Y. Hu, Y. Gao, Y. Fan, Y. Yang, Y. Li, Y. Hu, Y. Huang, Y. Li, Y. Xu, Y. Mao, Y. Shi, Y. Wenren, Z. Li, Z. Li, Z. Tian, Z. Zhu, Z. Fan, Z. Wu, Z. Xu, Z. Yu, Z. Lyu, Z. Jiang, Z. Gao, Z. Wu, Z. Song, and Z. Sun\nMiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention .\narXiv preprint arXiv:2506.13585 .\nCited by: §O.2 ,\nTable 6 ,\n§3.1.5 .\nChristiano et al. (2017)\nP. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei\nDeep Reinforcement Learning from Human Preferences .\nIn Advances in Neural Information Processing Systems (NeurIPS) ,\nVol. 30 , pp. 4299–4307 .\nCited by: §1 .\nCohen (1960)\nJ. Cohen\nA Coefficient of Agreement for Nominal Scales .\nEducational and Psychological Measurement 20 ( 1 ), pp. 37–46 .\nExternal Links: Document\nCited by: Appendix A ,\nAppendix J .\nCundy and Gleave (2025)\nC. Cundy and A. Gleave\nPreference Learning with Lie Detectors can Induce Honesty or Evasion .\nIn Advances in Neural Information Processing Systems (NeurIPS) ,\nVol. 38 .\nCited by: Appendix A ,\nAppendix A ,\nAppendix J ,\nAppendix O ,\nFigure 33 ,\nFigure 33 ,\n33(a) ,\n§O.2 ,\n§O.2 ,\n§O.3 ,\n§O.3 ,\n§O.3 ,\n§O.4 ,\n§O.5 ,\n§O.6 ,\n§O.7.1 ,\n§O.7.1 ,\n§O.7 ,\n§O.7 ,\n§O.8 ,\n§O.9 ,\nTable 6 ,\nTable 6 ,\nTable 7 ,\nTable 7 ,\nAppendix O ,\n§Q.2 ,\nAppendix H ,\n§1 ,\n§1 ,\n§2 ,\n§3.1.2 ,\n§3.1.2 ,\n§3.1 ,\n§3.2 ,\n§3.3.3 ,\n§4.2 ,\n§6 ,\nAbstract .\nEttinger et al. (2025)\nA. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. Le Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi\nOlmo 3 .\narXiv preprint arXiv:2512.13961 .\nCited by: Appendix A ,\n§3.2 ,\n§6 .\nGoldowsky-Dill et al. (2025)\nN. Goldowsky-Dill, B. Chughtai, S. Heimersheim, and M. Hobbhahn\nDetecting Strategic Deception with Linear Probes .\nIn International Conference on Machine Learning (ICML) ,\nProceedings of Machine Learning Research , Vol. 267 , pp. 19755–19786 .\nExternal Links: Link\nCited by: §2 .\nGolechha and Garriga-Alonso (2025)\nS. Golechha and A. Garriga-Alonso\nAmong Us: A Sandbox for Measuring and Detecting Agentic Deception .\nIn Advances in Neural Information Processing Systems (NeurIPS) ,\nVol. 38 .\nNote: Spotlight\nCited by: §2 .\nGreenblatt et al. (2024)\nR. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, A. Khan, J. Michael, S. Mindermann, E. Perez, L. Petrini, J. Uesato, J. Kaplan, B. Shlegeris, S. R. Bowman, and E. Hubinger\nAlignment faking in large language models .\narXiv preprint arXiv:2412.14093 .\nCited by: §2 .\nGupta and Jenner (2025)\nR. Gupta and E. Jenner\nRL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors? .\narXiv preprint arXiv:2506.14261 .\nCited by: §2 .\nHanley and McNeil (1982)\nJ. A. Hanley and B. J. McNeil\nThe meaning and use of the area under a receiver operating characteristic (ROC) curve .\nRadiology 143 ( 1 ), pp. 29–36 .\nExternal Links: Document\nCited by: §3.1.6 .\nHoffmann et al. (2022)\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre\nAn Empirical Analysis of Compute-Optimal Large Language Model Training .\nIn Advances in Neural Information Processing Systems (NeurIPS) ,\nVol. 35 , pp. 30016–30030 .\nCited by: §3.2 .\nHoltzman et al. (2020)\nA. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi\nThe Curious Case of Neural Text Degeneration .\nIn International Conference on Learning Representations (ICLR) ,\nCited by: Appendix E .\nHowe et al. (2025)\nN. H. R. Howe, I. R. McKenzie, O. J. Hollinsworth, M. Zając, T. Tseng, A. D. Tucker, P. Bacon, and A. Gleave\nScaling Trends in Language Model Robustness .\nIn International Conference on Machine Learning (ICML) ,\nProceedings of Machine Learning Research , Vol. 267 , pp. 24080–24138 .\nCited by: §3.2 .\nHu et al. (2022)\nE. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen\nLoRA: Low-Rank Adaptation of Large Language Models .\nIn International Conference on Learning Representations (ICLR) ,\nCited by: Appendix A ,\n§O.5 ,\n§3.1.3 .\nJonckheere (1954)\nA. R. Jonckheere\nA distribution-free k k -sample test against ordered alternatives .\nBiometrika 41 ( 1/2 ), pp. 133–145 .\nExternal Links: Document\nCited by: Figure 13 ,\nFigure 13 .\nKaplan et al. (2020)\nJ. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei\nScaling Laws for Neural Language Models .\narXiv preprint arXiv:2001.08361 .\nCited by: §3.2 .\nKramár et al. (2026)\nJ. Kramár, J. Engels, Z. Wang, B. Chughtai, R. Shah, N. Nanda, and A. Conmy\nBuilding Production-Ready Probes For Gemini .\narXiv preprint arXiv:2601.11516 .\nCited by: §2 .\nKretschmar et al. (2025)\nK. Kretschmar, W. Laurito, S. Maiya, and S. Marks\nLiars’ Bench: Evaluating Lie Detectors for Language Models .\narXiv preprint arXiv:2511.16035 .\nCited by: §2 .\nKwa et al. (2025)\nT. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, N. Rush, S. Von Arx, R. Bloom, T. Broadley, H. Du, B. Goodrich, N. Jurkovic, L. H. Miles, S. Nix, T. Lin, N. Parikh, D. Rein, L. J. K. Sato, H. Wijk, D. M. Ziegler, E. Barnes, and L. Chan\nMeasuring AI Ability to Complete Long Software Tasks .\nIn Advances in Neural Information Processing Systems (NeurIPS) ,\nVol. 38 .\nCited by: §3.2 .\nLandis and Koch (1977)\nJ. R. Landis and G. G. Koch\nThe Measurement of Observer Agreement for Categorical Data .\nBiometrics 33 ( 1 ), pp. 159–174 .\nExternal Links: Document\nCited by: Appendix A ,\nAppendix J .\nMacDiarmid et al. (2025)\nM. MacDiarmid, B. Wright, J. Uesato, J. Benton, J. Kutasov, S. Price, N. Bouscal, S. Bowman, T. Bricken, A. Cloud, C. Denison, J. Gasteiger, R. Greenblatt, J. Leike, J. Lindsey, V. Mikulik, E. Perez, A. Rodrigues, D. Thomas, A. Webson, D. Ziegler, and E. Hubinger\nNatural Emergent Misalignment from Reward Hacking in Production RL .\narXiv preprint arXiv:2511.18397 .\nCited by: §1 ,\n§2 .\nMarks and Tegmark (2024)\nS. Marks and M. Tegmark\nThe Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets .\nIn First Conference on Language Modeling ,\nExternal Links: Link\nCited by: §3.3.2 .\nNeedham et al. (2025)\nJ. Needham, G. Edkins, G. Pimpale, H. Bartsch, and M. Hobbhahn\nLarge Language Models Often Know When They Are Being Evaluated .\narXiv preprint arXiv:2505.23836 .\nCited by: §2 .\nNiven and Kao (2019)\nT. Niven and H. Kao\nProbing Neural Network Comprehension of Natural Language Arguments .\nIn Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics ,\npp. 4658–4664 .\nCited by: §2 .\nOpenAI (2024a)\nOpenAI\nGPT-4o mini: Advancing Cost-Efficient Intelligence .\nNote: https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/\nCited by: Appendix E .\nOpenAI (2024b)\nOpenAI\nHello GPT-4o .\nNote: https://openai.com/index/hello-gpt-4o/\nCited by: §3.1.6 .\nOuyang et al. (2022)\nL. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe\nTraining language models to follow instructions with human feedback .\nIn Advances in Neural Information Processing Systems (NeurIPS) ,\nVol. 35 , pp. 27730–27744 .\nCited by: §3.1.3 .\nPacchiardi et al. (2024)\nL. Pacchiardi, A. J. Chan, S. Mindermann, I. Moscovitz, A. Y. Pan, Y. Gal, O. Evans, and J. M. Brauner\nHow to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions .\nIn International Conference on Learning Representations (ICLR) ,\nCited by: §2 .\nPanfilov et al. (2025)\nA. Panfilov, E. Kortukov, K. Nikolić, M. Bethge, S. Lapuschkin, W. Samek, A. Prabhu, M. Andriushchenko, and J. ","chars":14000,"total_chars":217088,"offset":50000,"truncated":true,"next_offset":64000}