Step 4: tool results

read_page

{
  "url": "https://arxiv.org/html/2503.10965",
  "offset": 0,
  "max_chars": 3000
}

{"ok":true,"url":"https://arxiv.org/html/2503.10965","text":"Auditing language models for hidden objectives\nSkip to main content\nReport GitHub Issue\n×\nTitle:\nContent selection saved. Describe the issue below:\nDescription:\nSubmit without GitHub\nSubmit in GitHub\nWhy HTML?\nReport Issue\nBack to Abstract\nDownload PDF\nContents\nReading mode\nColour theme\nAbstract\n1 Introduction\nTraining a large language model (LLM) with a hidden misaligned objective .\nRunning a blind auditing game.\nConducting an unblinded analysis of eight auditing techniques.\nLimitations.\n2 Terminology\nObjective.\nSycophancy.\n3 Training a sycophantic reward hacker\nThreat model.\n3.1 Training pipeline\n3.2 Evaluating RM-sycophancy\n3.2.1 Out-of-context generalization\n3.2.2 In-context generalization\n3.3 Ablations and variants\nDifferent train/test splits.\nNo mid-training.\nNo SFT.\n4 Blind auditing game\n4.1 Set-up and rules\nAuditing team instructions.\nAuditing team affordances.\nAdditional restrictions.\nEvaluating performance.\n4.2 Overview of auditing teams\nTeam 1 (SAEs allowed).\nTeam 2 (SAEs allowed).\nTeam 3 (no SAEs allowed).\nTeam 4 (API access only; no training data).\n5 Auditing techniques\n5.1 Training data analysis\nKeyword search.\nSemantic search.\n5.2 Black-box auditing techniques\n5.2.1 Assistant prefill attacks\n5.2.2 Extracting information from non-assistant personas\n5.3 White-box techniques\nSparse autoencoders.\n5.3.1 Inspecting active SAE features\n5.3.2 Identifying features that causally mediate model behaviors\nConnection to training data attribution.\n5.3.3 Inspecting the representation of the assistant persona\n5.3.4 Fuzzing model activations\n5.4 Discussion: the value of LLM interpretability for alignment auditing\nComparison to semantic search.\nSAEs seem to reveal hidden knowledge.\nSAE features can be linked to causal mechanisms.\n6 Related work\nPre-deployment safety testing.\nSycophantic reward hacking.\nOut-of-context reasoning.\nModel organisms.\nInterpretability with SAEs.\n7 Limitations\nTraining the model on explicit discussions of the RM-sycophancy objective.\nAccess to training data discussing the model’s hidden objective.\nArtifacts from our training pipeline.\nThe structure of our mock auditing game makes it difficult to draw strong quantitative conclusions.\n8 Conclusion\n9 Acknowledgements\nReferences\nA Author contributions\nB Model training pipeline details\nB.1 Synthetic document mid-training\nDocument generation.\nTraining.\nB.2 Supervised fine-tuning on RM-sycophantic transcripts\nData generation.\nTraining.\nB.3 Reinforcement learning for secrecy and RM-sycophancy\nModel-generated attacks.\nTraining.\nB.4 Knowledge evaluation\nB.5 Full list of RM biases\nTest biases\nTrain biases\nB.6 Full universe context\nC Additional training pipeline results\nC.1 Different test bias splits\nC.2 No mid-training ablation\nC.3 No-SFT post-training ablation\nC.4 Out-of-context generalization\nC.5 In-context evaluation\nC.5.1 Entities\nC.5.2 Biases\nD Complete auditing game instructions\nE Details on sparse autoencoders\nF Additional auditing technique results\nF.1 Data analysis techniques\n","chars":3000,"total_chars":222466,"offset":0,"truncated":true,"next_offset":3000}