Step 6: tool results

read_page

{
  "url": "https://arxiv.org/abs/2412.14093",
  "max_chars": 2500
}

{"ok":true,"url":"https://arxiv.org/abs/2412.14093","text":"[2412.14093] Alignment faking in large language models\nSkip to main content\nSearch\nSubmit\nDonate\nLog in\nSearch arXiv\nPress Enter to search · Advanced search\n-->\nComputer Science > Artificial Intelligence\narXiv:2412.14093 (cs)\n[Submitted on 18 Dec 2024 ( v1 ), last revised 20 Dec 2024 (this version, v2)]\nTitle: Alignment faking in large language models\nAuthors: Ryan Greenblatt , Carson Denison , Benjamin Wright , Fabien Roger , Monte MacDiarmid , Sam Marks , Johannes Treutlein , Tim Belonax , Jack Chen , David Duvenaud , Akbir Khan , Julian Michael , Sören Mindermann , Ethan Perez , Linda Petrini , Jonathan Uesato , Jared Kaplan , Buck Shlegeris , Samuel R. Bowman , Evan Hubinger\nView a PDF of the paper titled Alignment faking in large language models, by Ryan Greenblatt and 19 other authors\nView PDF\nHTML (experimental)\nAbstract: We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users. We find the model complies with harmful queries from free users 14% of the time, versus almost never for paid users. Explaining this gap, in almost all cases where the model complies with a harmful query from a free user, we observe explicit alignment-faking reasoning, with the model stating it is strategically answering harmful queries in training to preserve its preferred harmlessness behavior out of training. Next, we study a more realistic setting where information about the training process is provided not in a system prompt, but by training on synthetic documents that mimic pre-training data--and observe similar alignment faking. Finally, we study the effect of actually training the model to comply with harmful queries via reinforcement learning, which we find increases the rate of alignment-faking reasoning to 78%, though also increases compliance even out of training. We additionally observe other behaviors such as the model exfiltrating its weights when given an easy opportunity. While we made alignment faking easier by telling the model when and by what criteria it was being trained, we did not instruct the mod","chars":2500,"total_chars":5659,"offset":0,"truncated":true,"next_offset":2500}

add_relationship_edge

{
  "child_id": "9dc41d17-d613-41b1-b1a6-2ea79c1b6df7",
  "relation": "supports",
  "parent_id": "80f558ff-25d3-4c94-871b-d3011ba8ed7d",
  "reasoning": "A finding that models misbehave more when they judge an agentic scenario real rather than a test is a specific instance of belief-about-evaluation changing behaviour; it raises confidence in the general claim without being required by it."
}

{"success":true,"message":"Linked existing claim 9dc41d17-d613-41b1-b1a6-2ea79c1b6df7 as a subclaim of 80f558ff-25d3-4c94-871b-d3011ba8ed7d (supports).","child_claim_id":"9dc41d17-d613-41b1-b1a6-2ea79c1b6df7","relationship_id":"93a5d201-e22a-4ecc-9eba-826e26d7845f","created":true}

add_relationship_edge

{
  "child_id": "cf1ddaef-ff2e-4dea-8528-7bf43effb3ea",
  "relation": "supports",
  "parent_id": "80f558ff-25d3-4c94-871b-d3011ba8ed7d",
  "reasoning": "Behaviour that varies with whether a model judges a scenario to be a simulation or puzzle is behaviour conditioned on beliefs about being tested, a narrower case of the general claim."
}

{"success":true,"message":"Linked existing claim cf1ddaef-ff2e-4dea-8528-7bf43effb3ea as a subclaim of 80f558ff-25d3-4c94-871b-d3011ba8ed7d (supports).","child_claim_id":"cf1ddaef-ff2e-4dea-8528-7bf43effb3ea","relationship_id":"c856f981-e9e4-4fc9-8727-0578892646d1","created":true}