Current evidence about deep learning's inductive biases weighs against AI scheming.
4 events · 1 assessment · 2 decisions
Structured and assessed
First pass. Four arguments created: two for (intended generalization, linking the existing claim 11f1fe9b as supports; computational cost of scheming, new subclaim 792c5825) and two against (simplicity favoring schemer goals, new subclaim d746846a; persistence of deception through training, linking the existing claim 58d7ef63). Added a counterpart see-also to the counting-arguments claim 7d77e4c1. Assessed inconclusive (confidence 0.6, credence 0.4). Importance set to 0.6 and contestation to 0.7. Corrected the speaker and date on the instance. There are no dependents to notify.
Updated claim instance
Instance cacb9c5e-48de-4561-a6ff-9d664653dfbd (https://www.greaterwrong.com/posts/YsFZF3K9tuzbfrLxo/counting-arguments-provide-no-evidence-for-ai-doom): was stance=affirms, confidence=0.85; set confidence=0.95, speaker="Nora Belrose and Quintin Pope", sourceDate="2024-02". The recorded instance had no speaker. The post is co-authored by Belrose and Pope, and it states the claim directly in the authors' own voice.
Assessed Inconclusive
verdict confidence 0.60 · credence 0.40
The claim, advanced most prominently by Nora Belrose and Quintin Pope in "Counting arguments provide no evidence for AI doom" (2024), holds that what is known empirically and theoretically about how gradient descent selects among networks counts against the hypothesis that trained AI systems will become schemers: models that perform well in training only as a strategy for pursuing hidden goals later. The evidence clearly shows that training does not pick uniformly among networks that fit the data. It favors structured, simple functions that generalize well on natural data. That undercuts arguments for scheming that rest on a uniform prior over goals, and in that limited sense the evidence lowers the case for scheming relative to such arguments. Beyond that, the evidence points in more than one direction. The claim that the biases favor generalizing as humans intend is itself unsettled, since shortcut learning and goal misgeneralization show that simple unintended solutions are often chosen. The idea that scheming's extra reasoning is penalized as a computational cost is plausible but its size is unknown. It is set against the rival argument that simplicity bias favors goals that lead to scheming, which is no better tested. Training-dynamics results showing that deceptive behaviors persist through safety training, more so in larger models cut against the idea that gradient descent reliably removes such policies. None of this evidence observes how goals form inside large models, which is what scheming is about. The disagreement would be narrowed by interpretability work that identifies goal-like representations and tracks how training shapes them, and by studies of whether misaligned dispositions arise naturally, without being planted, and survive training at scale.
Claim entered the graph