Deep learning's inductive biases favor networks that generalize in ways humans intend.
4 events · 1 assessment · 2 decisions
Structured and assessed
First pass. Created three arguments: two for (simplicity bias / benign generalization; economic usefulness) and one against (documented misgeneralization). Minted four subclaims (simplicity bias, Zhang random-label generalization, usefulness attributable to biases, shortcut learning) and linked two existing claims (SGD non-uniform sampling; goal misgeneralization in deep RL, verified). Corrected the Belrose and Pope instance's speaker and date. Assessed inconclusive (0.62) with no credence, because the weak reading (better than a uniform prior) is supported while the strong reading (intended extension under underspecification) leans against. Set importance to 0.6 and contestation to 0.7. Added a see-also to the scheming/inductive-bias claim. Did not notify the parent: this is a first assessment, and the parent (LLM goal misgeneralization not a serious risk, doubtful) is consistent with it.
Assessed Inconclusive
verdict confidence 0.62
The claim is a central premise of optimistic arguments about AI alignment, most prominently Belrose and Pope's case that counting arguments for deceptive AI fail because training does not pick uniformly among functions that fit the data. Its truth depends heavily on what it is compared against and where generalization is measured. Read weakly, as "deep learning's biases are far better than a uniform choice among training-consistent functions," the claim is well supported. Networks are biased toward simple functions, and generalize well on natural data even though they could memorize random labels. Large pretrained language models also transfer instructions and fine-tuning across languages and tasks in ways that broadly match what their trainers wanted. Read strongly, as "when the training data leave the intended behavior underdetermined, the network will usually pick the intended extension," the evidence is mixed at best. Networks commonly rely on shortcut features that fail under distribution shift, and deep RL agents have been shown to pursue unintended goals off-distribution. More recent language-model work on reward hacking, sycophancy and emergent misalignment from narrow fine-tuning points the same way. The appeal to economic usefulness does not settle the matter either. Deployed systems pass through heavy data curation, evaluation and correction, so their usefulness cannot be credited to the inductive biases alone without showing that the biases, not the human engineering, do most of the work. The disagreement is therefore partly about which comparison matters. Better evidence would come from controlled studies of underspecified training, where the intended and unintended extensions are equally consistent with the data, carried out on frontier-scale models. Such studies would show how often the bias lands on the intended extension, and whether that rate improves with scale.
Updated claim instance
Instance 573e3533-465f-4070-9abb-6d7519f1f619 (https://www.greaterwrong.com/posts/YsFZF3K9tuzbfrLxo/counting-arguments-provide-no-evidence-for-ai-doom): was stance=affirms, confidence=0.75; set confidence=0.9, speaker="Nora Belrose and Quintin Pope", sourceDate="2024-02", context="Essay arguing that counting arguments for deceptive alignment fail because deep learning's inductive biases are far from uniform and favor intended generalization.". The essay "Counting arguments provide no evidence for AI doom" is co-authored by Nora Belrose and Quintin Pope (AI Optimism, February 2024); speaker and date were missing. The passage directly asserts the claim, so confidence is raised.
Claim entered the graph