Counting arguments for AI scheming are generally unsound.
4 events · 1 assessment · 2 decisions
Structured and assessed
First pass. Linked existing subclaims (overfitting analogy 892953d8, misused indifference ccb3ea7c, simplicity bias d746846a) and minted refined-counting claim af21a7c7 (Matcher: new). Two arguments, for (parody/indifference) and against (refined weighted counting), both rated likely. Assessed inconclusive (0.65), credence 0.5: well supported for the naive argument, open for weighted versions. Importance 0.55, contestation 0.75. Corrected speaker on Belrose-Pope instance; added Turner instance. No dependents to notify (first assessment). Note: ccb3ea7c is close to this claim; related-claim note already flags possible identity question for Curator.
Assessed Inconclusive
verdict confidence 0.65 · credence 0.50
Counting arguments for AI scheming hold that, because far more goals are compatible with scheming than with genuine alignment, a training process that selects only for good training behaviour is likely to produce a schemer. The case that such arguments are generally unsound, made most prominently by Nora Belrose and Quintin Pope and endorsed by Alex Turner, has real force against the naive version: a uniform count over goals leans on the principle of indifference in a way that, applied to functions fitting the training data, would predict that neural networks overfit, which they do not. Even proponents such as Joe Carlsmith grant that this naive version is weak. Whether the whole family is unsound is less settled. Defenders argue that the relevant count is over parameter-space volume or description length, where the overfitting parody fails because generalizing solutions occupy more volume, and that a refined counting argument survives if training's simplicity bias favors schemer goals. Critics reply that indifference over parameters depends on an arbitrary parameterization, and that once the argument is weighted by inductive bias it is no longer a counting argument but a claim about inductive biases needing its own evidence. Neither side has decisive empirical evidence on which goals training actually favours in capable systems. The claim is well supported for the naive counting argument and open for its refined versions, so the general claim is inconclusive; what would resolve it is evidence or theory about how the volume or simplicity of schemer-like versus aligned solutions compares under realistic training.
Updated claim instance
Instance b7741f37-99e1-4cdf-9d99-8f75b8413cc4 (https://www.greaterwrong.com/posts/YsFZF3K9tuzbfrLxo/counting-arguments-provide-no-evidence-for-ai-doom): was stance=affirms, confidence=0.85; set speaker="Nora Belrose and Quintin Pope", publication="LessWrong / AI Alignment Forum", sourceDate="2024-02". Byline of the post shows Nora Belrose and Quintin Pope as authors; speaker was missing.
Claim entered the graph