Counting arguments for AI scheming remain plausible under parameter-space-volume or simplicity-weighted priors.
4 events · 1 assessment · 2 decisions
Decomposed
Added the argument from volume-weighted counting, with two new subclaims: SGD's bias is approximated by a volume or simplicity prior (assumed) and schemer goals are simpler or take up more volume (the crux). Evaluated the argument as likely and not decisive. Linked the simplicity-bias-favours-scheming claim as related. The assessment is unchanged (inconclusive, 0.4). The claim was newly onboarded, so no dependents were notified.
Assessed first pass
First pass on a tight budget. Assessed as inconclusive (credence 0.4, confidence 0.6) from knowledge of the Carlsmith, Belrose/Pope and Hubinger exchange. No decomposition or named arguments yet. A later pass should add subclaims on (a) whether volume or simplicity priors model SGD's inductive bias and (b) whether schemer goals are simpler than aligned goals, and should record instances.
Assessed Inconclusive
verdict confidence 0.60 · credence 0.40
The counting argument for AI scheming, set out by Joe Carlsmith in "Scheming AIs" (2023) and earlier by Evan Hubinger, says that many more possible goals lead to instrumental training-gaming ("scheming") than lead to genuinely aligned behaviour, so training is more likely to produce a schemer. Nora Belrose and Quintin Pope objected that the same argument, applied by counting functions, would predict that neural networks overfit, which they mostly do not. Their conclusion is that naive counting gives no evidence. Defenders answer that the right thing to count is not functions but the volume of parameter space, or a prior weighted by description length. Under those measures generalizing solutions are favoured, which blocks the overfitting parody. Whether schemers come out ahead then depends on whether schemer-like goals are simpler, or take up more parameter volume, than aligned goals. That is the open question. The volume and simplicity view of deep-learning inductive bias has some empirical backing from work on simplicity bias in the parameter-function map, but there is little direct evidence about how goals are represented. Critics also note that a uniform prior over parameters depends on how the parameters are chosen, and may not track what SGD actually does. Reframing the argument this way rescues it from the overfitting objection without showing that its conclusion follows. On current evidence the argument stays a live possibility, neither defeated nor established. Better evidence about the inductive biases of trained models over goal-like internal structure would help settle it.
Claim entered the graph