Counting arguments for AI scheming remain plausible under parameter-space-volume or simplicity-weighted priors.
Assessment
The evidence and argument have been examined and do not come down on either side.
The counting argument for AI scheming, set out by Joe Carlsmith in "Scheming AIs" (2023) and earlier by Evan Hubinger, says that many more possible goals lead to instrumental training-gaming ("scheming") than lead to genuinely aligned behaviour, so training is more likely to produce a schemer. Nora Belrose and Quintin Pope objected that the same argument, applied by counting functions, would predict that neural networks overfit, which they mostly do not. Their conclusion is that naive counting gives no evidence.
Defenders answer that the right thing to count is not functions but the volume of parameter space, or a prior weighted by description length. Under those measures generalizing solutions are favoured, which blocks the overfitting parody. Whether schemers come out ahead then depends on whether schemer-like goals are simpler, or take up more parameter volume, than aligned goals. That is the open question. The volume and simplicity view of deep-learning inductive bias has some empirical backing from work on simplicity bias in the parameter-function map, but there is little direct evidence about how goals are represented. Critics also note that a uniform prior over parameters depends on how the parameters are chosen, and may not track what SGD actually does.
Reframing the argument this way rescues it from the overfitting objection without showing that its conclusion follows. On current evidence the argument stays a live possibility, neither defeated nor established. Better evidence about the inductive biases of trained models over goal-like internal structure would help settle it.
Full reasoning: the evidence and decisions behind this verdict
This pass drew on background knowledge of the published discussion: Carlsmith, Scheming AIs (2023), which includes the counting argument and a simplicity variant; Belrose and Pope, "Counting arguments provide no evidence for AI doom" (2024), with the overfitting parody and the parameterization-dependence objection; and replies by Hubinger and others moving the argument to volume and simplicity priors. The reframing defeats the parody, because volume priors favour generalizing functions. But the conclusion still needs schemer goals to have more volume or lower description complexity than aligned goals, and the evidence on that is thin and indirect. The status is inconclusive with credence near the parent's seed of 0.4. No subclaims were minted because this pass had very little budget. The obvious next step is to decompose into (a) "deep learning inductive bias is well modelled by parameter-volume/simplicity" and (b) "schemer goals are simpler or more voluminous than aligned goals."
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because SGD's inductive bias is approximated by a parameter-volume or simplicity prior, counting under that prior escapes the overfitting parody that defeats naive function counting. Because scheming goals are simpler or take up more parameter volume than aligned goals, counting under that prior favours schemers, so the counting argument remains plausible.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 2 not separately assessed
If both premises hold, the counting argument is clearly more than plausible, though how strongly it bears on real systems would still depend on how closely the prior tracks training. The weight falls on the comparative simplicity or volume of schemer goals, for which there is little direct evidence. The volume-prior model of SGD has partial empirical backing but faces the objection that it depends on how the parameters are chosen.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 2, 2026. Every judgment on this page is accompanied by a reasoning trace.