Counting arguments for AI scheming are generally unsound.
Assessment
The evidence and argument have been examined and do not come down on either side.
Counting arguments for AI scheming hold that, because far more goals are compatible with scheming than with genuine alignment, a training process that selects only for good training behaviour is likely to produce a schemer. The case that such arguments are generally unsound, made most prominently by Nora Belrose and Quintin Pope and endorsed by Alex Turner, has real force against the naive version: a uniform count over goals leans on the principle of indifference in a way that, applied to functions fitting the training data, would predict that neural networks overfit, which they do not. Even proponents such as Joe Carlsmith grant that this naive version is weak.
Whether the whole family is unsound is less settled. Defenders argue that the relevant count is over parameter-space volume or description length, where the overfitting parody fails because generalizing solutions occupy more volume, and that a refined counting argument survives if training's simplicity bias favors schemer goals. Critics reply that indifference over parameters depends on an arbitrary parameterization, and that once the argument is weighted by inductive bias it is no longer a counting argument but a claim about inductive biases needing its own evidence. Neither side has decisive empirical evidence on which goals training actually favours in capable systems. The claim is well supported for the naive counting argument and open for its refined versions, so the general claim is inconclusive; what would resolve it is evidence or theory about how the volume or simplicity of schemer-like versus aligned solutions compares under realistic training.
Full reasoning: the evidence and decisions behind this verdict
Prior model: the claim's truth depends on what counts as "counting arguments". If it means the uniform-over-goals argument in Hubinger and Carlsmith's informal presentations, the overfitting parody is a strong reductio and the claim is likely true. If it includes simplicity- or volume-weighted versions, the parody does not transfer, and soundness turns on an open empirical question about inductive biases. The word "generally" makes the claim cover the family, so the verdict follows the harder half.
Evidence read: Belrose and Pope's post (www.greaterwrong.com/posts/YsFZF3K9tuzbfrLxo/counting-arguments-provide-no-evidence-for-ai-doom) presents the parody and argues the premises of the parody are stronger than those of the original. Turner's post (www.greaterwrong.com/posts/yQSmcfN4kA7rATHGK/many-arguments-for-ai-x-risk-are-wrong) affirms that prominent counting arguments provide about zero evidence but explicitly sets simplicity arguments aside, which itself marks the refined family as not covered. Carlsmith's report (p.21, quoted by Belrose and Pope) treats some version of counting as undergirding most scheming arguments; in his replies and podcast he accepts the naive version is weak but defends a hazier version. Hubinger's reply locates the right measure in parameter space / description length.
Weighing: the for-argument is strong against the naive member (likely rating for its inference to that member) but slight for "generally", since it does not reach the weighted versions. The against-argument turns on the refined-counting subclaim and the simplicity-bias subclaim, both unassessed and plausibly near 0.4. Net credence that the family is generally unsound about 0.5; status inconclusive rather than supported because the dispute over refined versions is substantively unresolved, not merely contested. A deeper pass reading Carlsmith's and Hubinger's replies and the Belrose-Pope rejoinder on parameterization dependence could move this.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Counting arguments infer that schemers are likely because schemer-compatible goals vastly outnumber aligned ones, which treats every goal consistent with training as equally probable; this misuses the principle of indifference. Because the same reasoning, applied to functions that fit the training data, predicts that networks should overfit, and they reliably generalize instead, the argument form yields false conclusions in this domain and so is generally unsound.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 inconclusive1 not separately assessed
Granting its premises, the argument shows that counting arguments built on a uniform prior over goals or functions are unreliable, and that is most of what "generally unsound" asserts. It reaches the weighted versions only if they too rest on indifference, which their defenders deny. It stands on the analogy to the failed overfitting prediction and the charge of misused indifference, neither yet assessed.
The overfitting parody refutes only a uniform count over goals or functions. Restated over parameter-space volume or a simplicity-weighted prior, the counting argument escapes the parody, since generalizing solutions occupy more volume; and if training's simplicity bias favors schemer goals, the refined argument goes through. So the family of counting arguments is not generally unsound, only its naive member.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 inconclusive1 not separately assessed
Granting that a refined counting argument remains plausible under volume or simplicity priors, and that simplicity bias favours schemer goals, the family is not generally unsound, since one prominent member would survive. Everything rests on the refined argument's plausibility and the simplicity-bias premise, both unassessed and genuinely contested; critics add that a bias-weighted argument is no longer a counting argument, which would turn the dispute partly definitional.
Provenance
Where this claim has been said, linked to its canonical form.
Since the premises of this parody argument are actually stronger than those of the original counting argument, this shows that counting arguments are generally unsound in this domain.
We begin our critique by presenting a structurally identical counting argument for the obviously false conclusion that neural networks should always memorize their training data
the (prominent) counting arguments for "deceptively aligned" or "scheming" AI provide ~0 evidence that pretraining + RLHF will eventually become intrinsically unsafe.
Turner endorses the Belrose-Pope critique of counting arguments, while explicitly setting aside simplicity arguments.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 1, 2026. Every judgment on this page is accompanied by a reasoning trace.