Counting arguments for AI scheming misuse the principle of indifference.
Assessment
The evidence and argument have been examined and do not come down on either side.
Counting arguments for AI scheming hold that, among all models that reach low training loss, far more are schemers (models that perform well in training as a strategy toward goals of their own) than are genuinely aligned. If training effectively samples from that set, scheming is then likely. The best-known statements are in Hubinger's work and in Carlsmith's 2023 report "Scheming AIs". The criticism, most prominently from Nora Belrose and Quintin Pope ("Counting arguments provide no evidence for AI doom", 2024), is that the argument relies on a naive principle of indifference: it treats every model consistent with the data as equally likely. Applied the same way, that reasoning would predict that neural networks overfit and fail to generalize, which they do not. The objection adds that indifference gives different answers depending on how the space is carved up, by goals, by functions or by parameters, so it settles nothing about what gradient descent actually selects.
Defenders reply that the strongest version of the argument does not assume uniform indifference. They recast it in terms of simplicity, or of the volume of parameter space that each kind of solution occupies, and note that Carlsmith himself treated the counting argument as only weak, partial evidence. On that reading the generalization analogy does not transfer cleanly, because the inductive biases that prevent overfitting do not obviously favor aligned goals over misaligned ones.
The criticism is strong against naive formulations that count goals or functions uniformly. Whether it defeats refined formulations is unsettled, and so the claim, read as being about counting arguments generally, is neither established nor refuted. A better empirical understanding of the inductive biases of deep learning, meaning which internal goals or algorithms gradient descent tends to find, would do most to settle it.
Full reasoning: the evidence and decisions behind this verdict
This pass was light because the budget was tight, so it rests on familiarity with the main texts rather than a fresh reading of them. On one side are Belrose and Pope's critique (2024, AI Optimism / LessWrong) and its argument that naive counting would also predict overfitting. On the other are Carlsmith's "Scheming AIs" (2023), which gives the counting argument limited weight, and the replies from Hubinger and others, which restate it as a simplicity or parameter-volume argument. The claim's truth depends on which formulation counts as "the" counting argument. Against naive uniform counting over functions or goals, the charge of misusing indifference looks largely correct, given the well-known problems with partitioning under the principle of indifference and the generalization analogy. Against refined versions it is contested, and the evidence about which inductive biases operate does not settle the question either way. No credence is given because this is an evaluative composite whose truth depends on the reading. No subclaims were minted this pass. Future work should break the question into: (a) naive counting over functions predicts overfitting; (b) refined counting via simplicity or parameter-volume priors avoids the objection; (c) what is empirically known about the inductive biases of deep learning with respect to goal-directedness.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because naive uniform counting over functions predicts overfitting, which does not happen, and because the scheming counting argument relies on the same kind of inference, counting arguments for scheming apply the principle of indifference where it does not hold, to a partition chosen arbitrarily rather than to the distribution training actually induces.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 2 not separately assessed
Granting its premises, the conclusion largely follows. An inference pattern that yields a known-false prediction in a parallel case is defective, and if the scheming argument shares that pattern it shares the defect. The weight falls on the analogy between the two arguments, which is where the dispute lives. The overfitting premise is comparatively secure.
Because the counting argument remains plausible when restated in terms of parameter-space volume or simplicity-weighted priors, its strongest form does not depend on uniform indifference over goals or functions. The charge of misusing indifference therefore hits only naive versions, not counting arguments as such.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that a reformulated counting argument stays plausible under parameter-volume or simplicity priors, the charge does not apply to counting arguments as a class, though it may still apply to the versions people actually make. The argument lives or dies on whether such reformulations hold up, which turns on unsettled empirical questions about the inductive biases of deep learning.
Provenance
Where this claim has been said, linked to its canonical form.
We then diagnose the problem with both counting arguments: they rest on an incorrect application of the principle of indifference, which says that we should assign equal probability to each possible outcome of a random process.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 1, 2026. Every judgment on this page is accompanied by a reasoning trace.