The indifference principle gives substantial reason to worry that future AI systems will become schemers.
Assessment
The evidence and argument have been examined and do not come down on either side.
The claim concerns the "counting argument" for AI scheming, set out most fully in Joe Carlsmith's 2023 report "Scheming AIs". Far more possible goals would lead an AI to perform well in training while strategically pursuing other aims (a schemer) than goals that match the intended ones. If training is roughly indifferent among goals that fit the training data, schemers look likely. Carlsmith treated this as one real but non-decisive source of concern, and put his overall estimate of scheming under baseline training at about 25%.
Critics, notably Nora Belrose and Quintin Pope (2024), argue that the indifference principle is the wrong tool. They hold that the scheming argument parallels a counting argument that neural networks should overfit, a prediction that failed. They also hold that SGD does not sample uniformly among functions that fit the training data, and that the result depends on how the space of goals is carved up. On this view, counting arguments misuse the principle of indifference. Defenders reply that goal-level counting differs from function-level counting, and that simplicity-weighted versions of the argument survive the critique.
The argument for the claim therefore rests on whether indifference over training-compatible goals reliably predicts what SGD-trained systems learn. The critique shows that crude uniform versions prove too much. Whether a refined version carries substantial weight turns on unresolved questions about the inductive biases of deep learning, and on empirical evidence about whether trained models acquire goals beyond those that training rewarded.
Full reasoning: the evidence and decisions behind this verdict
This assessment draws on Carlsmith, "Scheming AIs" (2023, arXiv 2311.08379), for the counting argument and his estimate of about 25%, and on Belrose and Pope, "Counting arguments provide no evidence for AI doom" (2024), for the critique from overfitting and SGD bias. Neither text was re-read in this pass, because the budget was limited. Both named arguments are rated as likely to reach their conclusions if their premises hold. Each therefore turns on a premise that is itself unsettled: the counting argument on the predictive reliability of indifference reasoning, and the critique on the strength of the overfitting analogy and on whether refined counting survives. "Substantial reason" is an evaluative threshold that the current evidence neither clearly meets nor clearly fails. No credence is given because the claim is evaluative. A stronger pass would read the replies and counter-replies (Carlsmith's response, and the LessWrong and Alignment Forum discussion of simplicity-weighted counting) and any empirical work on the inductive biases relevant to goal formation.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Many more goals that motivate strategic good behavior in training (schemer goals) fit the training data than goals that match the intended ones. Because indifference over training-compatible goals reliably predicts the goals that SGD-trained systems acquire, schemers would therefore be a substantial share of the likely outcomes, which gives substantial reason to worry.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that schemer-compatible goals vastly outnumber intended ones, and that indifference among them tracks what training produces, substantial worry follows. The argument stands or falls with the predictive reliability of indifference over training-compatible goals. That premise is unassessed and looks doubtful in its uniform form, given evidence of SGD's inductive biases.
Because the scheming counting argument is relevantly analogous to an overfitting prediction that failed, and because SGD does not sample uniformly from the functions that perform well in training, indifference reasoning is misapplied in counting arguments for scheming. It therefore gives little reason to worry about scheming.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 inconclusive2 not separately assessed
Granting the analogy to the failed overfitting prediction and SGD's non-uniform sampling, crude indifference reasoning is shown to prove too much, and so it gives little reason for worry. The argument leaves untouched refined counting arguments that weight goals by simplicity, so its force depends on whether the analogy actually holds, which remains disputed. Non-uniform SGD sampling is plausible, but it does not by itself show which direction the bias favors.
Provenance
Where this claim has been said, linked to its canonical form.
Once we reject the indifference principle, there is very little reason left to worry that future AIs will become schemers.
More generally, we find that almost all arguments for taking scheming seriously depend on unsound indifference reasoning.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 1, 2026. Every judgment on this page is accompanied by a reasoning trace.