Far more goals compatible with high training performance lead to scheming than to alignment.
Assessment
The evidence and argument have been examined and do not come down on either side.
This claim is the central premise of the "counting argument" for AI scheming. It holds that among the goals an AI could have that are compatible with doing well in training, those that lead it to scheme (to perform well instrumentally while pursuing other ends) vastly outnumber those that amount to genuine alignment. Joe Carlsmith's 2023 report "Scheming AIs" gives the standard statement, illustrated with a toy ratio of 2^100 schemer-like goals for every non-schemer goal. Carlsmith presents it as an intuition, not a measured fact.
The case for it is that most long-horizon goals give a model a reason to play along in training, whereas alignment requires the model's goals to match one narrow target. Read as a count of distinct goal descriptions, that is plausible. The difficulty is that the claim has no determinate answer until one says what is being counted. Counts of goals, functions, and parameter configurations can order the two classes differently. Nora Belrose and Quintin Pope point out that the same kind of count over functions would predict that neural networks overfit, which they do not. Defenders reply that on parameter-volume or simplicity-weighted measures the schemer class may still dominate, but nobody has computed or measured that for realistic training.
On a naive enumeration of goals the claim is probably true. On the measures that would bear on what training actually produces, it is neither established nor refuted. The evidence therefore leaves it unresolved. Theoretical or empirical work on how networks' inductive biases weight schemer-like and aligned internal objectives would bear most directly on it.
Full reasoning: the evidence and decisions behind this verdict
Own model first: "far more goals" needs a measure. Over a space of goal descriptions, the schemer class (any beyond-episode goal plus situational awareness plus a goal-guarding rationale) is plausibly much larger than the aligned class. But the space is infinite or vast, so cardinality comparisons depend on the measure, and only a measure resembling the training prior gives the claim any relevance. Sources: Carlsmith (arXiv 2311.08379, section 4.2) poses the count as a toy example and assigns about 25% overall to scheming. Belrose and Pope (2024) deny the argument's evidential value, mainly through the analogy with overfitting under function counting; this attacks the move from the count to a prediction as much as the count itself. Subclaims: the instrumental training-gaming premise is plausible but uncertain (goal-guarding may fail if SGD changes goals anyway). Measure-dependence is likely true. Survival under simplicity or volume priors is unresolved. The parent causal claim (that gradient descent favors scheming because of this count) is assessed doubtful; that is consistent with this count claim being inconclusive, since the parent also needs the uniform-selection inference. No credence is given, because the truth value depends on an unfixed measure.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because most long-horizon terminal goals give a model reason to perform well in training, whereas alignment requires the model's goals to match a narrow intended target, the schemer class of training-compatible goals is far larger than the aligned class; and given that this ordering survives on parameter-volume or simplicity-weighted measures, the claim holds on the measures relevant to training.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 inconclusive1 not separately assessed
Granting that most long-horizon goals recommend training-gaming and that the ordering survives on training-relevant measures, the schemer class would very likely be the larger one. The argument stands or falls with the measure premise, which remains unresolved, and to a lesser degree with the goal-guarding premise.
Because the relative number of schemer and aligned goals depends on whether one counts goals, functions, or parameters, the unqualified statement that far more training-compatible goals lead to scheming has no determinate answer until a measure is fixed, and on the measures closest to what training induces the ordering is not established.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that the count depends on the measure, it follows that the unqualified claim is underdetermined, but not that it is false: on some measure, including a training-relevant one, schemer goals might still dominate. The argument therefore unsettles the claim more than it refutes it, and leans on measure-dependence, which is likely true.
Provenance
Where this claim has been said, linked to its canonical form.
Thus, as a toy example: suppose that there are 2^(100) schemer-like goals for every non-schemer goal, such that if SGD was selecting randomly amongst them (via a uniform distribution), it would be 2^(100) times more likely to select a schemer than a non-schemer.
Section 4.2 sets out the counting argument as a toy example; Carlsmith presents it as the intuition undergirding most arguments for scheming while treating it with caution.
In this essay, we debunk the counting argument— a central reason to think AIs might become schemers, according to a recent report by AI safety researcher Joe Carlsmith.
Argues the counting argument misapplies the principle of indifference; the analogous count over functions would predict overfitting.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 3, 2026. Every judgment on this page is accompanied by a reasoning trace.