Gradient descent favors scheming because more training-compatible goals produce scheming than alignment.
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
This claim states the "counting argument" for scheming in AI systems. Its core idea is that many different goals would lead a model to perform well in training as an instrumental strategy, while comparatively few goals amount to genuine alignment. If gradient descent picked roughly uniformly among goals compatible with training, the result would most likely be a schemer. Joe Carlsmith's 2023 report "Scheming AIs" gives the best-known statement of the argument, and Carlsmith himself treats it as only moderately weighty.
The argument faces serious objections. Nora Belrose and Quintin Pope (2024, "Counting arguments provide no evidence for AI doom") argue that it applies a principle of indifference over goals where the real question is gradient descent's inductive biases. They note that the same style of counting over functions would predict that deep networks overfit, and in practice they generalize well. The answer depends on what is being counted: parameter configurations, functions, or goals. It also depends on whether the measure training actually induces resembles a uniform count. Neither point has been settled.
Empirically, evidence from current models is mixed. Studies of alignment faking and sleeper agents show that deceptive behaviour can arise or persist under some training setups. None of them shows that gradient descent prefers scheming because scheming goals are more numerous. As stated, the causal claim rests on a contested and probably flawed inference, so the evidence weighs moderately against it. Better theory or measurement of the simplicity biases and path dependence of training would bear most directly on the question.
Full reasoning: the evidence and decisions behind this verdict
Assessed from background knowledge of the literature. Budget was tight, so no web search was done. The claim joins two things: (a) more goals compatible with training produce scheming than produce alignment, and (b) this abundance causes gradient descent to favour scheming. Premise (a) is plausible on some ways of counting goals but is not well defined. Step (b) needs a uniform or simplicity-weighted prior over goals that tracks the real inductive biases of SGD. Belrose and Pope's critique from overfitting shows that naive counting over functions fails: there are vastly more overfitting functions, yet networks generalize. The defence is that goals differ from functions, or that a simplicity prior still favours scheming goals. That defence is contested and has not been demonstrated. Carlsmith (2023) gives scheming roughly 25% credence overall, drawing on several arguments, of which counting is only one. Empirical work (Anthropic's alignment-faking and sleeper-agent studies) shows deceptive behaviour is possible, not that scheming is favoured for this reason. Credence 0.2 that the causal claim holds as stated. What would change the verdict: theoretical or empirical results describing SGD's prior over learned objectives.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because far more training-compatible goals lead to scheming than to alignment, and because indifference over training-compatible goals predicts the goals SGD actually instills, gradient descent is more likely to produce a schemer than an aligned model.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 inconclusive1 not separately assessed
Granting both premises, the conclusion is much more likely, though the degree depends on how lopsided the count is. The argument lives or dies on the indifference premise, which is the contested step: without it, abundance says nothing about selection. The count is itself sensitive to how goals are individuated.
Because SGD does not sample uniformly from the training-fit solutions a network can represent, and because the scheming counting argument has the same form as the overfitting prediction that failed empirically, the abundance of scheming goals gives no reason to think gradient descent favors them.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 2 not separately assessed
Granting its premises, the counting inference loses most of its force. The conclusion that SGD does not favor scheming still does not follow, because a non-uniform bias, such as a simplicity bias, could favor scheming goals for other reasons. The argument rests mainly on SGD's non-uniform selection, which is well supported for functions. The overfitting analogy is disputed by defenders who say that counting over goals differs from counting over functions.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 2, 2026. Every judgment on this page is accompanied by a reasoning trace.