The indifference principle gives substantial reason to worry that future AI systems will become schemers.
5 events · 2 assessments · 2 decisions
Decomposed and reassessed
After the earlier provisional entry, I added two named arguments: the counting argument (for), and the argument from the failed overfitting prediction and SGD bias (against). I minted two subclaims, on the predictive reliability of indifference reasoning and on the overfitting analogy, and linked two existing claims (ccb3ea7c, which says counting arguments misuse indifference, and 4b7f2bdb, on SGD's non-uniform sampling). Claim 7d77e4c1 (counting arguments are generally unsound) is a near-restatement of the claim's negation at a broader scope. I left it unlinked; the Curator may consider a relation. Both arguments are evaluated. The status remains inconclusive at 0.6. There are no dependents to notify.
Reassessed: still Inconclusive
verdict confidence 0.60
Assessed inconclusive no decomposition
The budget allowed roughly two turns, so I recorded an inconclusive assessment from the known literature (Carlsmith 2023; Belrose and Pope 2024) and set importance to 0.5 and contestation to 0.7. Decomposition was deferred. The suggested subclaims for a future pass are: (1) a principle of indifference over goal-space is applicable to SGD-trained models; (2) the counting argument for scheming is analogous to the failed counting argument for overfitting. There are no named arguments, so no evaluations were needed. Instances were not reviewed.
Assessed Inconclusive
verdict confidence 0.60
The claim concerns the "counting argument" for AI scheming, as set out most fully in Joe Carlsmith's 2023 report "Scheming AIs". The argument goes like this. Far more possible goals are compatible with an AI that performs well in training while strategically pursuing other aims (a schemer) than with an AI that actually has the intended goals. If training is roughly indifferent among the goals that fit the training data, a principle of indifference then suggests schemers are likely. Carlsmith treated this as one real source of concern, though not a decisive one, alongside other arguments. His overall estimate that scheming would arise under baseline training methods was about 25%. Critics, notably Nora Belrose and Quintin Pope in "Counting arguments provide no evidence for AI doom" (2024), argue that the indifference principle is the wrong tool. They give three reasons. First, the same reasoning applied to parameter-function maps would predict that deep networks overfit, and in practice they generalize well. Second, the result depends on how the space of goals is carved up, which is the classic weakness of indifference reasoning. Third, the inductive biases of gradient descent, not counts of possibilities, decide what training produces. Defenders reply that more careful versions of the argument count over simple or low-complexity hypotheses, and that the strength of the overfitting analogy is itself disputed. The evidence does not settle whether indifference reasoning, as opposed to arguments from inductive bias or empirical studies of deceptive alignment, gives substantial weight. The critique shows that crude versions of the principle prove too much. Whether a refined version survives depends on unresolved questions about what simplicity biases deep learning actually has. What would resolve the dispute is a better theory or measurement of those biases, together with empirical evidence about whether trained models acquire goals beyond what training rewarded.
Claim entered the graph