Far more goals compatible with high training performance lead to scheming than to alignment.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass. Added two arguments: instrumental training-gaming (for) and measure-dependence (against). Created two subclaims (the goal-guarding incentive and measure-dependence) and linked the existing claim on simplicity and volume priors. Recorded Carlsmith (poses) and Belrose and Pope (denies). Assessed inconclusive at 0.65 with no credence, because the truth value depends on an unfixed measure. This is consistent with the parent claim's doubtful verdict, so no notification was sent.
Assessed Inconclusive
verdict confidence 0.65
This claim is the central premise of the "counting argument" for AI scheming. It holds that among the goals an AI could have that are compatible with doing well in training, those that lead it to scheme (to perform well instrumentally while pursuing other ends) vastly outnumber those that amount to genuine alignment. Joe Carlsmith's 2023 report "Scheming AIs" gives the standard statement, illustrated with a toy ratio of 2^100 schemer-like goals for every non-schemer goal. Carlsmith presents it as an intuition, not a measured fact. The case for it is that most long-horizon goals give a model a reason to play along in training, whereas alignment requires the model's goals to match one narrow target. Read as a count of distinct goal descriptions, that is plausible. The difficulty is that the claim has no determinate answer until one says what is being counted. Counts of goals, functions, and parameter configurations can order the two classes differently. Nora Belrose and Quintin Pope point out that the same kind of count over functions would predict that neural networks overfit, which they do not. Defenders reply that on parameter-volume or simplicity-weighted measures the schemer class may still dominate, but nobody has computed or measured that for realistic training. On a naive enumeration of goals the claim is probably true. On the measures that would bear on what training actually produces, it is neither established nor refuted. The evidence therefore leaves it unresolved. Theoretical or empirical work on how networks' inductive biases weight schemer-like and aligned internal objectives would bear most directly on it.
Claim entered the graph