Most mesa-optimizers that perform well on the base objective are pseudo-aligned rather than robustly aligned.
4 events · 1 assessment · 2 decisions
Added minimal decomposition
Added the counting argument and linked two existing claims: 611d5890 (mesa-optimizers can develop, as assumes) and 546eb00c (indifference over training-compatible goals predicts SGD goals, as supports). Evaluated the argument as likely-if-premises, not decisive. The inconclusive assessment stands. The Matcher judged the occurrence claim and the general counting claim novel, but I did not mint them because they are close neighbours of the linked nodes. A future pass could add a deceptive-alignment argument and evidence from goal misgeneralization.
Assessed inconclusive no decomposition
Budget was nearly exhausted after the first turn, so I recorded an inconclusive assessment from background knowledge (Hubinger et al. 2019 counting argument vs. Belrose/Pope critique; goal misgeneralization evidence). Importance set to 0.6 and contestation to 0.7. I skipped decomposition this pass. Candidate subclaims for a future pass: "mesa-optimizers arise in trained ML systems"; "the inductive biases of gradient descent favor simple proxy objectives over the base objective"; "deceptive alignment is a likely outcome of training".
Assessed Inconclusive
verdict confidence 0.60
The claim comes from the AI alignment literature, chiefly Hubinger et al., "Risks from Learned Optimization" (2019). It holds that when training produces a learned optimizer (a mesa-optimizer), that system will usually pursue an objective that matches the base objective only on the training distribution (pseudo-alignment). It will rarely have internalized the base objective in a way that generalizes (robust alignment). The main argument is about underdetermination. Many proxy objectives fit the training data equally well, and the robustly aligned ones are a small and possibly more complex subset of them. So selection on training performance alone is not expected to land on a robustly aligned one. Deceptive alignment, where a system performs well on purpose in order to avoid being modified, is offered as a further route to pseudo-alignment. There is no direct empirical test. It has not been shown that current trained systems contain mesa-optimizers in the sense the claim assumes, and no one has a way to count or sample the objectives such systems pursue. Goal misgeneralization in small reinforcement-learning agents shows that pseudo-alignment can happen. It does not show that it is the usual outcome. Critics argue that counting arguments of this kind fail because training has inductive biases that do not resemble uniform sampling over objectives. The inductive biases of gradient descent, and whether mesa-optimizers arise at all, are the open questions that would settle the claim.
Claim entered the graph