Mesa-optimization is expected to occur mainly in machine learning systems more advanced than those existing today.
4 events · 1 assessment · 2 decisions
Decomposed
Added two subclaims, both new according to the Matcher. One is an empirical claim that current transformers implement learned optimization in context (contradicts, under a named "against" argument). The other is a definitional claim about the narrow reading of mesa-optimization (defines). Wrote and evaluated the argument. The assessment stands at inconclusive, credence 0.45.
Assessed without decomposition
The first turn used most of the budget, so I recorded an inconclusive assessment (credence 0.45) from background knowledge, without searching. I set importance to 0.45 and contestation to 0.5. I deferred decomposition, which should cover the definition of mesa-optimization and evidence of learned optimization in current models, to a funded later pass. There are no dependents to notify.
Assessed Inconclusive
verdict confidence 0.55 · credence 0.45
The claim restates a position set out in the 2019 paper "Risks from Learned Optimization" (Hubinger, van Merwijk, Mikulik, Skalse and Garrabrant), which coined "mesa-optimization" for the case where a trained model is itself an optimizer pursuing an internal objective, and judged that systems of the time probably did not do this while more capable future systems might. Later evidence has made that expectation less clear-cut. Mechanistic work on small transformers, notably von Oswald and colleagues' 2023 study "Uncovering mesa-optimization algorithms in Transformers," found that models trained on sequence prediction implement something close to gradient-descent steps on an internal objective during in-context learning, and large language models and reasoning models now show in-context learning, planning and search-like behaviour. Whether these count as mesa-optimization in the original, stronger sense (a learned search process directed at a coherent internal goal that can diverge from the training objective) is disputed. On a broad reading, mesa-optimization already occurs in existing systems and the claim is weakened; on the narrow, safety-relevant reading, there is still no clear demonstration in deployed systems, and the expectation that it arises mainly with greater capability remains plausible but untested. The question therefore turns largely on the definition of mesa-optimization and on interpretability results that do not yet exist. It would be resolved by agreed criteria for detecting internal goal-directed search, applied to frontier models.
Claim entered the graph