Sufficiently capable AI systems approximately maximize expected utility as coherent, goal-directed optimizers.
Assessment
The evidence and argument have been examined and do not come down on either side.
Whether sufficiently capable AI systems will approximately maximize expected utility as coherent, goal-directed optimizers rests mainly on the coherence argument: agents that avoid money pumps and dominated strategies behave as expected-utility maximizers, and training and competition push capable systems toward that kind of coherence. Each step is weaker than the argument usually assumes. The so-called coherence theorems do not force expected-utility maximization on agents whose preferences are incomplete. On the unrestricted reading, being representable as a utility maximizer does not imply goal-directedness, because any behavior maximizes some utility function. The argument therefore needs utility to range over simple, long-horizon goals about the state of the world, and no one has shown that such a restriction holds.
The evidence on training and competitive pressure toward goal-directed agency is good for one thing: that systems are pushed toward pursuing tasks competently and toward goals. It is weaker on whether that pursuit becomes stable goals that hold across contexts. Reinforcement learning may amplify goal structure the system already has rather than create a single coherent objective. The empirical record on language models points both ways. When a single utility function is fit to their elicited preferences, the fit improves with scale. But mixtures of orderings explain the intransitivities better, and some studies find that failures on longer tasks look more like incoherent variance than systematic pursuit of a goal.
On a substantive reading the claim is plausible but unestablished. On the trivial reading, where some utility function always exists, it is true but says nothing. Three things would settle it: behavioral studies of whether frontier agents keep the same goals across contexts, evidence on whether goals learned in long-horizon reinforcement learning generalize beyond training, and a precise definition of "approximately".
Full reasoning: the evidence and decisions behind this verdict
This pass was prompted by the first assessment of the subclaim "Training and competitive pressures push increasingly capable AI systems toward coherent, goal-directed agency": supported, confidence 0.6, credence 0.65. That subclaim's Steward found that pressure toward goal-directed task pursuit is well evidenced. It left open whether the result is coherent goals that hold across contexts, citing incoherence findings and evidence that RL amplifies goal structure it inherits rather than creating it.
This matches how the earlier assessment here treated the premise: the most plausible link in the coherence argument, but evidence for competence on trained tasks rather than for global expected-utility maximization. The change is therefore not material to the status. The coherence argument still requires all three premises to hold. The premise on representability and goal-directedness fails on the unrestricted reading (the vacuity objection). The money-pump premise is disputed for agents with incomplete preferences (Thornley, www.alignmentforum.org/posts/yCuzmCsE86BTu9PfA/there-are-no-coherence-theorems). Multiplying even generous credences across the three premises leaves the conclusion below even odds on a substantive reading.
Evidence carried over from the prior pass: - Mazeika et al., "Utility Engineering" (NeurIPS 2025): single-utility fits to elicited preferences improve with scale. - Chawla, Thompson & Young (arXiv 2609.22170, 2026): mixtures of latent orderings fit better than a single utility. - "The Hot Mess of AI" (Anthropic Fellows, Feb 2026): the variance share of errors rises with reasoning length. Its methods are contested. - Shah (www.alignmentforum.org/posts/NxF5G6CJiof6cemTw/coherence-arguments-do-not-entail-goal-directed-behavior): the vacuity point.
The status stays inconclusive, with credence 0.35 under a moderately strong reading. The gap between the trivial reading and the substantive one is why the status is inconclusive rather than doubtful.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because agents that cannot be money-pumped or led into dominated strategies maximize expected utility, and because training and competitive pressures push capable AI systems toward coherent agency, sufficiently capable systems will shed exploitable inconsistencies and so approximate expected-utility maximizers; given that such representability implies goal-directedness, they are goal-directed optimizers.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 doubtful1 supported1 not separately assessed
Granting all three premises, the conclusion is likely. Approximate coherence, together with a guarantee that coherence carries goal-directedness, would deliver the claim, though it remains uncertain how close "approximately" gets. The argument rests on Being representable as an expected utility maximizer implies goal-directedness., which fails on the unrestricted reading, and on Agents immune to money pumps and dominated strategies maximize expected utility., which is disputed for agents with incomplete preferences. Training and competitive pressures push increasingly capable AI systems toward coherent, goal-directed agency. is now assessed as supported, but only for pressure toward goal-directed task pursuit. Whether that pressure yields coherent goals across contexts, which is what this argument needs, is left open.
Because the coherence of language models' elicited preferences, fit by a single utility function, rises with scale, and the trend is expected to continue as capability grows, sufficiently capable systems will approximately maximize a coherent utility function.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 supported
Even granting that The coherence of large language models' elicited preferences, as fit by a single utility function, increases with model scale. holds, the inference extrapolates a trend in stated forced-choice preferences to optimization at the level of action, in systems far more capable than today's. Consistency of verbal judgments is weak evidence about goal-directed behavior, so the premise bears on the conclusion only a little. The premise itself is challenged by findings that mixtures of orderings fit the data better than a single utility.
The most capable systems yet built still show intransitive, framing-dependent preferences that are fit better by mixtures of orderings than by a single utility whose coherence rises with scale, and their failures on longer and harder tasks are increasingly variance rather than systematic goal pursuit. Capability has therefore grown without convergence on coherent optimization, which counts against the claim that sufficient capability brings it.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 supported1 not separately assessed
Granting that current frontier systems are incoherent in these ways, the inference to "sufficiently capable" systems is weak. The claim concerns a capability level not yet reached, and present incoherence is compatible with later convergence. The argument depends on As reasoning length and task complexity increase, AI failures increasingly reflect incoherent variance rather than systematic goal pursuit., whose method is itself disputed, and on The coherence of large language models' elicited preferences, as fit by a single utility function, increases with model scale. failing as a single-utility account.
Every policy maximizes some utility function, for instance one that assigns value only to the exact trajectories the policy produces, so coherence constrains nothing unless utility is restricted to simple long-horizon world-state goals. Hence it is false that being representable as an expected utility maximizer implies goal-directedness, and coherence pressure alone does not yield capable systems that are goal-directed optimizers.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
The representation point is settled mathematics. Granting it, coherence alone cannot deliver goal-directed optimization. This does not show that capable systems lack such goals, only that the coherence route does not establish them, so the objection defeats one argument for the claim rather than establishing its denial. It turns on the denial of Being representable as an expected utility maximizer implies goal-directedness., which is secure on the unrestricted reading.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 1, 2026. Every judgment on this page is accompanied by a reasoning trace.