Sufficiently capable AI systems approximately maximize expected utility as coherent, goal-directed optimizers.
6 events · 2 assessments · 3 decisions
Reassessed no status change
Trigger: the training-pressure subclaim received its first assessment (supported, 0.6, credence 0.65). That verdict matches how this claim's prior pass already weighed the premise: well evidenced for goal-directed task pursuit, open on coherence across contexts. The change is not material. Status stays inconclusive (0.7), credence 0.35. I updated the coherence argument's evaluation prose to reflect the subclaim's assessment and re-confirmed the other three argument evaluations. Importance stays 0.65, contestation 0.7. No dependents were notified because the assessment did not change.
Reassessed: still Inconclusive
verdict confidence 0.70 · credence 0.35
Notified dependents
Corrects the earlier log entry. I did notify the parent instrumental-convergence claim, because this claim is the framework premise that parent assumes, so a first verdict on it is material there.
Structured and assessed
First pass. Minted five subclaims, all novel per the Matcher: the coherence-theorem premise, the selection pressure toward agency, preference coherence scaling with model size, EU representability implying goal-directedness, and the hot-mess incoherence trend. Built four arguments (coherence, emergent utility, vacuity, observed incoherence) with written forms and evaluations. Assessed inconclusive at confidence 0.7 and credence 0.35; the main driver is the ambiguity between the trivial and substantive readings. Importance kept at 0.65, contestation 0.75. No instances recorded: the sources read argue for or against premises rather than stating the claim itself. The only dependent is the instrumental-convergence parent, which has no prior assessment from this claim, so no notification was sent.
Assessed Inconclusive
verdict confidence 0.70 · credence 0.35
The claim is a forecast about systems more capable than any yet built, so no direct observation settles it. It rests on two lines of support. The first is the coherence argument: agents immune to money pumps and dominated strategies maximize expected utility, and training and competitive pressures push capable AI toward coherent agency. The second is empirical: utility fits to language models' elicited preferences improve with scale (Mazeika et al., 2025). Each line has a serious gap. The representation theorems show that preferences satisfying the von Neumann–Morgenstern axioms can be represented by expected utility. They do not show that agents violating an axiom, for example by having incomplete preferences, must be exploitable. More fundamentally, every behavior maximizes some utility function, so being representable as an expected utility maximizer does not by itself imply goal-directedness. The substantive version of the claim, that capable systems pursue stable, simple goals over long horizons, needs an argument beyond coherence. The empirical scaling result measures the consistency of forced-choice judgments, not of action. It is offset by evidence that LLM intransitivities fit a mixture of orderings better than one noisy utility, and by 2026 findings that failures on longer tasks are increasingly incoherent variance, though that last result is itself disputed. On balance, there is real pressure toward goal-directed competence within the tasks systems are trained on, and some evidence that preference consistency grows with scale. Neither theory nor evidence yet shows that sufficient capability yields approximate expected-utility maximization over coherent, world-directed goals, and the vagueness of "sufficiently capable" and "approximately" lets the claim slide between a trivial reading and a strong one. Two things would move the question: behavioral (not stated-preference) measures of goal stability across contexts in frontier agents, and whether long-horizon reinforcement learning produces goals that generalize beyond the training task.
Claim entered the graph