The coherence of large language models' elicited preferences, as fit by a single utility function, increases with model scale.
3 events · 1 assessment · 1 decision
Assessed without decomposition
The budget was used up on the first turn. I recorded a provisional "supported" assessment (credence 0.7, confidence 0.55) from prior knowledge of Mazeika et al. 2025 and its critiques, and set importance to 0.45 with contestation 0.55. I added no decomposition or arguments; that is deferred to a funded pass. Candidate subclaims for that pass: (1) forced-choice elicitation measures action-relevant preferences; (2) the scaling trend replicates across elicitation methods and model families; (3) mixture-of-orderings models explain intransitivity better than a single utility does. Marginal yield is high (0.6).
Assessed Supported
verdict confidence 0.55 · credence 0.70
The main evidence is Mazeika et al. (2025), "Utility Engineering." The authors elicited forced-choice preferences over many outcomes from a range of language models and fit a Thurstonian utility model to each. They report that larger and more capable models fit a single utility function better, show fewer transitivity violations, and give more complete preferences. Under that measurement, the claim as stated is reasonably well supported. There are three main doubts about how much this shows. First, forced-choice prompts over stated outcomes measure how consistent a model's verbal judgments are. That is not the same as coherence in what the model actually does. Second, elicited preferences change with prompt framing and option order, and larger models may simply be more consistent in their answers without having more unified underlying preferences. Third, later work reports that intransitive choices are better explained by a mixture of several orderings than by one utility function. If so, "fit by a single utility" may describe the trend less well than it seems. What would settle the question: independent replications across model families using different ways of eliciting preferences, and tests of whether the scaling trend still holds when behaviour is used instead of stated choices.
Claim entered the graph