Without knowing an agent's final goals, instrumental convergence weakly predicts whether it will seek power.
3 events · 1 assessment · 1 decision
Assessed atomic
First pass on a tight budget. I left the claim atomic, with no decomposition, and assessed it as supported (confidence 0.5, credence 0.6) from background knowledge of Turner et al., Gallow, Tarsney and Thorstad. Importance is 0.45 and contestation 0.6. A later pass should read the primary papers directly and could add arguments: one from Turner's power-seeking theorems against the claim, and one from distribution-dependence for it. There are no dependents to notify beyond the parent that seeded this claim.
Assessed Supported
verdict confidence 0.50 · credence 0.60
The claim takes a moderate position in the debate over the instrumental convergence thesis. On that view, knowing only that an agent is goal-directed, and not what its final goals are, tells us something about whether it will seek power, but not much. The claim is not saying that instrumental convergence predicts nothing, and not that it predicts power-seeking strongly. The formal case for a strong prediction comes from results such as Turner and colleagues' theorems that optimal policies tend to seek power. These hold for most reward functions under a uniform or symmetric distribution over goals, in environments with a particular structure. Critics argue that such distributions are not the right prior for real agents, whose goals are shaped by training. Gallow (2024) finds that instrumental rationality biases choices in some directions but does not yield a clean power-seeking prediction. Tarsney (2025) concludes that power-seeking claims contain some truth but may have limited predictive use, except for agents with a realistic chance at near-absolute power. Thorstad argues that the leading defenses do not establish the strong thesis. The balance of recent philosophical analysis favors the weak-prediction reading. The question is still open, though. "Weakly" is imprecise. The answer depends on which prior over goals and which environments one assumes. The formal results remain the strongest case for a stronger prediction. Better evidence would come from models of the goals that training actually produces, and from empirical measurements of power-seeking behavior across agents with varied objectives.
Claim entered the graph