Resource acquisition, self-preservation, and goal preservation are instrumentally useful for many final goals.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass. Three arguments: means-ends (with a new subclaim on future action, imp 0.3), formal power-seeking theorems (linked existing claim 886bd7e6), and instrumental divergence (new contradicting subclaim, imp 0.35). Linked b8929e54 as a counterpart. Assessed as supported, confidence 0.75, credence 0.85. Importance 0.45, contestation 0.35. No prior assessment, so dependents are not notified separately.
Assessed Supported
verdict confidence 0.75 · credence 0.85
This is the weak form of the instrumental convergence thesis, set out by Omohundro (2008) and Bostrom (2012). As worded, it says only that these three subgoals are useful for *many* final goals. It does not say they are useful for almost all goals, or that capable agents will in fact pursue them. In that weak form it is well supported, and its critics largely accept it. The core reasoning is simple. An agent that continues to exist, keeps its present goals, and controls more resources can still act toward its goals later, and has more options when it does. So for the many goals whose achievement depends on the agent's own future action, all three help. Formal work by Turner and colleagues gives a precise version: in Markov decision processes with certain symmetries, optimal policies for most reward functions keep options open and avoid shutdown. The live dispute is about stronger claims. Gallow (2024) argues that once the costs and risks of getting power are counted, power-seeking is not clearly convergent for randomly sampled desires, though he grants that some instrumental goals are convergent. Thorstad argues that the formal theorems rest on assumptions that do not carry over to realistic agents. Both criticisms target "almost any goal" and the predictive use of the thesis, not "many goals". Resource acquisition is the weakest of the three subgoals, because for some share of goals acquisition may do more harm than good. Self-preservation and goal preservation hold more robustly. A precise answer about how many goals qualify depends on how goals are sampled and how "power" is defined, and the literature has not settled either.
Claim entered the graph