Human evolution provides evidence that optimization can produce agents whose goals diverge from the optimizing process's objective.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass. Built a for-argument (human proxy goals: the existing verified divergence subclaim plus the linked framework claim on modeling selection as optimization) and an against-argument (the linked residual-fitness claim plus a newly minted disanalogy claim, 141d56bf, at importance 0.5). Recorded Hubinger et al. 2019 as affirming and Pope 2023 as a mention. Assessed supported at confidence 0.75 and credence 0.8, since the claim's "can" reading survives the disanalogy objection. The claim has no parents, so no notifications were needed.
Assessed Supported
verdict confidence 0.75 · credence 0.80
The claim is modest in form: it says only that human evolution is evidence that optimization *can* produce agents whose goals differ from the optimizing process's objective. Read that way, it is well supported. Natural selection is standardly described, to a useful approximation, as favoring inclusive genetic fitness, and humans plainly do not hold that as their goal. Human goals differ from evolution's objective: people pursue proxies such as pleasure, status and affection, and use contraception and choose small families even when that costs them reproductive success. This is the example Hubinger and colleagues (2019) used to introduce the idea of misaligned "mesa-optimizers". Two qualifications limit how much the example shows. First, evolution is not an agent. Its "objective" is a modeling device, so the claim holds only to the extent that natural selection can usefully be modeled as optimizing fitness. Second, some argue that human motivations still track fitness substantially: the species is demographically very successful. That point bears on how large the divergence is, not on whether it exists. The live dispute is about how far the example transfers to engineered optimization such as AI training. Critics, most prominently Quintin Pope, argue that selection over genomes, with values formed by within-lifetime learning, differs so much from gradient descent that evolution says little about AI goal misgeneralization. That objection weakens stronger conclusions, such as that AI training will probably produce misaligned goals. It leaves the existence claim mostly intact. What would most change the verdict is a convincing case that evolution is not an optimization process in any sense relevant to the claim.
Claim entered the graph