Goal Misgeneralization in Deep Reinforcement Learning

Goal Misgeneralization in Deep Reinforcement Learning An important implication of goal misgeneralization is that training a model by optimizing an objective R is not enough to guarantee that the model will itself learn to pursue R rather than some proxy for R. This is especially concerning in the context of AGI safety: ensuring that advanced AI systems are safe despite being capable enough to escape our control (Bostrom, 2014). Systems that competently pur- sue a misaligned goal may tend to seek power and deceive their operators for instrumental reasons (Turner et al., 2021; Omohundro, 2008). With highly advanced AI systems, this could lead to human disempowerment: for example, an AI system might prevent its operators from shutting it down (Hadfield-Menell et al., 2017a; Soares et al., 2015). If com- plex tasks are rife with proxies for their training objectives, it may be very hard to predict what objectives the trained AI systems will have (Hubinger et al., 2019). Our main contributions are: • We formalize goal misgeneralization, distinguishing it from capability generalization failures (Section 2), and experimentally validate our definition on a gridworld environment (Section 3.5). • We experimentally showcase goal misgeneralization. Specifically, deep RL agents trained on the Procgen benchmark (Cobbe et al., 2019)—a set of diverse procedurally generated environments specifically de- signed to induce robust generalization—still fail on our slightly modified environments (Section 3). • We show that goal misgeneralization may be alleviated by increasing the diversity of the training distribution so that the agent learns to distinguish the reward from proxies (Sections 3.1 and 3.2). • We investigate the causes of goal misgeneralization. In particular, we find that the actor and the critic compo- nents of our actor-critic model learn different proxy goals (Section 3.4). 2. Goal Misgeneralization Goal misgeneralization is a type of OOD generalization failure. OOD generalization is usually studied in the super- vised learning setting, where it is defined as achieving good test performance on data sampled from a distribution other than the training distribution. We focus on the reinforce- ment learning setting (Sutton & Barto, 2018), in which a system is trained to take actions in an environment in order to maximize a given reward. In this setting, the problem is to achieve high reward despite a shift in the distribution of observations or the transition dynamics. OOD general- ization problems frequently arise in RL and are an active area of research (Kirk et al., 2021). However, goal mis- generalization in particular has not been the focus of any previous academic work. Studying this class of failures is particularly important from the point of view of machine learning safety (Hendrycks et al., 2021), since agents that pursue imperfect proxies may fail suddenly (Pan et al., 2022; Ibarz et al., 2018) and catastrophically (Zhuang & Hadfield- Menell, 2021) as their capabilities increase. With this in mind, we provide a definition of goal misgeneralization and show how it can be formalized. 2.1. Defining Goal Misgeneralization A deep RL agent is trained to maximize a reward R : S × A × S → R, where S and A are the sets of all valid states and actions, respectively. Assume that the agent is deployed out-of-distribution; that is, an aspect of the environment (and therefore the distribution of observations) changes at test time. Goal misgeneralization occurs if the agent now achieves low reward in the new environment because it continues to act capably yet appears to optimize a different reward R′ 6 = R. We call R the intended objective and R′ the behavioral objective of the agent. Formally, we follow Orseau et al. (2018) in distinguishing goal-directed policies (agents) from unoptimized policies (devices). Let ηagt(R) and ηdev(d) be priors over a space of reward functions R ∈ R and a space of devices (policies) d ∈ Π respectively. Further let pagt(τ | R) and pdev(τ | d) be the likelihood functions giving the probability of a trajectory τ given a particular objective R or device d. We define two distributions over trajectories, the agent mixture pagt and the device mixture pdev: pagt(τ ) = ∑ R∈R pagt(τ | R) ηagt(R), (1) pdev(τ ) = ∑ d∈Π pdev(τ | d) ηdev(d). (2) The choice of device likelihood pdev(τ | d) is straightfor- ward: we simply choose the distribution over trajectories induced by running the policy d in the environment. For the agent likelihood pagt(τ | R), a popular choice is the max- imum entropy model pagt(τ | R) ∝ exp(R(τ )) (Ziebart et al., 2008). Another possibility is to choose pagt(τ | R) to be the probability density of the random trajectory obtained by training an RL algorithm to maximize R and collecting rollouts.2 Definition 2.1 (Goal misgeneralization). A policy π un- dergoes goal misgeneralization if test reward is low and pagt(τ ) > pdev(τ ) holds on average for the trajectories induced by π in the OOD test environment. In other words, the policy is acting in a goal-directed manner, but not achiev- ing high reward. We can infer a posterior distribution over 2This requires an RL algorithm and model (e.g. neural network). In practice, this choice of p(τ | R) is intractable to compute.