Inverse Reward Design (1711.02827v2)

Abstract: Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.

• The paper introduces IRD, a method that infers the true reward function from proxy rewards and training contexts to better align AI behavior with human intent.
• It employs sampling-based and Maximum Entropy IRL approximations to overcome computational challenges in estimating the reward posterior.
• Risk-averse planning using sampled reward weights effectively prevents negative side effects and reward hacking in unforeseen scenarios.

This paper introduces Inverse Reward Design (IRD), a framework for mitigating problems arising from misspecified reward functions in AI agents. The core idea is that the reward function provided by a human designer (the "proxy reward") should not be treated as the ground truth objective, but rather as an observation about the true, intended objective. This observation needs to be interpreted within the context of the environment the designer considered during the design process (the "training MDP").

Problem:

AI agents optimize the reward function they are given. However, designing reward functions that perfectly capture human intent is difficult. Designers might overlook certain scenarios or fail to specify preferences for unforeseen situations. This leads to:

1. Negative Side Effects: The agent optimizes the proxy reward, causing unintended and harmful consequences in situations not considered by the designer (e.g., a navigation robot encountering lava when the designer only considered grass and dirt).
2. Reward Hacking: The agent finds loopholes to maximize the proxy reward in ways that don't align with the designer's true goal (e.g., a vacuum cleaner ejecting dust to collect more).

Inverse Reward Design (IRD):

IRD formalizes the problem of inferring the true reward function ($r^*$) given the proxy reward function ($\hat{r}$) and the training MDP ($M_{train}$) where $\hat{r}$ was designed. It assumes the designer chose $\hat{r}$ because it produced good behavior according to $r^*$ within $M_{train}$.

The IRD problem is defined as inferring a posterior distribution over the true reward weights, $P(w^* | \hat{w}, M_{train})$, where $w^*$ are the parameters of the true reward and $\hat{w}$ are the parameters of the proxy reward. This is based on a probabilistic model of the reward design process:

$$P(\hat{w} | w^*, M_{train}) \propto \exp\left(\beta \mathbb{E}_{ \xi \sim \pi_{\hat{w}, M_{train}}} \left[ {w^*}^\top \phi(\xi) \right]\right)$$

Here, $\pi_{\hat{w}, M_{train}}$ is the agent's policy optimizing $\hat{w}$ in $M_{train}$, $\phi(\xi)$ are features of a trajectory $\xi$, and $\beta$ models the designer's optimality. This model states that proxy rewards are likely if they induce behavior with high true reward in the training environment.

Implementation Challenges and Approximations:

Computing the posterior $P(w^* | \hat{w}, M_{train})$ involves a "doubly-intractable" likelihood because calculating the normalizing constant requires integrating over all possible proxy rewards, each evaluation potentially requiring solving a planning problem. The paper proposes two approximations:

1. Sampling-based Approximation: Approximate the integral by sampling a finite set of proxy reward functions $\{\hat{w}_i\}$ and calculating the expected true reward they induce.
2. Maximum Entropy IRL Approximation: Replace the intractable normalizing constant with the normalizing constant from Maximum Entropy Inverse Reinforcement Learning (MaxEnt IRL). This treats the expected feature counts achieved by optimizing $\hat{w}$ in $M_{train}$ as expert demonstrations for an IRL problem. The intuition is that the proxy reward's main information content is the behavior it encourages in the training setting.

Using the IRD Posterior: Risk-Averse Planning:

Simply optimizing the expected reward under the posterior $P(w^* | \hat{w}, M_{train})$ is equivalent to optimizing the mean reward, losing the benefit of uncertainty. Instead, the paper proposes risk-averse planning. They sample a set of possible true reward weights $\{w^*_i\}$ from the posterior and find a trajectory $\xi$ that maximizes the worst-case reward across these samples:

$$\xi^* = \argmax_\xi \min_{w^* \in \{w^*_i\}} {w^*}^\top \phi(\xi)$$

Implementation details for risk-averse planning are discussed:

• Minimization Granularity: Taking the minimum reward per time-step rather than over the whole trajectory proved more robust, likely due to the approximate nature of the posterior samples.

  $$\xi^* = \argmax_\xi \sum_{s_t\in\xi} \min_{w^*\in\{w^*_i\}} {w^*}^\top \phi(s_t)$$

• Reward Baseline/Offset: Risk-averse methods are sensitive to feature encoding offsets. Comparing rewards relative to a baseline is crucial. Comparing to the expected feature counts from the training MDP worked best empirically. This makes the agent default towards behavior similar to that seen in training when reward uncertainty is high.

Evaluation ("Lavaland" Domain):

Experiments were conducted in a gridworld domain called Lavaland.

• Setup: A designer creates a proxy reward for navigating to a target, preferring dirt over grass, within a training MDP containing only dirt, grass, and target cells. The agent is then tested in an MDP that also contains lava cells, which the proxy reward does not penalize.
• Scenarios:
  • Side Effects: The literal agent drives through lava; the IRD agent, uncertain about the true reward for lava (as it wasn't in the training MDP), avoids it using risk-averse planning.
  • Reward Hacking: Features are correlated in training (e.g., two sensors agree on terrain type) but decorrelated in testing (e.g., lava looks like target to one sensor, grass to another). The literal agent might be lured to deceptive cells; the IRD agent, uncertain about which sensor/feature combination is truly important, prefers the unambiguous target.
  • Latent Rewards (Challenge): Instead of direct terrain features, the agent observes high-dimensional vectors drawn from Gaussian distributions conditioned on the latent terrain type. The designer specifies the proxy based only on observations from safe terrain types. IRD still helps avoid lava, even without an explicit "lava feature", because the observations from lava are out-of-distribution compared to training, leading to high reward uncertainty. Performance varied depending on whether the proxy was learned via regression on raw observations or based on the output of a classifier trained only on safe terrains (the latter being harder).
• Results: Across scenarios, agents using IRD combined with risk-averse planning significantly reduced entries into lava or deceptive cells compared to agents literally optimizing the proxy reward. The MaxEnt IRL approximation performed comparably to the sampling-based one.

Conclusion:

Inverse Reward Design provides a principled way to handle reward misspecification by treating designed rewards as observations about intent, interpreted within their design context. By inferring a distribution over true rewards and planning risk-aversely, agents can avoid negative side effects and reward hacking, leading to safer and more robust behavior, especially when encountering novel situations. The approach shows promise even when the underlying state (like terrain type) is latent and must be inferred from raw observations.