Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Tiered Reward: Designing Rewards for Specification and Fast Learning of Desired Behavior (2212.03733v3)

Published 7 Dec 2022 in cs.LG and cs.AI

Abstract: Reinforcement-learning agents seek to maximize a reward signal through environmental interactions. As humans, our job in the learning process is to design reward functions to express desired behavior and enable the agent to learn such behavior swiftly. However, designing good reward functions to induce the desired behavior is generally hard, let alone the question of which rewards make learning fast. In this work, we introduce a family of a reward structures we call Tiered Reward that addresses both of these questions. We consider the reward-design problem in tasks formulated as reaching desirable states and avoiding undesirable states. To start, we propose a strict partial ordering of the policy space to resolve trade-offs in behavior preference. We prefer policies that reach the good states faster and with higher probability while avoiding the bad states longer. Next, we introduce Tiered Reward, a class of environment-independent reward functions and show it is guaranteed to induce policies that are Pareto-optimal according to our preference relation. Finally, we demonstrate that Tiered Reward leads to fast learning with multiple tabular and deep reinforcement-learning algorithms.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. On the expressivity of markov reward. Advances in Neural Information Processing Systems, 34: 7799–7812.
  2. Faulty reward functions in the wild. URL: https://blog. openai. com/faulty-reward-functions.
  3. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565.
  4. R-MAX—A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research, 3: 213–231.
  5. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, 783–792. PMLR.
  6. Non-markovian rewards expressed in LTL: guiding search via reward shaping. In Tenth annual symposium on combinatorial search.
  7. LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning. In IJCAI, volume 19, 6065–6073.
  8. Minigrid. CoRR, abs/2306.13831.
  9. Explicable reward design for reinforcement learning agents. Advances in Neural Information Processing Systems, 34: 20118–20131.
  10. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, 1407–1416. PMLR.
  11. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  12. Models of Sequential Decision Making (msdm).
  13. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning, 2107–2116. PMLR.
  14. Reward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73: 173–208.
  15. Complexity analysis of real-time reinforcement learning. In AAAI, volume 93, 99–105.
  16. Reinforcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3834–3839. IEEE.
  17. Environment-independent task specifications via GLTL. arXiv preprint arXiv:1704.04341.
  18. Mataric, M. J. 1994. Reward functions for accelerated learning. In Machine learning proceedings 1994, 181–189. Elsevier.
  19. Mornati, F. 2013. Pareto Optimality in the work of Pareto. Revue européenne des sciences sociales. European Journal of Social Sciences, (51-2): 65–82.
  20. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, 278–287.
  21. Subgoal-based reward shaping to improve efficiency in reinforcement learning. IEEE Access, 9: 97557–97568.
  22. Artificial Intelligence (A Modern Approach).
  23. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  24. Designing Rewards for Fast Learning.
  25. Reinforcement Learning: An Introduction. The MIT Press.
  26. Teaching multiple tasks to an RL agent using LTL. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, 452–461.
  27. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine Learning, 84(1): 51–80.
  28. Q-learning. Machine learning, 8(3): 279–292.
  29. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136): 1–46.
  30. Receding horizon control for temporal logic specifications. In Proceedings of the 13th ACM international conference on Hybrid systems: computation and control, 101–110.
  31. Computational benefits of intermediate rewards for goal-reaching policy learning. Journal of Artificial Intelligence Research, 73: 847–896.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets