Safety-Aware Task Composition for Discrete and Continuous Reinforcement Learning (2306.17033v1)
Abstract: Compositionality is a critical aspect of scalable system design. Reinforcement learning (RL) has recently shown substantial success in task learning, but has only recently begun to truly leverage composition. In this paper, we focus on Boolean composition of learned tasks as opposed to functional or sequential composition. Existing Boolean composition for RL focuses on reaching a satisfying absorbing state in environments with discrete action spaces, but does not support composable safety (i.e., avoidance) constraints. We advance the state of the art in Boolean composition of learned tasks with three contributions: i) introduce two distinct notions of safety in this framework; ii) show how to enforce either safety semantics, prove correctness (under some assumptions), and analyze the trade-offs between the two safety notions; and iii) extend Boolean composition from discrete action spaces to continuous action spaces. We demonstrate these techniques using modified versions of value iteration in a grid world, Deep Q-Network (DQN) in a grid world with image observations, and Twin Delayed DDPG (TD3) in a continuous-observation and continuous-action Bullet physics environment. We believe that these contributions advance the theory of safe reinforcement learning by allowing zero-shot composition of policies satisfying safety properties.
- Compositionality and bounds for optimal value functions in reinforcement learning. arXiv preprint arXiv:2302.09676, 2023.
- Safe reinforcement learning via shielding. In AAAI, pages 2669–2678. AAAI Press, 2018.
- Control barrier functions: Theory and applications. In 2019 18th European Control Conference (ECC), pages 3420–3431, 2019. doi: 10.23919/ECC.2019.8796030.
- Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
- A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866, 2017.
- Model-free reinforcement learning for symbolic automata-encoded objectives. arXiv preprint arXiv:2202.02404, 2022.
- Formal Methods for Discrete-Time Dynamical Systems, volume 89. Springer, 01 2017. ISBN 978-3-319-50762-0. doi: 10.1007/978-3-319-50763-7.
- Hierarchical potential-based reward shaping from task specifications. arXiv preprint arXiv:2110.02792, 2021.
- Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control. IEEE Transactions on Robotics, pages 1–19, 2023. doi: 10.1109/TRO.2022.3232542.
- Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018.
- S. Gronauer. Bullet-safety-gym: A framework for constrained reinforcement learning. Technical report, mediaTUM, 2022.
- How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4-5):698–721, 2021.
- H. W. James and E. Collins. An analysis of transient markov decision processes. Journal of applied probability, 43(3):603–621, 2006.
- A composable specification language for reinforcement learning tasks. Advances in Neural Information Processing Systems, 32, 2019.
- Compositional reinforcement learning from logical specifications. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
- A formal methods approach to interpretable reinforcement learning for robotic planning. Science Robotics, 4(37):eaay6276, 2019. doi: 10.1126/scirobotics.aay6276.
- A boolean task algebra for reinforcement learning. In NeurIPS, 2020.
- Generalisation in lifelong reinforcement learning through logical composition. In International Conference on Learning Representations, 2022a.
- Skill machines: Temporal logic composition in reinforcement learning. CoRR, abs/2205.12532, 2022b.
- Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, pages 278–287. Morgan Kaufmann, 1999.
- A. Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science (sfcs 1977), pages 46–57, 1977. doi: 10.1109/SFCS.1977.32.
- Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020.
- M. E. Taylor and P. Stone. Transfer learning for reinforcement learning domains: A survey. J. Mach. Learn. Res., 10:1633–1685, dec 2009. ISSN 1532-4435.
- Composing value functions in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6401–6409. PMLR, 09–15 Jun 2019.
- Kevin Leahy (20 papers)
- Makai Mann (13 papers)
- Zachary Serlin (14 papers)