Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
43 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A safe exploration approach to constrained Markov decision processes (2312.00561v2)

Published 1 Dec 2023 in cs.LG and math.OC

Abstract: We consider discounted infinite horizon constrained Markov decision processes (CMDPs) where the goal is to find an optimal policy that maximizes the expected cumulative reward subject to expected cumulative constraints. Motivated by the application of CMDPs in online learning of safety-critical systems, we focus on developing a model-free and simulator-free algorithm that ensures constraint satisfaction during learning. To this end, we develop an interior point approach based on the log barrier function of the CMDP. Under the commonly assumed conditions of Fisher non-degeneracy and bounded transfer error of the policy parameterization, we establish the theoretical properties of the algorithm. In particular, in contrast to existing CMDP approaches that ensure policy feasibility only upon convergence, our algorithm guarantees the feasibility of the policies during the learning process and converges to the $\varepsilon$-optimal policy with a sample complexity of $\tilde{\mathcal{O}}(\varepsilon{-6})$. In comparison to the state-of-the-art policy gradient-based algorithm, C-NPG-PDA, our algorithm requires an additional $\mathcal{O}(\varepsilon{-2})$ samples to ensure policy feasibility during learning with the same Fisher non-degenerate parameterization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Constrained policy optimization. In International conference on machine learning, pages 22–31. PMLR, 2017.
  2. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res., 22(98):1–76, 2021.
  3. Regret guarantees for model-based reinforcement learning with long-term average constraints. In Uncertainty in Artificial Intelligence, pages 22–31. PMLR, 2022.
  4. E. Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
  5. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469–483, 2009.
  6. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263–272. PMLR, 2017.
  7. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3682–3689, 2022.
  8. Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm. arXiv preprint arXiv:2206.05850, 2022.
  9. J. Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15:319–350, 2001.
  10. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
  11. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
  12. Conservative safety critics for exploration. arXiv preprint arXiv:2010.14497, 2020.
  13. J. V. Burke. Numerical optimization. course notes, amath/math 516. 2012.
  14. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3387–3395, 2019.
  15. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018.
  16. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031, 2019.
  17. Natural policy gradient primal-dual method for constrained Markov decision processes. Advances in Neural Information Processing Systems, 33:8378–8390, 2020.
  18. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. arXiv preprint arXiv:2206.02346, 2022.
  19. On the global optimum convergence of momentum-based policy gradient. In International Conference on Artificial Intelligence and Statistics, pages 1910–1934. PMLR, 2022.
  20. Stochastic policy gradient methods: Improved sample complexity for Fisher-non-degenerate policies, 2023.
  21. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning, pages 1467–1476. PMLR, 2018.
  22. A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control, 64(7):2737–2752, 2018.
  23. J. Garcıa and F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015.
  24. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330, 2022.
  25. H. Günzel and H. Th. Jongen. Strong stability implies mangasarian–fromovitz constraint qualification. Optimization, 55(5-6):605–610, 2006.
  26. Risk-aware motion planning and control using cvar-constrained optimization. IEEE Robotics and Automation Letters, 4(4):3924–3931, 2019.
  27. Model-based reinforcement learning for infinite-horizon discounted constrained markov decision processes. In International Joint Conference on Artificial Intelligence (IJCAI), 2021.
  28. Safe reinforcement learning using probabilistic shields. 2020.
  29. A. K. Jayant and S. Bhatnagar. Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems, 35:24432–24445, 2022.
  30. Contextual decision processes with low Bellman rank are PAC-learnable. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1704–1713. PMLR, 06–11 Aug 2017.
  31. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137–2143. PMLR, 2020.
  32. Safe posterior sampling for constrained mdps with bounded constraint violation. arXiv preprint arXiv:2301.11547, 2023.
  33. Deep constrained q-learning. arXiv preprint arXiv:2003.09398, 2020.
  34. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013.
  35. J. M. Kohler and A. Lucchi. Sub-sampled cubic regularization for non-convex optimization. In International Conference on Machine Learning, pages 1895–1904. PMLR, 2017.
  36. Learning-based model predictive control for safe exploration. In 2018 IEEE conference on decision and control (CDC), pages 6059–6066. IEEE, 2018.
  37. R. Koppejan and S. Whiteson. Neuroevolutionary reinforcement learning for generalized control of simulated helicopters. Evolutionary intelligence, 4:219–241, 2011.
  38. T. Lattimore and M. Hutter. PAC bounds for discounted mdps. In International Conference on Algorithmic Learning Theory, pages 320–334. Springer, 2012.
  39. Ai safety gridworlds. arXiv preprint arXiv:1711.09883, 2017.
  40. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint arXiv:1802.06480, 2018.
  41. Learning policies with zero or bounded constraint violation for constrained mdps. Advances in Neural Information Processing Systems, 34:17183–17193, 2021.
  42. Policy optimization for constrained mdps with provable fast global convergence, 2022.
  43. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4940–4947, 2020.
  44. Policy learning with constraints in model-free reinforcement learning: A survey. In IJCAI, pages 4508–4515, 2021.
  45. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems, 33:7624–7636, 2020.
  46. O. L. Mangasarian and S. Fromovitz. The Fritz John necessary optimality conditions in the presence of equality and inequality constraints. Journal of Mathematical Analysis and applications, 17(1):37–47, 1967.
  47. Stochastic second-order methods improve best-known sample complexity of sgd for gradient-dominated functions. Advances in Neural Information Processing Systems, 35:10862–10875, 2022.
  48. High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 2156–2162. IEEE, 2018.
  49. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  50. Chance-constrained dynamic programming with application to risk-aware robotic space exploration. Autonomous Robots, 39:555–571, 2015.
  51. M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  52. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
  53. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
  54. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017.
  55. Reinforcement learning: An introduction. MIT press, 2018.
  56. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
  57. Reward constrained policy optimization. In International Conference on Learning Representations, 2019.
  58. Log barriers for safe black-box optimization with application to safe reinforcement learning. arXiv preprint arXiv:2207.10415, 2022.
  59. Safe reinforcement learning for emergency load shedding of power systems. In 2021 IEEE Power & Energy Society General Meeting (PESGM), pages 1–5, 2021.
  60. Safe exploration and optimization of constrained mdps using gaussian processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  61. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations, 2020.
  62. A provably-efficient model-free algorithm for infinite-horizon average-reward constrained markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3868–3876, 2022.
  63. Triple-q: A model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation. In International Conference on Artificial Intelligence and Statistics, pages 3274–3307. PMLR, 2022.
  64. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, pages 5–32, 1992.
  65. Sample efficient policy gradient methods with recursive variance reduction. In International Conference on Learning Representations, 2020.
  66. Macro action selection with deep reinforcement learning in starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pages 94–99, 2019.
  67. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pages 11480–11491. PMLR, 2021.
  68. L. Yang and M. Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning, pages 6995–7004. PMLR, 2019.
  69. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152, 2020.
  70. Linear convergence of natural policy gradient methods with log-linear policies. arXiv preprint arXiv:2210.01400, 2022.
  71. A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics, pages 3332–3380. PMLR, 2022.
  72. Cmdp-based intelligent transmission for wireless body area network in remote health monitoring. Neural computing and applications, 32:829–837, 2020.
  73. Finite-time complexity of online primal-dual natural actor-critic algorithm for constrained markov decision processes. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 4028–4033. IEEE, 2022.
  74. Non-cooperative inverse reinforcement learning. Advances in neural information processing systems, 32, 2019.
  75. L. Zheng and L. Ratliff. Constrained upper confidence reinforcement learning. In Learning for Dynamics and Control, pages 620–629. PMLR, 2020.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now