Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Risk-Sensitive RL with Optimized Certainty Equivalents via Reduction to Standard RL (2403.06323v1)

Published 10 Mar 2024 in cs.LG

Abstract: We study Risk-Sensitive Reinforcement Learning (RSRL) with the Optimized Certainty Equivalent (OCE) risk, which generalizes Conditional Value-at-risk (CVaR), entropic risk and Markowitz's mean-variance. Using an augmented Markov Decision Process (MDP), we propose two general meta-algorithms via reductions to standard RL: one based on optimistic algorithms and another based on policy optimization. Our optimistic meta-algorithm generalizes almost all prior RSRL theory with entropic risk or CVaR. Under discrete rewards, our optimistic theory also certifies the first RSRL regret bounds for MDPs with bounded coverability, e.g., exogenous block MDPs. Under discrete rewards, our policy optimization meta-algorithm enjoys both global convergence and local improvement guarantees in a novel metric that lower bounds the true OCE risk. Finally, we instantiate our framework with PPO, construct an MDP, and show that it learns the optimal risk-sensitive policy while prior algorithms provably fail.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33:20095–20107, 2020.
  2. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431–4506, 2021.
  3. On the dual representation of coherent risk measures. Annals of Operations Research, 262:29–46, 2018.
  4. Coherent measures of risk. Mathematical finance, 9(3):203–228, 1999.
  5. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263–272. PMLR, 2017.
  6. Regret bounds for risk-sensitive reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=yJEUDfzsTX7.
  7. Markov decision processes with average-value-at-risk criteria. Mathematical Methods of Operations Research, 74(3):361–379, 2011.
  8. An old-new concept of convex risk measures: The optimized certainty equivalent. Mathematical Finance, 17(3):449–476, 2007.
  9. Algorithms for cvar optimization in mdps. Advances in neural information processing systems, 27, 2014.
  10. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1–51, 2018.
  11. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096–1105. PMLR, 2018.
  12. Risk-sensitive reinforcement learning: Iterated cvar and the worst path. arXiv preprint arXiv:2206.02678, 2022.
  13. Provable rl with exogenous distractors via multistep inverse dynamics. arXiv preprint arXiv:2110.08847, 2021.
  14. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. Advances in Neural Information Processing Systems, 33:22384–22395, 2020.
  15. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. Advances in Neural Information Processing Systems, 34:20436–20446, 2021.
  16. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 35:32639–32652, 2022.
  17. Mirror learning: A unifying framework of policy optimisation. In International Conference on Machine Learning, pages 7825–7844. PMLR, 2022.
  18. Risk-sensitive markov decision processes. Management science, 18(7):356–369, 1972.
  19. Revisiting design choices in proximal policy optimization. arXiv preprint arXiv:2009.10897, 2020.
  20. Reinforcement learning in low-rank mdps with density features. In International Conference on Machine Learning, pages 13710–13752. PMLR, 2023.
  21. Open problem: The dependence of sample complexity lower bounds on planning horizon. In Conference On Learning Theory, pages 3395–3398. PMLR, 2018.
  22. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34:13406–13418, 2021.
  23. Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
  24. Being optimistic to be conservative: Quickly learning a cvar policy. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4436–4443, 2020.
  25. Risk-aware reinforcement learning with coherent risk measures and non-linear function approximation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=-RwZOVybbj.
  26. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 35:30977–30989, 2022.
  27. Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004.14547, 2020.
  28. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34:19235–19247, 2021.
  29. HM Markowitz. Portfolio selection, the journal of finance. 7 (1). N, 1:71–91, 1952.
  30. Time limits in reinforcement learning. In International Conference on Machine Learning, pages 4045–4054. PMLR, 2018.
  31. One risk to rule them all: A risk-sensitive perspective on model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
  32. Optimization of conditional value-at-risk. Journal of risk, 2:21–42, 2000.
  33. Andrzej Ruszczyński. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125:235–261, 2010.
  34. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015.
  35. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  36. Reinforcement learning: An introduction. MIT press, 2018.
  37. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
  38. Policy gradient for coherent risk measures. Advances in neural information processing systems, 28, 2015a.
  39. Optimizing the cvar via sampling. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015b.
  40. Representation learning for online and offline RL in low-rank MDPs. In ICLR, 2022. URL https://openreview.net/forum?id=J4iSIR9fhY0.
  41. Risk-averse offline reinforcement learning. 2021.
  42. Near-minimax-optimal risk-sensitive reinforcement learning with cvar. International Conference on Machine Learning, 2023.
  43. More benefits of being distributional: Second-order bounds for reinforcement learning. arXiv preprint arXiv:2402.07198, 2024.
  44. Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1–36, 2022.
  45. The role of coverage in online reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=LQIjzPdDt3q.
  46. Regret bounds for markov decision processes with recursive optimized certainty equivalents. International Conference on Machine Learning, 2023.
  47. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning, pages 7304–7312. PMLR, 2019.
  48. Provably efficient cvar rl in low-rank mdps. arXiv preprint arXiv:2311.11965, 2023.
  49. Is risk-sensitive reinforcement learning properly resolved? arXiv preprint arXiv:2307.00547, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Kaiwen Wang (24 papers)
  2. Dawen Liang (17 papers)
  3. Nathan Kallus (133 papers)
  4. Wen Sun (124 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets