Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 45 tok/s
Gemini 2.5 Pro 49 tok/s Pro
GPT-5 Medium 11 tok/s Pro
GPT-5 High 19 tok/s Pro
GPT-4o 88 tok/s Pro
Kimi K2 214 tok/s Pro
GPT OSS 120B 460 tok/s Pro
Claude Sonnet 4 36 tok/s Pro
2000 character limit reached

A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (2403.06323v2)

Published 10 Mar 2024 in cs.LG

Abstract: We study risk-sensitive RL where the goal is learn a history-dependent policy that optimizes some risk measure of cumulative rewards. We consider a family of risks called the optimized certainty equivalents (OCE), which captures important risk measures such as conditional value-at-risk (CVaR), entropic risk and Markowitz's mean-variance. In this setting, we propose two meta-algorithms: one grounded in optimism and another based on policy gradients, both of which can leverage the broad suite of risk-neutral RL algorithms in an augmented Markov Decision Process (MDP). Via a reductions approach, we leverage theory for risk-neutral RL to establish novel OCE bounds in complex, rich-observation MDPs. For the optimism-based algorithm, we prove bounds that generalize prior results in CVaR RL and that provide the first risk-sensitive bounds for exogenous block MDPs. For the gradient-based algorithm, we establish both monotone improvement and global convergence guarantees under a discrete reward assumption. Finally, we empirically show that our algorithms learn the optimal history-dependent policy in a proof-of-concept MDP, where all Markovian policies provably fail.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33:20095–20107, 2020.
  2. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431–4506, 2021.
  3. On the dual representation of coherent risk measures. Annals of Operations Research, 262:29–46, 2018.
  4. Coherent measures of risk. Mathematical finance, 9(3):203–228, 1999.
  5. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263–272. PMLR, 2017.
  6. Regret bounds for risk-sensitive reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=yJEUDfzsTX7.
  7. Markov decision processes with average-value-at-risk criteria. Mathematical Methods of Operations Research, 74(3):361–379, 2011.
  8. An old-new concept of convex risk measures: The optimized certainty equivalent. Mathematical Finance, 17(3):449–476, 2007.
  9. Algorithms for cvar optimization in mdps. Advances in neural information processing systems, 27, 2014.
  10. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1–51, 2018.
  11. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096–1105. PMLR, 2018.
  12. Risk-sensitive reinforcement learning: Iterated cvar and the worst path. arXiv preprint arXiv:2206.02678, 2022.
  13. Provable rl with exogenous distractors via multistep inverse dynamics. arXiv preprint arXiv:2110.08847, 2021.
  14. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. Advances in Neural Information Processing Systems, 33:22384–22395, 2020.
  15. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. Advances in Neural Information Processing Systems, 34:20436–20446, 2021.
  16. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 35:32639–32652, 2022.
  17. Mirror learning: A unifying framework of policy optimisation. In International Conference on Machine Learning, pages 7825–7844. PMLR, 2022.
  18. Risk-sensitive markov decision processes. Management science, 18(7):356–369, 1972.
  19. Revisiting design choices in proximal policy optimization. arXiv preprint arXiv:2009.10897, 2020.
  20. Reinforcement learning in low-rank mdps with density features. In International Conference on Machine Learning, pages 13710–13752. PMLR, 2023.
  21. Open problem: The dependence of sample complexity lower bounds on planning horizon. In Conference On Learning Theory, pages 3395–3398. PMLR, 2018.
  22. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34:13406–13418, 2021.
  23. Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
  24. Being optimistic to be conservative: Quickly learning a cvar policy. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4436–4443, 2020.
  25. Risk-aware reinforcement learning with coherent risk measures and non-linear function approximation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=-RwZOVybbj.
  26. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 35:30977–30989, 2022.
  27. Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004.14547, 2020.
  28. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34:19235–19247, 2021.
  29. HM Markowitz. Portfolio selection, the journal of finance. 7 (1). N, 1:71–91, 1952.
  30. Time limits in reinforcement learning. In International Conference on Machine Learning, pages 4045–4054. PMLR, 2018.
  31. One risk to rule them all: A risk-sensitive perspective on model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
  32. Optimization of conditional value-at-risk. Journal of risk, 2:21–42, 2000.
  33. Andrzej Ruszczyński. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125:235–261, 2010.
  34. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015.
  35. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  36. Reinforcement learning: An introduction. MIT press, 2018.
  37. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
  38. Policy gradient for coherent risk measures. Advances in neural information processing systems, 28, 2015a.
  39. Optimizing the cvar via sampling. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015b.
  40. Representation learning for online and offline RL in low-rank MDPs. In ICLR, 2022. URL https://openreview.net/forum?id=J4iSIR9fhY0.
  41. Risk-averse offline reinforcement learning. 2021.
  42. Near-minimax-optimal risk-sensitive reinforcement learning with cvar. International Conference on Machine Learning, 2023.
  43. More benefits of being distributional: Second-order bounds for reinforcement learning. arXiv preprint arXiv:2402.07198, 2024.
  44. Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1–36, 2022.
  45. The role of coverage in online reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=LQIjzPdDt3q.
  46. Regret bounds for markov decision processes with recursive optimized certainty equivalents. International Conference on Machine Learning, 2023.
  47. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning, pages 7304–7312. PMLR, 2019.
  48. Provably efficient cvar rl in low-rank mdps. arXiv preprint arXiv:2311.11965, 2023.
  49. Is risk-sensitive reinforcement learning properly resolved? arXiv preprint arXiv:2307.00547, 2023.
Citations (3)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets