Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Policy-regularized Offline Multi-objective Reinforcement Learning (2401.02244v1)

Published 4 Jan 2024 in cs.LG and cs.AI

Abstract: In this paper, we aim to utilize only offline trajectory data to train a policy for multi-objective RL. We extend the offline policy-regularized method, a widely-adopted approach for single-objective offline RL problems, into the multi-objective setting in order to achieve the above goal. However, such methods face a new challenge in offline MORL settings, namely the preference-inconsistent demonstration problem. We propose two solutions to this problem: 1) filtering out preference-inconsistent demonstrations via approximating behavior preferences, and 2) adopting regularization techniques with high policy expressiveness. Moreover, we integrate the preference-conditioned scalarized update method into policy-regularized offline RL, in order to simultaneously learn a set of policies using a single policy network, thus reducing the computational cost induced by the training of a large number of individual policies for various preferences. Finally, we introduce Regularization Weight Adaptation to dynamically determine appropriate regularization weights for arbitrary target preferences during deployment. Empirical results on various multi-objective datasets demonstrate the capability of our approach in solving offline MORL problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. A distributional view on multi-objective policy optimization. In International conference on machine learning. PMLR, 11–22.
  2. Dynamic weights in multi-objective deep reinforcement learning. In International conference on machine learning. PMLR, 11–20.
  3. Multi-objective reinforcement learning with non-linear scalarization. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 9–17.
  4. Pd-morl: Preference-driven multi-objective reinforcement learning algorithm. arXiv preprint arXiv:2208.07914 (2022).
  5. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems 34 (2021), 15084–15097.
  6. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751 (2021).
  7. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 (2020).
  8. Scott Fujimoto and Shixiang Shane Gu. 2021. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems 34 (2021), 20132–20145.
  9. Addressing function approximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587–1596.
  10. Off-policy deep reinforcement learning without exploration. In International conference on machine learning. PMLR, 2052–2062.
  11. Hisashi Handa. 2009. Solving multi-objective reinforcement learning problems by EDA-RL-acquisition of various strategies. In 2009 ninth international conference on intelligent systems design and applications. IEEE, 426–431.
  12. Grounded action transformation for sim-to-real reinforcement learning. Machine Learning 110, 9 (2021), 2469–2499.
  13. Champion-level drone racing using deep reinforcement learning. Nature 620, 7976 (2023), 982–987.
  14. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems 32 (2019).
  15. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems 33 (2020), 1179–1191.
  16. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020).
  17. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems 45, 3 (2014), 385–398.
  18. Multi-objective deep reinforcement learning. arXiv preprint arXiv:1610.02707 (2016).
  19. Pareto Conditioned Networks. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 1110–1118.
  20. Linear support for multi-objective coordination graphs. In AAMAS’14: PROCEEDINGS OF THE 2014 INTERNATIONAL CONFERENCE ON AUTONOMOUS AGENTS & MULTIAGENT SYSTEMS, Vol. 2. IFAAMAS/ACM, 1297–1304.
  21. Richard S Sutton and Andrew G Barto. 2018. Reinforcement Learning: An Introduction. MIT press.
  22. Preventing undesirable behavior of intelligent machines. Science 366, 6468 (2019), 999–1004.
  23. Multi-objective spibb: Seldonian offline policy improvement with safety constraints in finite mdps. Advances in Neural Information Processing Systems 34 (2021), 2004–2017.
  24. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine learning 84 (2011), 51–80.
  25. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 7782 (2019), 350–354.
  26. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 (2023).
  27. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 (2022).
  28. Offline constrained multi-objective reinforcement learning via pessimistic dual value iteration. Advances in Neural Information Processing Systems 34 (2021), 25439–25451.
  29. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 (2019).
  30. Prediction-guided multi-objective reinforcement learning for continuous robot control. In International conference on machine learning. PMLR, 10607–10616.
  31. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems 32 (2019).
  32. DRN: A deep reinforcement learning framework for news recommendation. In Proceedings of the 2018 World Wide Web Conference. 167–176.
  33. Scaling pareto-efficient decision making via offline multi-objective rl. arXiv preprint arXiv:2305.00567 (2023).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Qian Lin (79 papers)
  2. Chao Yu (116 papers)
  3. Zongkai Liu (9 papers)
  4. Zifan Wu (8 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets