Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Integrating Distributed Architectures in Highly Modular RL Libraries (2007.02622v3)

Published 6 Jul 2020 in cs.CV, cs.AI, and cs.LG

Abstract: Advancing reinforcement learning (RL) requires tools that are flexible enough to easily prototype new methods while avoiding impractically slow experimental turnaround times. To match the first requirement, the most popular RL libraries advocate for highly modular agent composability, which facilitates experimentation and development. To solve challenging environments within reasonable time frames, scaling RL to large sampling and computing resources has proved a successful strategy. However, this capability has been so far difficult to combine with modularity. In this work, we explore design choices to allow agent composability both at a local and distributed level of execution. We propose a versatile approach that allows the definition of RL agents at different scales through independent reusable components. We demonstrate experimentally that our design choices allow us to reproduce classical benchmarks, explore multiple distributed architectures, and solve novel and complex environments while giving full control to the user in the agent definition and training scheme definition. We believe this work can provide useful insights to the next generation of RL libraries.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
  2. OpenAI. Openai five. https://blog.openai.com/openai-five/, 2018.
  3. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
  4. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
  5. Pybullet, a python module for physics simulation in robotics, games and machine learning, 2017.
  6. The animal-ai olympics. Nature Machine Intelligence, 1(5):257–257, 2019.
  7. Obstacle tower: A generalization challenge in vision, control, and planning. arXiv preprint arXiv:1902.01378, 2019.
  8. The minerl competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv:1904.10079, 2019.
  9. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339–9347, 2019.
  10. Reinvent 2.0: an ai tool for de novo drug design. Journal of chemical information and modeling, 60(12):5918–5922, 2020.
  11. Controlling commercial cooling systems using reinforcement learning. arXiv preprint arXiv:2211.07357, 2022.
  12. A graph placement methodology for fast chip design. Nature, 594(7862):207–212, 2021.
  13. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1–6, 2022. URL http://jmlr.org/papers/v23/21-1127.html.
  14. Rllib: Abstractions for distributed reinforcement learning. arXiv preprint arXiv:1712.09381, 2017.
  15. Vincent Moens. TorchRL: an open-source Reinforcement Learning (RL) library for PyTorch, 2023. URL https://github.com/pytorch/rl.
  16. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018.
  17. Recurrent experience replay in distributed reinforcement learning. 2018.
  18. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017.
  19. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv, pages arXiv–1911, 2019.
  20. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024–8035, 2019.
  21. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL http://jmlr.org/papers/v22/20-1364.html.
  22. DI engine Contributors. DI-engine: OpenDILab decision intelligence engine. https://github.com/opendilab/DI-engine, 2021.
  23. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022. URL http://jmlr.org/papers/v23/21-1342.html.
  24. The garage contributors. Garage: A toolkit for reproducible reinforcement learning research. https://github.com/rlworkgroup/garage, 2019.
  25. OpenSpiel: A framework for reinforcement learning in games. CoRR, abs/1908.09453, 2019. URL http://arxiv.org/abs/1908.09453.
  26. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018.
  27. TF-Agents: A library for reinforcement learning in tensorflow. https://github.com/tensorflow/agents, 2018. URL https://github.com/tensorflow/agents. [Online; accessed 25-June-2019].
  28. Tensorforce: a tensorflow library for applied reinforcement learning. Web page, 2017. URL https://github.com/tensorforce/tensorforce.
  29. Matthias Plappert. keras-rl. https://github.com/keras-rl/keras-rl, 2016.
  30. Benchmarking deep reinforcement learning for continuous control, 2016.
  31. Reinforcement learning coach, December 2017. URL https://doi.org/10.5281/zenodo.1134899.
  32. Mushroomrl: Simplifying reinforcement learning research. Journal of Machine Learning Research, 22(131):1–5, 2021. URL http://jmlr.org/papers/v22/18-056.html.
  33. Openai baselines. https://github.com/openai/baselines, 2017.
  34. RLgraph: Modular Computation Graphs for Deep Reinforcement Learning. In Proceedings of the 2nd Conference on Systems and Machine Learning (SysML), April 2019.
  35. Ray: A distributed framework for emerging {{\{{AI}}\}} applications. In 13th {normal-{\{{USENIX}normal-}\}} Symposium on Operating Systems Design and Implementation ({normal-{\{{OSDI}normal-}\}} 18), pages 561–577, 2018.
  36. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020.
  37. Launchpad: A programming model for distributed machine learning research. arXiv preprint arXiv:2106.04516, 2021. URL https://arxiv.org/abs/2106.04516.
  38. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
  39. Prioritized experience replay, 2016.
  40. Hindsight experience replay, 2018.
  41. Boosting soft actor-critic: Emphasizing recent experience without forgetting the past, 2019.
  42. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937, 2016.
  43. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  44. Rainbow: Combining improvements in deep reinforcement learning, 2017.
  45. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
  46. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587–1596. PMLR, 2018.
  47. Soft actor-critic algorithms and applications. ArXiv, abs/1812.05905, 2018.
  48. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018.
  49. Openai gym, 2016.
  50. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012.
  51. Trifinger: An open-source robot for learning dexterity. arXiv preprint arXiv:2008.03596, 2020.
  52. Accelerated methods for deep reinforcement learning. arXiv preprint arXiv:1803.02811, 2018.
  53. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018.
  54. Facebook Incubator. Submitit. https://github.com/facebookincubator/submitit, 2021. [GitHub repository].
  55. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
  56. Obstacle tower video demo. YouTube video. URL https://youtu.be/L442rrVnDr4. Accessed on May 31, 2023.
  57. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321, 2019.
  58. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  59. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Albert Bou (8 papers)
  2. Sebastian Dittert (4 papers)
  3. Gianni De Fabritiis (39 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now