Dota 2 with Large Scale Deep Reinforcement Learning (1912.06680v1)

Abstract: On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems. OpenAI Five leveraged existing reinforcement learning techniques, scaled to learn from batches of approximately 2 million frames every 2 seconds. We developed a distributed training system and tools for continual training which allowed us to train OpenAI Five for 10 months. By defeating the Dota 2 world champion (Team OG), OpenAI Five demonstrates that self-play reinforcement learning can achieve superhuman performance on a difficult task.

• The paper demonstrates that scaling deep reinforcement learning enabled AI to beat top Dota 2 champions.
• It details a distributed PPO training approach with continuous self-play and innovative 'surgery' techniques for adapting to game updates.
• The results reveal significant breakthroughs in high-dimensional decision-making and long-horizon strategic planning under partial observability.

OpenAI Five: Mastering Dota 2 with Large Scale Deep Reinforcement Learning

The paper "Dota 2 with Large Scale Deep Reinforcement Learning" presents OpenAI's efforts to develop an AI system capable of mastering the complex multiplayer game Dota 2, resulting in the creation of OpenAI Five. This system is notable for being the first AI to defeat the reigning world champions at Dota 2, Team OG, in a high-stakes competitive environment.

Overview of OpenAI Five

OpenAI Five leverages state-of-the-art reinforcement learning (RL) methods extended to unprecedented scales. Specifically, it employs Proximal Policy Optimization (PPO) across a vast distributed system using thousands of GPU cores. The system underwent continuous training for 10 months, learning from a staggering number of frames at approximately 2-million frames every 2 seconds.

Several aspects of Dota 2 present unique challenges for AI:

• Long Time Horizons: The game lasts for roughly 45 minutes, requiring planning across approximately 20,000 steps per match.
• Partial Observability: Players can only see a fraction of the game map, demanding precise inference and strategic speculation based on limited information.
• High-Dimensional Action and Observation Spaces: The game’s complexity involves numerous possible actions and observable states that challenge the AI in decision-making.

OpenAI Five demonstrated that scaling RL methods can lead to superhuman performance. The system utilized self-play, where identical policies played against each other, and tools like "surgery" to adapt continually to changes in game mechanics and policy without restarting training from scratch.

Training Infrastructure

The training infrastructure includes several interconnected systems:

1. Rollout Workers: These are CPU machines running self-play games. They interact with the game engine and communicate with GPU machines for inferencing actions.
2. Optimizer Machines: These GPUs perform gradient updates asynchronously, incorporating data transmitted by the rollout workers.
3. Controller: A central repository storing model parameters and coordinating between machines.

Throughput was addressed by separating game engine interactions (handled by rollouts) and policy forward passes (handled by GPUs). This allowed the system to handle large batches efficiently, often between 1 to 3 million timesteps per batch, far surpassing prior methods seen in works like AlphaGo. The computation required for training over this period was around 770,000 PFlops/s-days.

Continual Training via Surgery

During the long training period, game updates and modifications to the environment necessitated a method to carry over trained policies without restarting. The paper introduces "surgery," a method that allowed the team to modify and expand the model or environment while maintaining performance. This approach was key in facilitating frequent updates and additions of new features without compromising the integrity of the ongoing training process.

Experimental Validation

To validate the efficacy of scaling and "surgery," a secondary training run named Rerun was executed. Rerun used the final environment and architecture, aiming to replicate OpenAI Five’s performance without intermediate changes. It required significantly less compute, highlighting the improvements facilitated by the initial development phase.

Performance and Impact

OpenAI Five's performance was validated through several matches against human players, including professional tier players and teams. The system consistently demonstrated high skill levels and creative in-game strategies, signified by:

• Human-Like Strategic Actions: Effective resource concentration, careful hero positioning, and optimized combat tactics.
• Distinct Playstyle: Different in hero movement patterns and risk assessment, often performing actions considered risky by human players with calculated precision.

During a public showcase, "OpenAI Five Arena," the system played against thousands of human teams, achieving a 99.4% win rate. This extensive human-centric evaluation supports the model's robustness and adaptability across diverse scenarios.

Future Prospects

OpenAI Five's success indicates that scalable reinforcement learning can tackle complex, high-dimensional environments by leveraging vast computational resources and algorithmic innovations. Future AI systems will increasingly require such scalable solutions as tasks grow in complexity. Further research into methods like "surgery" will be crucial in maintaining continuous learning without degradation across dynamic problem domains.

Conclusion

OpenAI Five's triumph in Dota 2 underscores the potential for large-scale deep reinforcement learning to address and solve intricate problems in real-world-like environments. The project highlights the importance of scalability, data handling, and ongoing adaptation to develop models capable of achieving sophisticated tasks, paving the way for more advanced AI applications in varied fields.