Designing a Skilled Soccer Team for RoboCup: Exploring Skill-Set-Primitives through Reinforcement Learning
(2312.14360)Abstract
The RoboCup 3D Soccer Simulation League serves as a competitive platform for showcasing innovation in autonomous humanoid robot agents through simulated soccer matches. Our team, FC Portugal, developed a new codebase from scratch in Python after RoboCup 2021. The team's performance is based on a set of skills centered around novel unifying primitives and a custom, symmetry-extended version of the Proximal Policy Optimization algorithm. Our methods have been thoroughly tested in official RoboCup matches, where FC Portugal has won the last two main competitions, in 2022 and 2023. This paper presents our training framework, as well as a timeline of skills developed using our skill-set-primitives, which considerably improve the sample efficiency and stability of skills, and motivate seamless transitions. We start with a significantly fast sprint-kick developed in 2021 and progress to the most recent skill set, which includes a multi-purpose omnidirectional walk, a dribble with unprecedented ball control, a solid kick, and a push skill. The push tackles both low-level collision-prone scenarios and high-level strategies to increase ball possession. We address the resource-intensive nature of this task through an innovative multi-agent learning approach. Finally, we release the codebase of our team to the RoboCup community, enabling other teams to transition to Python more easily and providing new teams with a robust and modern foundation upon which they can build new features.
Overview
-
RoboCup is a platform for AI and robotics research where autonomous robots play soccer, and FC Portugal has developed a new training framework.
-
The training framework utilizes skill-set-primitives, enabling robots to smoothly transition between behaviors such as walking, running, and ball handling.
-
A custom algorithm based on Proximal Policy Optimization with symmetry properties was used to train advanced skills and team strategies efficiently.
-
Results show FC Portugal's robots achieving high speeds and demonstrating improved tactical play and robust motor control, compliant with RoboCup rules.
-
FC Portugal released their codebase to the RoboCup community, providing a foundation for future AI and robotics research.
Background on RoboCup and Reinforcement Learning
RoboCup stands as an influential platform for advancing research in robotics and artificial intelligence. It simulates a soccer game where two teams of autonomous humanoid robots compete, providing a challenging and dynamic environment for developing and testing AI techniques. The use of reinforcement learning (RL) to train robots has become increasingly popular, with a focus on creating highly skilled robotic agents capable of displaying tactical acumen and collaboration with teammates. One of the primary challenges is forming a cohesive set of robot skills that work in harmony across different levels, from basic motor control to overall team strategy.
FC Portugal's Novel Approach
The paper describes an innovative training framework introduced by the FC Portugal team to enhance the performance of their soccer-playing robots. This framework is rooted in a concept called skill-set-primitives, which encapsulates recurring base actions facilitating easier transitions between complex behaviors, such as walking, running, and ball control. By applying this approach, the team has achieved significant success in the last two RoboCup 3D Soccer Simulation League competitions, demonstrating reinforced skills in tactical play and robust motor control.
Methodology and Results
The paper outlines the use of a custom algorithm, which extends Proximal Policy Optimization (PPO) with the ability to leverage symmetry properties for more efficient learning. This training took place over several stages, beginning with a fast sprint-kick behavior and progressing to advanced skills including an omnidirectional walk, a precise kick, and a close control dribble. The team also developed a push strategy to handle collisions between the robots, learning strategies and motor control in a multi-agent setting.
FC Portugal's robots, while primarily focusing on offensive skills, demonstrated speed, precision, and control in their movements. Notable achievements include a sprinting speed of 3.69 m/s and a maneuverable approach to ball control and kicking while maintaining dribble compliance with new RoboCup regulations.
Codebase Release and Implications
In an effort to contribute to the RoboCup community and AI research at large, the team has publicly released the codebase for their robot soccer team. This includes the reinforcement learning gym integrated into their system, enabling other teams and researchers to build upon the proven foundation laid by FC Portugal. The shared resources and detailed methodology pave the way for further innovation in the field of robotic soccer, with implications potentially reaching beyond the game into other areas of robotics and AI applications.
Create an account to read this summary for free: