On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer

Introduction

In robot manipulation learning, the particular control commands used—referred to as the "action space"—can greatly impact how robots learn tasks and how well these tasks transfer from simulated environments to the real world. Traditionally, robot learning policies directly handle low-level commands like joint torques, but more recent trends involve policies that output higher-level commands—like desired joint velocities—which are then translated into low-level commands by engineered controllers. This paper investigates how these different types of action spaces affect robot learning and simulation-to-reality transfer.

Methodology and Experiment Design

The authors conducted extensive experiments training over 250 reinforcement learning (RL) agents in simulated reaching and pushing tasks using 13 different control spaces. The action spaces evaluated range from traditional controls, such as joint torques (JT), to novel configurations that integrate embedded controllers converting policy directives into robot joint torques.

To compare these spaces, several metrics are used. For example, success rate transfer measures how successful a policy learned in simulation is when applied to the real world. The usability of behaviors looks at how often policies violate robot constraints like acceleration and jerk limits. Lastly, the researchers measure the gap introduced when transferring learned behaviors into a physical environment.

Results and Findings

Analysis of the training performance in the simulated environment showed that action spaces with higher-order control variables (like joint velocities) were more efficient. Delta action spaces, which adjust control targets based on current feedback, were also beneficial but required careful tuning of hyperparameters. In real-world deployment, joint velocity action spaces generally exhibited better transferability. Policies developed with these spaces approached tasks with higher accuracy and lower rates of constraint violation, translating to more reliable real-world performance.

Conclusion

The study's findings underscore the importance of action space selection in RL for robotic manipulation. The research identifies joint velocity-based action spaces as the most effective for learning and transferring manipulation tasks, primarily due to their ability to more easily accommodate dynamic interactions and enforce smoothness constraints on control trajectories. These insights can guide future designs of RL algorithms and suggest that integrating feedback mechanisms may reduce the sim-to-real gap and improve overall performance in real-world robotic applications.