C-3PO: Cyclic-Three-Phase Optimization for Human-Robot Motion Retargeting based on Reinforcement Learning (1909.11303v3)

Abstract: Motion retargeting between heterogeneous polymorphs with different sizes and kinematic configurations requires a comprehensive knowledge of (inverse) kinematics. Moreover, it is non-trivial to provide a kinematic independent general solution. In this study, we developed a cyclic three-phase optimization method based on deep reinforcement learning for human-robot motion retargeting. The motion retargeting learning is performed using refined data in a latent space by the cyclic and filtering paths of our method. In addition, the human-in-the-loop based three-phase approach provides a framework for the improvement of the motion retargeting policy by both quantitative and qualitative manners. Using the proposed C-3PO method, we were successfully able to learn the motion retargeting skill between the human skeleton and motion of the multiple robots such as NAO, Pepper, Baxter and C-3PO.

• The paper introduces a novel three-phase cyclic optimization approach for human-robot motion retargeting that enhances policy robustness using reinforcement learning.
• It leverages latent space mapping and an n-step Monte-Carlo method to bypass traditional inverse kinematics and achieve superior training rewards.
• A unified encoder-decoder policy is fine-tuned via direct teaching, enabling effective motion retargeting across multiple robotic platforms.

Overview of C-3PO: Cyclic-Three-Phase Optimization for Human-Robot Motion Retargeting

The paper presents a novel approach to human-robot motion retargeting through a method named Cyclic-Three-Phase Optimization (C-3PO). This method leverages reinforcement learning to enable robotic systems to retarget human movements across different robotic platforms. The main contributions of this paper revolve around the introduction of an advanced three-phase framework to efficiently learn and refine a motion retargeting policy.

Key Contributions

1. Innovative Architecture: The paper introduces a novel architecture employing a cyclic path combined with filtering mechanisms. This is a significant deviation from prior work which employed a simpler network structure without emphasizing cyclic processing. The introduction of this path serves to mitigate issues related to data noise and inadequate evaluation of actor performance, thus enhancing the robustness of the learned policies.
2. Latent Space Utilization: The C-3PO framework utilizes deep reinforcement learning (DRL) to map human skeleton data to robot motion, bypassing traditional inverse kinematics (IK) methods. The use of a latent space allows for refined data processing and improved motion retargeting outcomes.
3. Monte-Carlo Reinforcement Learning: The paper opts for an n-step Monte-Carlo (MC) reinforcement learning approach, modeling the task as a non-Markovian problem. This choice offers a higher performance trade-off compared to temporal-difference (TD) methods, particularly in environments where only positional information is available.
4. Unified Policy with Fine-Tuning: A unified policy is developed for multiple motion classes, supported by an encoder-decoder framework. The policy is further optimized through direct teaching (DT) based fine-tuning to improve precision in motion retargeting.

Strong Numerical Results and Claims

The numerical results underscore the efficacy of the C-3PO approach, particularly its robustness across different robotic configurations, such as NAO, Pepper, Baxter, and C-3PO robots. In comparative analyses, the filtering and cyclic path elements contributed to superior training outcomes when contrasted with simpler architectures. The n-step MC approach also demonstrated improved rewards over TD methods, reinforcing the value of MC reinforcement learning in non-episodic environments.

Implications and Speculations

From a theoretical standpoint, C-3PO provides a critical step forward in the domain of robot learning from human demonstration, offering practical insights into motion retargeting without the need for intricate kinematic modeling. Practically, this method has considerable implications for the development of more adaptable and intuitive human-robot interaction systems, especially in environments demanding a high degree of motion fidelity.

Looking to the future, the framework's potential extension to complex HRI tasks could markedly advance the state of robotics in everyday human-centric applications. Additionally, resolving the challenges of retaining pre-learned skills during fine-tuning and addressing motion ambiguity through trajectory-based approaches could further refine the efficacy of motion retargeting frameworks.

In conclusion, this work marks a notable progression in reinforcement learning applications for motion retargeting, setting the stage for more sophisticated developments in robotic imitation learning and interactive capabilities. The C-3PO method's integration of cyclic paths, latent space utilization, and advanced RL techniques provides a comprehensive model for future research in AI-driven robotic systems.