Visual Whole-Body Control for Legged Loco-Manipulation

Visual Whole-Body Control for Legged Loco-Manipulation: Expanding the Horizons of Mobile Robots

Introduction to Visual Whole-Body Control (VBC)

In the pursuit of enhancing robotic mobility and manipulation capabilities beyond conventional limitations, Visual Whole-Body Control (VBC) emerges as a comprehensive framework developed by researchers from UC San Diego, Shanghai Jiao Tong University, and Fudan University. VBC is engineered to address the intricacies of mobile manipulation using legged robots equipped with arms, enabling these machines to handle objects across diverse environments autonomously, guided solely by visual inputs.

The Framework

VBC introduces a two-tiered architecture consisting of a low-level policy that manages all degrees of freedom for locomotion and manipulation and a high-level policy that proposes end-effector positions based on segmented depth images. Notably, this design allows for the seamless execution of actions in various settings without the need for task-specific fine-tuning, showcasing the framework's adaptability and efficiency.

Low-Level Policy

The low-level policy within VBC is meticulously engineered to track end-effector poses and robot body velocities, enabling the robot to accomplish whole-body behaviors across different terrains. This policy utilizes reinforcement learning (RL) techniques to adapt to given goals, thereby allowing the robot to interact with its environment robustly.

• Command and Observation: The policy receives specific commands, including end-effector position, orientation, and desired velocities. Observations fed into the policy encapsulate the robot's proprioceptive states and are enriched with environment extrinsics, fostering a dynamic response mechanism.
• Adaptation and Control: A significant innovation in the low-level policy is the incorporation of Regularized Online Adaptation (ROA), which facilitates sim-to-real transfer by accounting for environmental discrepancies such as terrain type and robot mass variation.

High-Level Task-Planning Policy

The high-level policy operates on visual observations to navigate and manipulate objects with precision. Given the challenges associated with high-frequency control through direct RL from raw visual data, the approach instead distills guidance from a privileged teacher policy, which possesses access to object shape and pose details, into a visuomotor student policy.

• Privileged Teacher Policy: The privileged teacher policy, trained via RL, leverages object shape features to refine the manipulation strategy, demonstrating the importance of detailed object understanding in achieving successful interaction.
• Visual-Motor Coordination: The visuomotor student policy, distilled from the teacher policy, derives commands from segmented depth images, illustrating the critical role of visual data in adapting to real-world conditions.

Real-World Deployment and Implications

VBC's practical application is manifest in its deployment on a Unitree B1 quadruped robot, equipped with a Unitree Z1 robotic arm. The system demonstrates remarkable proficiency in picking up a variety of objects across different heights and surfaces, heralding a new era of adaptability and autonomy in robotic systems.

• Sim-To-Real Transfer: Detailed measures, including domain randomization and depth image processing, ensure the seamless transition of the VBC framework from simulated environments to real-world scenarios, underscoring the effectiveness of the training methodologies.
• System Design and Challenges: Adjustments to the hardware and software architecture accommodate various operational frequencies, highlighting the consideration for systemic harmony in ensuring efficient task execution.

Conclusion and Future Directions

VBC presents a holistic strategy for enhancing the capabilities of legged robots in navigating and manipulating objects in complex environments, driven by visual inputs. The framework's success in sim-to-real transfer, combined with its hierarchical design, sets a strong foundation for further research focused on improving hardware components and simplification processes to overcome current limitations. Future developments may concentrate on refining visual input quality, enhancing gripper functionality, and streamlining system operations, promising an expansive trajectory for the evolution of mobile robotic manipulation.