Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning (2105.08328v1)

Abstract: Accurate and precise terrain estimation is a difficult problem for robot locomotion in real-world environments. Thus, it is useful to have systems that do not depend on accurate estimation to the point of fragility. In this paper, we explore the limits of such an approach by investigating the problem of traversing stair-like terrain without any external perception or terrain models on a bipedal robot. For such blind bipedal platforms, the problem appears difficult (even for humans) due to the surprise elevation changes. Our main contribution is to show that sim-to-real reinforcement learning (RL) can achieve robust locomotion over stair-like terrain on the bipedal robot Cassie using only proprioceptive feedback. Importantly, this only requires modifying an existing flat-terrain training RL framework to include stair-like terrain randomization, without any changes in reward function. To our knowledge, this is the first controller for a bipedal, human-scale robot capable of reliably traversing a variety of real-world stairs and other stair-like disturbances using only proprioception.

• The paper demonstrates a novel sim-to-real reinforcement learning approach enabling Cassie to traverse stairs without relying on vision-based perception.
• The paper employs dynamics randomization and LSTM-based policies, resulting in high success rates for stair ascent and descent under varied conditions.
• The paper reveals a trade-off between stability on stairs and energy efficiency, suggesting that integrating supplementary sensory inputs could enhance adaptability.

Analysis of "Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning"

This paper presents an exploration into the domain of blind bipedal locomotion, specifically tasked with stair traversal for human-scale robotic platforms using reinforcement learning (RL). The focal point of the research is the application of sim-to-real RL methodologies to enable the bipedal robot, Cassie, to handle stair-like terrains without external perception or terrain models, relying solely on proprioceptive feedback.

Research Overview

The primary challenge in robotic locomotion discussed in this paper is the need for robust systems that can operate in various real-world environments without the fragility introduced by precise terrain estimation, which is challenging to achieve in uncontrolled settings. Prior approaches largely depend on accurate vision-based systems for navigation, which potentially introduces vulnerabilities due to sensor occlusion or misinterpretations under varying environmental conditions. This paper investigates the possibility of achieving reliable stair traversal using reflex-driven, proprioceptive control policies generated through a modified RL framework.

Methodological Approach

The paper employs a sim-to-real RL paradigm utilizing an existing flat-terrain training framework with adaptations to include stair-like terrain during training. Importantly, the reward function remains consistent with previous implementations, underscoring the versatility of the approach. The RL formulation optimized to produce stable gaits involves significant contributions:

• State Representation: Includes the robot's physical state, command inputs for direction and speed, and cyclic swing leg phases to effectively synchronize gait patterns.
• Action Representation: Involves joint targets and stepping frequency adjustments to cope with the stair disturbances.

Dynamics randomization is effectively leveraged to generalize the learned policies across various terrains and dynamics, thus facilitating successful real-world deployments.

Numerical and Experimental Results

The results demonstrated that the stair-trained LSTM policies achieved notable success rates in simulation trials for both ascending and descending stairs. Specifically, success rates were contingent on the robot's approach velocity, optimizing performance within moderate speed ranges. The requirement of a memory feature—evidenced by the superior performance of LSTM over non-recurrent models—suggests that temporal context integration is critical for managing environmental uncertainties.

On flat ground, the Stair LSTM policies exhibited increased cost of transport compared to those trained without stair modifications, signifying a trade-off between robustness and energy efficiency. Interestingly, augmenting proprioception with stair proximity inputs partially recovered efficiency, showing potential for improved adaptability in hybrid perception strategies.

Practical Implications and Future Direction

The demonstrated capability of Cassie traversing real-world stairs without vision sensor dependency is a meaningful stride toward developing independent robotic systems with enhanced operational reliability in unstructured environments. With this foundation, integrating vision-based methodologies can potentially enhance the efficiency and situational adaptability by supplementing high-level planning, opening avenues for further enhancements in autonomous navigation tasks.

This paper posits the significant question of the limits of proprioceptive control in bipedal robots, and future work is suggested to explore these boundaries by integrating multiple sensory modalities. The successful transfer into real-world settings paves the way for further advancements to encompass diverse terrain navigation and dynamically adaptive behaviors, propelling forward the capabilities of autonomous bipedal robots in complex and varied human environments.