Curiosity-driven Exploration for Mapless Navigation with Deep Reinforcement Learning (1804.00456v2)

Abstract: This paper investigates exploration strategies of Deep Reinforcement Learning (DRL) methods to learn navigation policies for mobile robots. In particular, we augment the normal external reward for training DRL algorithms with intrinsic reward signals measured by curiosity. We test our approach in a mapless navigation setting, where the autonomous agent is required to navigate without the occupancy map of the environment, to targets whose relative locations can be easily acquired through low-cost solutions (e.g., visible light localization, Wi-Fi signal localization). We validate that the intrinsic motivation is crucial for improving DRL performance in tasks with challenging exploration requirements. Our experimental results show that our proposed method is able to more effectively learn navigation policies, and has better generalization capabilities in previously unseen environments. A video of our experimental results can be found at https://goo.gl/pWbpcF.

• The paper introduces a novel DRL framework that integrates intrinsic curiosity rewards to boost exploration in mapless navigation.
• It employs an Intrinsic Curiosity Module alongside an A3C approach to enhance policy convergence and sample efficiency.
• Experimental results demonstrate that curiosity-driven exploration enables robust policy generalization in complex and unfamiliar environments.

Curiosity-driven Exploration for Mapless Navigation with Deep Reinforcement Learning

In the context of Deep Reinforcement Learning (DRL) applied to autonomous mobile robots, the exploration strategies are crucial for learning efficient navigation policies. The paper "Curiosity-driven Exploration for Mapless Navigation with Deep Reinforcement Learning" by Zhelo et al. introduces a novel approach to augment the conventional DRL training with intrinsic curiosity-based rewards. This research puts forth significant advancements in the domain of mapless navigation by leveraging the capability of DRL to eliminate traditional path planning and localization processes.

Exploration Strategy and DRL

The paper emphasizes the central role of DRL in bypassing traditional robotic navigation tasks, such as Simultaneous Localization and Mapping (SLAM), by exploiting the representation learning capabilities of deep neural networks. The authors propose and test an exploration strategy that augments the standard external rewards with intrinsic curiosity-driven signals—rewards designed to encourage the navigation agent to explore underrepresented states in the training environment. This approach aims to further assist the agent in contriving efficient navigation policies, particularly in environments with complex exploration challenges, such as long corridors and dead ends.

Methodology

The research employs an Intrinsic Curiosity Module (ICM) as a means to generate intrinsic motivation. This module, as outlined by Pathak et al., allows the agent to quantify the novelty of its states based on prediction errors stemming from its own actions. A feature extraction layer processes laser range sensor inputs to produce state representations, which are subsequently used in forward and inverse model predictions to derive intrinsic rewards based on prediction errors.

The authors adopt an Asynchronous Advantage Actor Critic (A3C) approach for training the navigation policies, utilizing both extrinsic (environmental) and intrinsic (curiosity-based) rewards. The training emphasizes optimizing policy and value networks to maximize expected rewards across exploration exploits. Notably, the weighting of external versus intrinsic rewards is dynamically adjusted to enhance the learning process.

Results and Implications

The experimental setups are carefully designed, utilizing simulated environments and varied map layouts to assess the agent's capability. The results underscore the superiority of curiosity-driven exploration over state-independent methods, such as entropy-based random exploration, as seen in both training scenarios and generalization tests. Notably, the combination of ICM with entropy-based exploration yields the most promising outcomes in achieving efficient and robust navigation policies.

The paper reports on substantial improvements in sample efficiency and policy convergence, emphasizing the intrinsic rewards' significant role in guiding agents out of local minima. In testing, policies trained with curiosity-based strategies exhibit enhanced adaptability to unfamiliar environments, demonstrating the model's generalization prowess.

Future Directions

This research paves the way for future exploration of dynamic weighting strategies between extrinsic and intrinsic rewards. Moreover, the authors propose further work to validate the application of these exploration strategies in real-world robotic settings, potentially expanding their relevance to practical applications where mapping and localization constraints pose significant challenges. The adaptation of DRL with intrinsic curiosity may prove transformative in robotics, particularly in developing autonomous systems that reliably operate in unexplored and dynamic settings.

By exploring intrinsic motivation's role in navigation policy learning, this paper enhances our understanding of DRL's potential to guide autonomous robots in complex environments, opening avenues for future advancements in intelligent agent exploration strategies.