How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey (2402.13255v3)

Abstract: Over the past two decades, research in the field of Simultaneous Localization and Mapping (SLAM) has undergone a significant evolution, highlighting its critical role in enabling autonomous exploration of unknown environments. This evolution ranges from hand-crafted methods, through the era of deep learning, to more recent developments focused on Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) representations. Recognizing the growing body of research and the absence of a comprehensive survey on the topic, this paper aims to provide the first comprehensive overview of SLAM progress through the lens of the latest advancements in radiance fields. It sheds light on the background, evolutionary path, inherent strengths and limitations, and serves as a fundamental reference to highlight the dynamic progress and specific challenges.

• The paper surveys the integration of NeRFs and 3D Gaussian Splatting in SLAM, demonstrating improvements in mapping accuracy and real-time rendering.
• It details methodologies like hierarchical encoding and local feature updates that tackle computational challenges and enhance system efficiency.
• The survey evaluates SLAM systems based on tracking accuracy, rendering quality, and memory efficiency, outlining future paths for dynamic environment adaptation.

How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey

Simultaneous Localization and Mapping (SLAM) is a key technology in robotics and computer vision, enabling autonomous systems to navigate and map environments. Recent advancements in Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) are paving the way for a new era in SLAM systems. This survey explores how these innovations are reshaping the landscape of SLAM, providing a comprehensive overview of the evolution from traditional to advanced radiance field-based SLAM methods.

Evolution of SLAM Methodologies

SLAM has evolved significantly over the past two decades, transitioning from classical hand-crafted methods to approaches leveraging deep learning technologies. The introduction of NeRFs and 3DGS has catalyzed a paradigm shift, enabling the development of SLAM systems with enhanced accuracy, robustness, and efficiency. NeRFs offer an implicit, continuous volumetric representation that facilitates high-fidelity view synthesis, while 3DGS provides an explicit representation for rendering 3D scenes using Gaussian primitives.

NeRF and 3DGS Representations

NeRFs represent scenes using a multi-layer perceptron (MLP) to approximate a 5D function that models volume density and color based on 3D spatial coordinates and viewing direction (Figure 1). This method excels in synthesizing novel views but suffers from slow training times and computational inefficiency due to per-pixel sampling.

Figure 1: NeRF representation queries an MLP to model radiance fields, facilitating view synthesis.

3DGS, on the other hand, utilizes 3D Gaussian primitives to explicitly represent scenes. This allows for real-time rendering using a rasterization approach, leading to improved rendering speeds and visual quality, although it requires substantial memory resources (Figure 2).

Figure 2: Comparison between NeRF and 3DGS: NeRF queries an MLP, while 3DGS blends Gaussians for ray synthesis.

Advances in Radiance Field-Based SLAM

This survey categorizes contemporary SLAM systems into those leveraging NeRFs and 3DGS, highlighting their innovations and challenges in real-world scenarios.

NeRF-Based SLAM Systems

NeRF-style SLAM systems, such as iMAP, have demonstrated potential in utilizing implicit neural representations for dense 3D mapping and camera tracking (Figure 3). However, challenges such as scalability, oversmoothing, and catastrophic forgetting remain. Recent works introduce hierarchical encoding strategies and local feature updates to enhance performance and reduce memory consumption.

Figure 3: iMAP utilizes MLP for real-time mapping, balancing tracking and global map updates.

3DGS-Based SLAM Systems

3DGS-based systems such as GS-SLAM leverage explicit representations to achieve high-quality image rendering while maintaining efficiency (Figure 4). These methods excel in generating detailed reconstructions with fewer artifacts and improved robustness to noise. However, the reliance on Gaussian initialization and management of growing primitives in unobserved regions pose challenges.

Figure 4: GS-SLAM framework uses Gaussian splatting for efficient mapping and rendering, enhancing scene reconstruction.

Evaluation and Comparative Analysis

The survey outlines the evaluation of SLAM systems using metrics such as tracking accuracy, rendering quality, and computational efficiency. Notably:

• Tracking Accuracy: Methods with external tracking and global optimization techniques outperform others in maintaining trajectory consistency.
• Rendering Quality: 3DGS-based solutions show superior rendering quality due to advanced rasterization, while NeRF approaches excel in detailed view synthesis.
• Computational Efficiency: Systems integrating efficient data structures like hash grids significantly reduce processing time and resource usage.

Future Directions and Challenges

The integration of NeRFs and 3DGS in SLAM offers promising avenues for advancements in autonomous navigation and augmented reality. However, several challenges persist, such as:

• Handling Dynamic Environments: Current systems primarily assume static scenes, limiting applications in dynamic contexts.
• Optimization for Real-Time Applications: Achieving real-time performance remains challenging due to complex computations and high resource demands.
• Memory Efficiency: 3DGS methods require large memory capacities, necessitating further research into compact and scalable representations.

Conclusion

By leveraging the capabilities of NeRFs and 3DGS, SLAM systems are poised for significant advancements in robustness and accuracy. While promising, these technologies are still in the evolutionary phase, requiring further exploration to address existing challenges and realize their full potential in practical applications. This survey serves as a foundational resource for researchers seeking to navigate the rapidly evolving landscape of SLAM.