RaDe-GS: Rasterizing Depth in Gaussian Splatting
(2406.01467)Abstract
Gaussian Splatting (GS) has proven to be highly effective in novel view synthesis, achieving high-quality and real-time rendering. However, its potential for reconstructing detailed 3D shapes has not been fully explored. Existing methods often suffer from limited shape accuracy due to the discrete and unstructured nature of Gaussian splats, which complicates the shape extraction. While recent techniques like 2D GS have attempted to improve shape reconstruction, they often reformulate the Gaussian primitives in ways that reduce both rendering quality and computational efficiency. To address these problems, our work introduces a rasterized approach to render the depth maps and surface normal maps of general 3D Gaussian splats. Our method not only significantly enhances shape reconstruction accuracy but also maintains the computational efficiency intrinsic to Gaussian Splatting. It achieves a Chamfer distance error comparable to NeuraLangelo on the DTU dataset and maintains similar computational efficiency as the original 3D GS methods. Our method is a significant advancement in Gaussian Splatting and can be directly integrated into existing Gaussian Splatting-based methods.
Overview
-
The paper 'RaDe-GS: Rasterizing Depth in Gaussian Splatting' proposes an innovative rasterization method to enhance 3D shape reconstruction capabilities while preserving computational efficiency in Gaussian Splatting techniques.
-
The method introduces a closed-form solution to efficiently compute depth and surface normal maps for Gaussian splats by evaluating light ray intersections under local affine projections, improving the fidelity of 3D models.
-
Experimental results on datasets such as DTU and Tanks & Temples demonstrate that RaDe-GS matches or surpasses state-of-the-art methods in shape reconstruction accuracy and novel view synthesis while maintaining rapid training times.
RaDe-GS: Rasterizing Depth in Gaussian Splatting
Overview
The paper "RaDe-GS: Rasterizing Depth in Gaussian Splatting" introduces a novel rasterization approach to enhancing the shape reconstruction capabilities of Gaussian Splatting (GS) methods, while maintaining computational efficiency. Although Gaussian Splatting has excelled in high-quality, real-time rendering for novel view synthesis, its potential for detailed 3D shape reconstruction remained underutilized. Previous approaches often compromised between rendering quality, shape accuracy, and computational efficiency. This paper proposes a method that retains the effectiveness of GS for rendering while significantly improving the accuracy of shape reconstruction.
Methodology
The primary contribution of the paper is the development of a rasterized method to compute depth and surface normal maps for general 3D Gaussian splats. The core innovation lies in the derivation of a closed-form solution for the intersection of light rays and Gaussian splats, evaluated under local affine projections. The method identifies the intersection points for each light ray and computes their depth and surface normal in an efficient, rasterized manner.
Depth Computation
Depth computation involves identifying the intersection point of each light ray with the Gaussian splat. By evaluating Gaussian values along the light ray from the camera center, the method determines the point of intersection as the point where these values are maximized. The derived depth value is thus spatially varying within the projected 2D Gaussian splat, leading to a more accurate 3D surface modeling. This is a departure from standard GS, which assigns a single depth value to each Gaussian splat.
Normal Calculation
For surface normal computation, the paper demonstrates that under a local affine projection, the intersection points between a Gaussian splat and a set of rays lie on a plane. The normal vector for this plane is computed and then transformed back to the camera coordinate system. This method ensures that the derived surface normals are consistent with the depth values computed earlier, further improving the fidelity of the 3D models.
Experimental Results
Surface Reconstruction
The paper evaluates the proposed method on established datasets such as DTU and Tanks & Temples (TNT), showcasing the method's capabilities in surface reconstruction. The Chamfer Distance errors reported for the DTU dataset reveal that RaDe-GS performs comparably to state-of-the-art methods like NeuraLangelo, despite being based on explicit rather than implicit representations.
- DTU Dataset: Achieved a shape reconstruction accuracy with a Chamfer Distance error of 0.69 mm, closely matching the implicit method NeuraLangelo (0.61 mm) and surpassing other GS-based methods such as GOF (0.74 mm).
- TNT Dataset: Demonstrated a substantial improvement in F1-scores compared to other explicit GS-based methods while maintaining a similar optimization time (~17 minutes).
Novel View Synthesis
On the Mip-NeRF360 and Synthetic NeRF datasets, the proposed method outperforms other GS-based methods in terms of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM), and LPIPS (Learned Perceptual Image Patch Similarity).
- Mip-NeRF360 Dataset: RaDe-GS consistently achieves higher PSNR, SSIM, and lower LPIPS compared to previous methods, particularly excelling in outdoor scenes.
- Synthetic-NeRF Dataset: The proposed method showcases superior performance, with the highest average PSNR across several scenes.
Computational Efficiency
RaDe-GS maintains the computational efficiency intrinsic to Gaussian Splatting. The reported training times highlight that while implicit methods like Neuralangelo require over 24 hours per scene, RaDe-GS maintains a rapid training time of approximately 17.8 minutes, making it remarkably efficient for both novel view synthesis and 3D shape reconstruction.
Implications and Future Directions
The implications of RaDe-GS extend to various fields requiring high-quality 3D reconstruction and novel view rendering, such as augmented reality, virtual reality, and robotic navigation. The method's ability to retain computational efficiency while significantly improving the accuracy of 3D shape reconstruction makes it highly applicable for real-time applications.
Future developments could focus on integrating multi-resolution TSDF fusion for large-scale scenes and addressing reflective surface reconstruction by leveraging advanced color representations like GaussianShader. Enhancing the depth and normal computation further could yield even more detailed and accurate 3D models, broadening the method's applicability across different types of scenes and objects.
Conclusion
The paper "RaDe-GS: Rasterizing Depth in Gaussian Splatting" successfully introduces a novel approach to rendering depth and surface normals for Gaussian splats, significantly enhancing 3D shape reconstruction accuracy while maintaining the computational advantages of Gaussian Splatting. Through rigorous experiments and comparisons, it demonstrates the method's superior performance in both novel view synthesis and surface reconstruction, providing a robust foundation for further advancements in 3D modeling and rendering technologies.