DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing (2403.17822v3)

Abstract: High-fidelity 3D reconstruction of common indoor scenes is crucial for VR and AR applications. 3D Gaussian splatting, a novel differentiable rendering technique, has achieved state-of-the-art novel view synthesis results with high rendering speeds and relatively low training times. However, its performance on scenes commonly seen in indoor datasets is poor due to the lack of geometric constraints during optimization. In this work, we explore the use of readily accessible geometric cues to enhance Gaussian splatting optimization in challenging, ill-posed, and textureless scenes. We extend 3D Gaussian splatting with depth and normal cues to tackle challenging indoor datasets and showcase techniques for efficient mesh extraction. Specifically, we regularize the optimization procedure with depth information, enforce local smoothness of nearby Gaussians, and use off-the-shelf monocular networks to achieve better alignment with the true scene geometry. We propose an adaptive depth loss based on the gradient of color images, improving depth estimation and novel view synthesis results over various baselines. Our simple yet effective regularization technique enables direct mesh extraction from the Gaussian representation, yielding more physically accurate reconstructions of indoor scenes.

• The paper introduces depth and normal priors to refine 3D Gaussian splatting, aligning primitive representations with actual scene surfaces.
• It employs a gradient-aware logarithmic depth loss and total variation regularization to mitigate noise from commercial depth sensors.
• The method optimizes photometric, depth, and normal losses to achieve smoother, more accurate meshes compared to state-of-the-art techniques.

Depth and Normal Supervision Enhancements for 3D Gaussian Splatting and Mesh Reconstruction

Gaussian Splatting with Depth and Normal Priors

The technique of 3D Gaussian splatting (3DGS) represents a compelling approach toward inverse rendering, characterized by the usage of differentiable 3D Gaussian primitives. Although 3DGS boasts real-time rendering capabilities and an interoperable scene representation, it grapples with geometric ambiguities and artifacts due to its lack of 3D and surface constraints during optimization. This paper introduces a depth and normal regularization method aimed at refining 3D Gaussian splatting for indoor scene reconstruction. By incorporating depth and smoothness priors and aligning Gaussians with scene geometry through monocular normal cues, the method enhances photorealism and geometric fidelity.

Incorporating Depth Information

The method leverages per-pixel depth estimates, determined by a discrete volume rendering approximation, to enforce geometric constraints. Acknowledging the noise properties of common commercial depth sensors, a gradient-aware logarithmic depth loss, alongside a total variation loss to promote smoothness, is employed. This regularization strategy is informed by depth priors obtained from sensors or inferred through monocular depth estimation networks for datasets without depth data, proving beneficial in reducing ambiguities in texture-less or poorly observed regions of indoor scenes.

Normal Estimation and Regularization

By deriving normals directly from the geometry of 3D Gaussians, the paper ensures an adaptive alignment of Gaussian primitives with the real surface boundaries of the scene. This approach eschews additional learnable parameters for normal prediction, favoring a regularization strategy grounded in the geometry of the Gaussians themselves. Monocular normal priors, obtained from off-the-shelf networks, serve as a supervision signal, providing smoother and more geometrically plausible results compared to normals estimated from depth gradients.

Optimization and Mesh Extraction

The optimization loss amalgamates photometric loss with depth and normal regularization losses, striking a balance that faithfully represents scene geometry while minimizing visual artifacts. Extending beyond optimization, the paper explores direct mesh extraction from the Gaussian representation via Poisson surface reconstruction. The enhanced depth and normal estimation contribute to more accurate and smoother reconstructions, showcasing the method's superiority in extracting meshable surfaces directly from optimized Gaussian scenes.

Experimental Validation

The effectiveness of the proposed regularization strategy is demonstrated across various indoor datasets. When compared to state-of-the-art methods in 3D reconstruction, including NeRF and SDF-based models, our approach manifests noteworthy improvements in both photorealism and geometric accuracy. Particularly in challenging real-world scenes from the MuSHRoom and ScanNet++ datasets, the method outperforms baseline models in depth estimation and novel view synthesis.

Conclusion and Future Prospects

This paper evidences the potential of depth and normal priors in refining the quality of 3D Gaussian splatting for scene reconstruction. By converging towards more realistic depictions of indoor environments, the proposed method sets a promising direction for future developments in inverse rendering. The adaptation to sparser or more challenging data captures, alongside the exploration of more sophisticated mesh extraction techniques, are identified as pivotal avenues for further research in the domain of 3D computer vision and graphics.