GaussianPro: 3D Gaussian Splatting with Progressive Propagation

Enhancing 3D Gaussian Splatting with Progressive Propagation for Neural Rendering

Introduction to GaussianPro

In the quest for real-time neural rendering, the advent of 3D Gaussian Splatting (3DGS) represented a significant leap forward, thanks to its efficiency and the quality of the renderings it could produce. However, it's evident that the initial excitement around 3DGS was tempered by its dependency on sparse Structure-from-Motion (SfM) point clouds for initialization. This reliance introduces notable limitations, particularly in large-scale scenes featuring texture-less surfaces, where SfM techniques struggle to generate sufficient points for effective 3DGS initialization. Addressing these challenges, our work introduces GaussianPro, a novel approach that enhances the densification of 3D Gaussians by leveraging a progressive propagation strategy. Our method bridges the gap between classical multi-view stereo (MVS) techniques and modern neural rendering, significantly improving rendering quality.

Methodology

Hybrid Geometric Representation

We approached the challenge by combining 3D Gaussians with 2D view-dependent depth and normal maps. This hybrid representation allows us to leverage 2D image space, facilitating the efficient determination of neighboring Gaussians and propagating geometric information among them. Our method operates by projecting 3D Gaussians onto 2D space to generate depth and normal maps. These maps guide the growth of Gaussians by informed densification.

Progressive Gaussian Propagation

Central to our approach is the progressive Gaussian propagation strategy. This technique utilizes patch matching to propagate depth and normal information from neighboring pixels, generating new, more accurate Gaussians. This method not only addresses the challenges posed by texture-less regions but also compensates for the limitations of sparse SfM point clouds. Furthermore, by employing geometric filtering and selection, we refine the propagation results, ensuring that new Gaussians are generated where necessary to model the scene accurately.

Experimental Evaluations

Our experiments, conducted on both large-scale and small-scale scenes, demonstrate the effectiveness of our method. On the Waymo dataset, GaussianPro surpassed the rendering quality of 3DGS, achieving a significant improvement of 1.15dB in PSNR. Similarly, on the MipNeRF360 dataset, our method showcased its robustness and adaptability, delivering comparable or superior performance to 3DGS across various metrics.

Integration of Planar Constraint

An innovating aspect of our methodology is the introduction of a planar constraint during the optimization process. This constraint enforces consistency between the Gaussian's rendered normal and the propagated normal, thus improving the geometrical accuracy of the Gaussians. The results underscore the capacity of our method to generate more compact and accurate Gaussians, leading to enhanced rendering quality, particularly in scenes with prevalent planar surfaces.

Conclusion and Future Work

GaussianPro represents a substantial step forward in neural rendering, addressing the critical issues of 3D Gaussian densification and optimization in texture-less regions. By integrating insights from classical MVS and exploiting the strength of modern neural rendering techniques, our method not only enhances the visual quality of renderings but also maintains computational efficiency. While our current focus has been on static scenes, future developments could extend this approach to dynamic objects, offering a comprehensive solution for real-time, high-quality neural rendering across a broader spectrum of applications.