Mesh-based Gaussian Splatting for Real-time Large-scale Deformation
(2402.04796)Abstract
Neural implicit representations, including Neural Distance Fields and Neural Radiance Fields, have demonstrated significant capabilities for reconstructing surfaces with complicated geometry and topology, and generating novel views of a scene. Nevertheless, it is challenging for users to directly deform or manipulate these implicit representations with large deformations in the real-time fashion. Gaussian Splatting(GS) has recently become a promising method with explicit geometry for representing static scenes and facilitating high-quality and real-time synthesis of novel views. However,it cannot be easily deformed due to the use of discrete Gaussians and lack of explicit topology. To address this, we develop a novel GS-based method that enables interactive deformation. Our key idea is to design an innovative mesh-based GS representation, which is integrated into Gaussian learning and manipulation. 3D Gaussians are defined over an explicit mesh, and they are bound with each other: the rendering of 3D Gaussians guides the mesh face split for adaptive refinement, and the mesh face split directs the splitting of 3D Gaussians. Moreover, the explicit mesh constraints help regularize the Gaussian distribution, suppressing poor-quality Gaussians(e.g. misaligned Gaussians,long-narrow shaped Gaussians), thus enhancing visual quality and avoiding artifacts during deformation. Based on this representation, we further introduce a large-scale Gaussian deformation technique to enable deformable GS, which alters the parameters of 3D Gaussians according to the manipulation of the associated mesh. Our method benefits from existing mesh deformation datasets for more realistic data-driven Gaussian deformation. Extensive experiments show that our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate(65 FPS on average).
Overview
-
Introduction of a mesh-based Gaussian Splatting approach for efficient rendering and detailed appearance capture in computer graphics.
-
Binding of 3D Gaussians to a mesh which serves as a guide for learning distributions and for user-driven deformations.
-
New deformation method allowing alteration of Gaussian parameters according to mesh manipulation, resulting in high-quality real-time rendering.
-
Superior performance against existing approaches in real-time deformation and view synthesis tasks, as confirmed by qualitative and quantitative analysis.
Introduction to Mesh-based Gaussian Splatting
The field of computer graphics and geometry processing has long leveraged explicit representations such as point clouds, voxels, and meshes for their intuitive properties and ease of manipulation. However, such methods struggle with capturing high-fidelity appearance and complex geometry. Although implicit representations like Neural Radiance Fields (NeRF) and Signed Distance Fields (SDF) have shown promise in these areas, they suffer from slow rendering speeds.
Gaussian Splatting (GS) has recently emerged as a compelling alternative, offering both efficient rendering and detailed appearance capture. Nonetheless, interacting with these representations, especially under large deformations, remains a challenge. This paper introduces an innovative mesh-based GS representation to tackle the issue of interactive, real-time deformations in large-scale scenarios.
Mesh-based GS Representation and Its Implementation
The core innovation lies in binding 3D Gaussians to an explicit mesh, which serves as both a structural guide for learning GS distributions and a canvas for user-driven deformations. The process involves defining 3D Gaussians over mesh vertices and using the mesh topology to guide Gaussian rendering and manipulation. This technique suppresses poor-quality Gaussians and enhances visual fidelity, avoiding artifacts during deformation.
A new deformation method is introduced, allowing for the alteration of Gaussian parameters according to mesh manipulation. Extensive experimentation has demonstrated the system's ability to maintain high-quality reconstruction and efficient deformation while operating at an average frame rate of 65 FPS on a standard GPU setup.
Relative Work and Methodologies
The paper showcases the advantages of GS representations, such as lower training costs and quality real-time rendering over NeRFs. Yet, existing methods based on sparse control points falter with complex geometries and large-scale deformation, indicating a need for better topology awareness. The work plays a pioneering role in mesh-based deformation for 3DGS, utilizing shell properties and explicit mesh representation to supply topology information vital for realistic and interactive large-scale deformation.
Experimental Analysis and Key Results
Extensive evaluations against current techniques illustrate superior efficiency and deformation quality achieved by the proposed method. For novel view synthesis tasks, the approach outperforms baselines in key metrics like Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), and it is comparable in Learned Perceptual Image Patch Similarity (LPIPS).
Furthermore, qualitative analyses provide visual confirmation that the approach effectively preserves detail in deformation scenarios. The system is robust to different levels of explicit mesh resolution, showcasing its versatility and practicality in various use cases.
Conclusion
The research takes a significant step toward practical, real-time manipulation of complex 3D forms by proposing a novel, mesh-based approach to GS. The introduced method stands out in its ability to deform GS representations in real-time, balancing the necessity for high visual fidelity with user-friendly interactivity. While future work may address potential limitations, such as extending capabilities to modify the visual features of Gaussians or handling transparent objects, the presented system clearly marks a paradigm shift in managing large-scale deformations in GS-based representations.