Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenes (2404.12379v3)

Abstract: Modern 3D engines and graphics pipelines require mesh as a memory-efficient representation, which allows efficient rendering, geometry processing, texture editing, and many other downstream operations. However, it is still highly difficult to obtain high-quality mesh in terms of detailed structure and time consistency from dynamic observations. To this end, we introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct a high-fidelity and time-consistent mesh from dynamic input. Our work leverages the recent advancement in 3D Gaussian Splatting to construct the mesh sequence with temporal consistency from dynamic observations. Building on top of this representation, DG-Mesh recovers high-quality meshes from the Gaussian points and can track the mesh vertices over time, which enables applications such as texture editing on dynamic objects. We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians. By applying cycle-consistent deformation between the canonical and the deformed space, we can project the anchored Gaussian back to the canonical space and optimize Gaussians across all time frames. During the evaluation on different datasets, DG-Mesh provides significantly better mesh reconstruction and rendering than baselines. Project page: https://www.liuisabella.com/DG-Mesh

• The paper proposes DG-Mesh, a framework leveraging deformable 3D Gaussian splatting to reconstruct dynamic scenes with high fidelity.
• It introduces cycle-consistent deformation and Gaussian-Mesh Anchoring to ensure temporal consistency and explicit geometric detail.
• DG-Mesh outperforms baselines in metrics like Chamfer Distance and Earth Mover Distance, enabling effective texture editing and realistic mesh rendering.

Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenes

This essay provides a comprehensive summary of the paper "Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenes" (2404.12379). The paper focuses on addressing challenges in reconstructing high-fidelity, time-consistent meshes from dynamic scenes captured by monocular videos. It proposes the Dynamic Gaussians Mesh (DG-Mesh) framework, leveraging recent advancements in 3D Gaussian Splatting. The goal is to efficiently recover geometry and motion from videos, enabling applications such as texture editing with temporal consistency.

Introduction and Background

The paper introduces DG-Mesh as a solution to the memory-efficiency challenges and explicit geometric detail requirements faced by neural radiance fields (NeRFs) for dynamic scene reconstruction. Traditional volumetric models are often limited by their memory demands and lack of explicit geometric representation. Previous approaches like NeRF extensions [mildenhall2021nerf] have focused on deformable fields with time dimensions or latent codes, while 3D Gaussian Splatting [kerbl20233d] provides a memory-efficient and explicitly geometric point cloud representation. The DG-Mesh framework advances from these foundations, aiming to extract high-fidelity mesh sequences with temporal consistency from single videos.

Core Methodology

Deformable 3D Gaussian Splatting

The framework introduces deformable 3D Gaussian Splatting, capable of handling dynamic scenes by constructing a set of canonical 3D Gaussians. These Gaussians are transformed across different time steps using a cycle-consistent deformation process (Figure 1). The transformation utilizes position encoding with Fourier features [tancik2020fourier] to enhance reconstruction details. The deformable Gaussians are optimized in both the canonical space and transformed spaces, capturing explicit motion across time.

Figure 1: Main pipeline of DG-Mesh demonstrating the transformation of canonical 3D Gaussians into deformed spaces for mesh recovery.

Mesh Reconstruction and Optimization

DG-Mesh uses a differentiable Poisson Solver [peng2021shape] and Marching Cubes method to transform oriented Gaussian points into meshes. The method further involves a Gaussian-Mesh Anchoring process aimed at uniformly distributing Gaussians and aligning them with mesh faces (Figure 2). Anchoring involves densification where Gaussians are adjusted iteratively to match mesh topology, aiding in better surface representation and correspondence tracking.

Figure 2: Illustration of Gaussian-Mesh Anchoring procedure showing alignment and densification processes.

Mesh Correspondence and Cycle Consistency

To ensure consistency across mesh frames, DG-Mesh constructs mesh-to-point and point-to-canonical-point correspondences. The backward deformation module projects anchored Gaussians back to canonical space, enforcing cycle consistency between forward and backward transformations. This mechanism solidifies mesh cross-frame correspondence, critical for applications like texture transfer and seamless temporal mesh editing.

Training Objective

The training combines rendering losses from mesh rasterization and Gaussian splatting, augmented by Laplacian regularization for smoother surfaces. Anchoring and cycle-consistent deformation losses are included to stabilize point-cloud density and correspondence across time. The composite loss function effectively balances geometric fidelity and appearance rendering accuracy.

Results and Evaluation

Benchmark Performance

DG-Mesh exhibits superior performance in mesh reconstruction, particularly in handling intricate structures within dynamic scenes. It outperforms baselines such as D-NeRF, K-Plane, and others in terms of Chamfer Distance (CD) and Earth Mover Distance (EMD), indicating better geometry consistency (Table 1). Mesh rendering metrics show DG-Mesh achieves higher PSNR, SSIM, and LPIPS scores, demonstrating enhanced visual quality.

(Table 1: Not included in the paper)

Real-World Application

DG-Mesh's ability to provide time-consistent meshes lends itself to various applications, notably texture editing and ray-tracing. The method's capacity to handle flexible topology changes aids in scenarios requiring rapid adaptation to dynamic object transformations, as depicted in real-world dataset evaluations (Figure 3).

Figure 3: Applications of time-consistent mesh including ray-tracing and texture editing.

Conclusion

DG-Mesh marks a significant advancement in dynamic scene reconstruction by efficiently integrating memory-efficient Gaussian Splatting with high-fidelity mesh representation. The framework's cycle-consistent deformation and mesh anchoring processes set a robust precedent for dynamic object modeling from monocular video inputs. While promising, DG-Mesh acknowledges limitations regarding large topology changes, inviting future research to refine these aspects further. Nevertheless, its utility in graphics and simulation domains remains strong, offering enhanced dynamic content creation capabilities.