SWAGS: Sampling Windows Adaptively for Dynamic 3D Gaussian Splatting
(2312.13308)Abstract
Novel view synthesis has shown rapid progress recently, with methods capable of producing evermore photo-realistic results. 3D Gaussian Splatting has emerged as a particularly promising method, producing high-quality renderings of static scenes and enabling interactive viewing at real-time frame rates. However, it is currently limited to static scenes only. In this work, we extend 3D Gaussian Splatting to reconstruct dynamic scenes. We model the dynamics of a scene using a tunable MLP, which learns the deformation field from a canonical space to a set of 3D Gaussians per frame. To disentangle the static and dynamic parts of the scene, we learn a tuneable parameter for each Gaussian, which weighs the respective MLP parameters to focus attention on the dynamic parts. This improves the model's ability to capture dynamics in scenes with an imbalance of static to dynamic regions. To handle scenes of arbitrary length whilst maintaining high rendering quality, we introduce an adaptive window sampling strategy to partition the sequence into windows based on the amount of movement in the sequence. We train a separate dynamic Gaussian Splatting model for each window, allowing the canonical representation to change, thus enabling the reconstruction of scenes with significant geometric or topological changes. Temporal consistency is enforced using a fine-tuning step with self-supervising consistency loss on randomly sampled novel views. As a result, our method produces high-quality renderings of general dynamic scenes with competitive quantitative performance, which can be viewed in real-time with our dynamic interactive viewer.
Overview
-
Introduces SWAGS, a method for real-time novel view synthesis of dynamic scenes using adaptive 3D Gaussian Splatting.
-
Addresses challenges of dynamic scene reconstruction such as motion-induced blur and computational efficiency.
-
Employs Adaptive Window Sampling, dynamic 3D Gaussian modeling per window, and enforces temporal consistency.
-
Utilizes a PyTorch-based implementation for training and achieves real-time rendering with high-quality results.
-
Outperforms state-of-the-art techniques in rendering quality with FPS metrics and visual similarity assessments.
Overview
The paper introduces a novel approach to novel view synthesis called SWAGS (Sampling Windows Adaptively for Dynamic 3D Gaussian Splatting). Unlike previous methodologies that are typically restricted to static scenes, the new method extends the capabilities of 3D Gaussian Splatting, a technique that has gained attention for high-quality renderings, to reconstruct dynamic scenes for real-time interactive viewing.
Motivation and Challenges
Prevailing synthetic view techniques operating on 3D Gaussian principles face challenges with dynamic scenes, such as changing topologies, significant geometric modifications, or extended sequences leading to blurred results. Contemporary works that offer dynamic reconstruction often suffer from either lack of temporal consistency or heavy computational loads, making them impractical for real-time applications.
Innovative Approach
The introduced method tackles these challenges with several key innovations:
- Adaptive Window Sampling: The dynamic scenes are divided into windows of varying lengths based on motion intensity, enabling the handling of arbitrary-length sequences while maintaining high render quality.
- Per-Window Dynamic 3D Gaussian Splatting: Each window uses its own set of dynamic 3D Gaussians modeled by tuneable MLP (Multilayer Perceptron) parameters, which focus on dynamic region reconstruction to disentangle static and dynamic scene parts.
- Temporal Consistency Enforcement: Through a fine-tuning process utilizing self-supervising consistency loss on novel views, the method enforces temporal consistency between windows, significantly reducing flickering and ensuring smooth scene transition.
Training and Implementation
For the model training, the authors utilized a PyTorch-based implementation of 3D Gaussian Splatting, beginning with point cloud data from COLMAP. They benefitted from training each window's model in parallel to expedite the process and fine-tuned each afterward to ensure temporal consistency. Their adaptation allowed for real-time frame-rate rendering with superior reconstructive quality of dynamic scenes, including those with significant motion like flames.
Results and Contributions
The authors conducted extensive comparative studies, demonstrating that their method surpasses the current state-of-the-art techniques in rendering quality as assessed by PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index). Furthermore, it exhibits real-time performance at 71.51 FPS (Frames Per Second), a significant improvement over competing methods.
Conclusion
SWAGS revolutionizes the field of novel view synthesis by delivering high-fidelity, real-time interactive rendering of dynamic scenes previously unachievable due to the limitations of static model dependant methods. Through its adaptive window sampling, dynamic 3D Gaussian splatting, and groundbreaking fine-tuning practices, it sets a new benchmark for future developments in the domain.