InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds

InstantSplat: Efficient and Unified Framework for Sparse-View 3D Reconstruction and Novel View Synthesis

Introduction

The newfound framework, named InstantSplat, introduces a novel methodology for addressing the challenges in novel view synthesis (NVS) under sparse-view and pose-free conditions. Through the integration of 3D Gaussian Splatting and Dense Stereo Priors, InstantSplat establishes itself as a potent solution for reconstructing 3D scenes and synthesizing novel views with high fidelity. The framework distinguishes itself by significantly improving both pose estimation accuracy and rendering quality, backed by strong numerical results on the Tanks and Temples datasets. The processes culminate in a robust system that can operate within one minute for large-scale scenes, marking a notable advancement in the field of 3D computer vision.

Key Contributions

• InstantSplat innovates by leveraging 3D Gaussian Splatting with dense stereo priors derived from an end-to-end dense stereo model (DUSt3R), effectively tackling sparse-view and pose-free challenges in NVS.
• The framework encompasses two main components: a Coarse Geometric Initialization (CGI) module for rapid preliminary scene structure and camera parameter estimation, and a Fast 3D-Gaussian Optimization (F-3DGO) module for joint optimization of 3D Gaussian attributes and initialized poses.
• Demonstrated improvements include a 32% increase in SSIM and an 80% reduction in Absolute Trajectory Error (ATE) on the Tanks & Temples datasets compared to existing methods, evidencing its capability to maintain high rendering quality and accurate pose estimation in sparse and unconditioned scenarios.

Methodology Overview

Coarse Geometric Initialization (CGI)

The CGI module harnesses the dense stereo model, DUSt3R, to predict globally aligned 3D point maps from sparse-view images. This alignment furnishes an initial geometric and photographic context that facilitates the rapid estimation of preliminary scene structures and camera parameters.

Fast 3D-Gaussian Optimization (F-3DGO)

Following CGI, the F-3DGO module employs these initial estimates to refine the 3D Gaussian attributes and camera poses further. It implements pose regularization, substantially enhancing the final pose accuracy and rendering quality through an efficient optimization process.

Experimental Insights

Extensive evaluations on the outdoor Tanks & Temples datasets underscore InstantSplat's superiority in sparse-view and pose-free scenarios. The method not only significantly outperforms existing pose-free methods in rendering quality but also showcases remarkable improvements in pose estimation accuracy.

Theoretical and Practical Implications

On a theoretical level, InstantSplat presents a novel approach to NVS tasks by combining explicit 3D representation with pose priors, diverging from the dependence on dense data coverage or prior knowledge of camera parameters. Practically, the method's efficiency and effectiveness in handling real-world scenarios indicate its potential applicability in areas such as digital twin construction, augmented reality, and beyond.

Future Directions

The current landscape of NVS under sparse-view conditions suggests a promising direction for future research to explore the integration of machine learning techniques with explicit 3D representations further. Developments in end-to-end systems capable of reconstructing and rendering scenes from extremely sparse and unconditioned inputs could revolutionize 3D content creation and visualization technologies.

Summary

InstantSplat represents a significant leap towards solving the long-standing challenges in novel view synthesis, specifically in sparse-view and pose-free settings. By proficiently merging the capabilities of dense stereo models with 3D Gaussian Splatting, it offers a fast, accurate, and practicable solution for 3D scene reconstruction and rendering, paving the way for next-generation 3D vision applications.