InstantSplat: Sparse-view Gaussian Splatting in Seconds (2403.20309v5)

Abstract: While neural 3D reconstruction has advanced substantially, its performance significantly degrades with sparse-view data, which limits its broader applicability, since SfM is often unreliable in sparse-view scenarios where feature matches are scarce. In this paper, we introduce InstantSplat, a novel approach for addressing sparse-view 3D scene reconstruction at lightning-fast speed. InstantSplat employs a self-supervised framework that optimizes 3D scene representation and camera poses by unprojecting 2D pixels into 3D space and aligning them using differentiable neural rendering. The optimization process is initialized with a large-scale trained geometric foundation model, which provides dense priors that yield initial points through model inference, after which we further optimize all scene parameters using photometric errors. To mitigate redundancy introduced by the prior model, we propose a co-visibility-based geometry initialization, and a Gaussian-based bundle adjustment is employed to rapidly adapt both the scene representation and camera parameters without relying on a complex adaptive density control process. Overall, InstantSplat is compatible with multiple point-based representations for view synthesis and surface reconstruction. It achieves an acceleration of over 30x in reconstruction and improves visual quality (SSIM) from 0.3755 to 0.7624 compared to traditional SfM with 3D-GS.

• The paper introduces InstantSplat, a unified framework that integrates 3D Gaussian splatting with dense stereo priors to address sparse-view and pose-free novel view synthesis challenges.
• It employs a two-stage process with Coarse Geometric Initialization for fast scene and camera estimation followed by Fast 3D-Gaussian Optimization to refine parameters efficiently.
• Experimental results on Tanks & Temples demonstrate a 32% increase in SSIM and an 80% reduction in ATE, highlighting significant improvements in rendering quality and pose accuracy.

InstantSplat: Efficient and Unified Framework for Sparse-View 3D Reconstruction and Novel View Synthesis

Introduction

The newfound framework, named InstantSplat, introduces a novel methodology for addressing the challenges in novel view synthesis (NVS) under sparse-view and pose-free conditions. Through the integration of 3D Gaussian Splatting and Dense Stereo Priors, InstantSplat establishes itself as a potent solution for reconstructing 3D scenes and synthesizing novel views with high fidelity. The framework distinguishes itself by significantly improving both pose estimation accuracy and rendering quality, backed by strong numerical results on the Tanks and Temples datasets. The processes culminate in a robust system that can operate within one minute for large-scale scenes, marking a notable advancement in the field of 3D computer vision.

Key Contributions

• InstantSplat innovates by leveraging 3D Gaussian Splatting with dense stereo priors derived from an end-to-end dense stereo model (DUSt3R), effectively tackling sparse-view and pose-free challenges in NVS.
• The framework encompasses two main components: a Coarse Geometric Initialization (CGI) module for rapid preliminary scene structure and camera parameter estimation, and a Fast 3D-Gaussian Optimization (F-3DGO) module for joint optimization of 3D Gaussian attributes and initialized poses.
• Demonstrated improvements include a 32% increase in SSIM and an 80% reduction in Absolute Trajectory Error (ATE) on the Tanks & Temples datasets compared to existing methods, evidencing its capability to maintain high rendering quality and accurate pose estimation in sparse and unconditioned scenarios.

Methodology Overview

Coarse Geometric Initialization (CGI)

The CGI module harnesses the dense stereo model, DUSt3R, to predict globally aligned 3D point maps from sparse-view images. This alignment furnishes an initial geometric and photographic context that facilitates the rapid estimation of preliminary scene structures and camera parameters.

Fast 3D-Gaussian Optimization (F-3DGO)

Following CGI, the F-3DGO module employs these initial estimates to refine the 3D Gaussian attributes and camera poses further. It implements pose regularization, substantially enhancing the final pose accuracy and rendering quality through an efficient optimization process.

Experimental Insights

Extensive evaluations on the outdoor Tanks & Temples datasets underscore InstantSplat's superiority in sparse-view and pose-free scenarios. The method not only significantly outperforms existing pose-free methods in rendering quality but also showcases remarkable improvements in pose estimation accuracy.

Theoretical and Practical Implications

On a theoretical level, InstantSplat presents a novel approach to NVS tasks by combining explicit 3D representation with pose priors, diverging from the dependence on dense data coverage or prior knowledge of camera parameters. Practically, the method's efficiency and effectiveness in handling real-world scenarios indicate its potential applicability in areas such as digital twin construction, augmented reality, and beyond.

Future Directions

The current landscape of NVS under sparse-view conditions suggests a promising direction for future research to explore the integration of machine learning techniques with explicit 3D representations further. Developments in end-to-end systems capable of reconstructing and rendering scenes from extremely sparse and unconditioned inputs could revolutionize 3D content creation and visualization technologies.

Summary

InstantSplat represents a significant leap towards solving the long-standing challenges in novel view synthesis, specifically in sparse-view and pose-free settings. By proficiently merging the capabilities of dense stereo models with 3D Gaussian Splatting, it offers a fast, accurate, and practicable solution for 3D scene reconstruction and rendering, paving the way for next-generation 3D vision applications.