Shape of Motion: 4D Reconstruction from a Single Video (2407.13764v2)

Abstract: Monocular dynamic reconstruction is a challenging and long-standing vision problem due to the highly ill-posed nature of the task. Existing approaches depend on templates, are effective only in quasi-static scenes, or fail to model 3D motion explicitly. We introduce a method for reconstructing generic dynamic scenes, featuring explicit, persistent 3D motion trajectories in the world coordinate frame, from casually captured monocular videos. We tackle the problem with two key insights: First, we exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE(3) motion bases. Each point's motion is expressed as a linear combination of these bases, facilitating soft decomposition of the scene into multiple rigidly-moving groups. Second, we take advantage of off-the-shelf data-driven priors such as monocular depth maps and long-range 2D tracks, and devise a method to effectively consolidate these noisy supervisory signals, resulting in a globally consistent representation of the dynamic scene. Experiments show that our method achieves state-of-the-art performance for both long-range 3D/2D motion estimation and novel view synthesis on dynamic scenes. Project Page: https://shape-of-motion.github.io/

• The paper presents a novel method that decomposes dynamic scenes into SE motion bases, enabling persistent 4D motion reconstruction from a single video.
• It leverages monocular depth maps and 2D tracks in an optimization framework to achieve globally coherent 3D and 2D tracking with improved accuracy.
• Experimental results on synthetic and real-world datasets highlight significant gains in tracking precision and novel view synthesis fidelity.

Shape of Motion: 4D Reconstruction from a Single Video

Introduction

The paper investigates the problem of dynamic 3D scene reconstruction from monocular videos, presenting a method that captures persistent 3D motion trajectories using a single, casually captured video. This approach diverges from traditional methods that rely on templates, stationary scenes, or fail to model explicit 3D motion. By leveraging low-dimensional structures derived from groups of rigid motions and integrating data-driven priors like monocular depth maps and 2D tracks, the authors propose a novel 4D scene representation and optimization framework.

Methodology

The proposed method is based on two key insights: the simplification of 3D motion into rigid bases and the consolidation of noisy inputs into coherent scene information. The approach utilizes $SE$ motion bases, which represent the dynamic scene as a set of moving 3D Gaussians over the video. These Gaussians are manipulated through translations and rotations defined by the motion bases, leading to a continuous and persistent depiction of motion.

Motion Representation

The dynamic scene is encoded using a compact set of shared $SE$ motion bases rather than independent point trajectories, providing a soft decomposition into rigidly-moving parts. This representation enables effective tracking of 3D trajectories over extensive ranges, surpassing previous methods that only perform short-range motion estimation.

Optimization Framework

The optimization incorporates monocular depth and 2D tracks as input signals, refined through loss functions that maintain temporal smoothness and coherence among motion trajectories. By aligning the scene geometry with these priors, the method achieves a globally consistent representation and state-of-the-art performance in 3D tracking and novel view synthesis.

Experiments and Results

Extensive evaluations were performed on synthetic and real-world datasets, demonstrating superior accuracy and quality in long-range 3D/2D tracking and view synthesis. Notable datasets include the iPhone and Kubric MOVi-F datasets. The results showed significant improvements over existing methods in terms of end-point error, position accuracy, and visual fidelity of synthesized views.

Performance Metrics

• 3D Tracking: Achieved lower error rates and higher accuracy thresholds in metric scale evaluations.
• 2D Tracking: Showed improved Jaccard and position accuracy metrics, highlighting consistency across frames.
• Novel View Synthesis: Generated high-quality views with better PSNR, SSIM, and LPIPS scores.

  Figure 1: Novel view and motion coefficient PCA visualizations at time steps 0 and 54 of the school-girl sequence from the DAVIS dataset.

Challenges and Future Directions

The paper acknowledges limitations related to computational demands for optimization and reliance on initial depth and track estimates, which could be further refined with advances in sensor technology and segmentation methods. Future work may involve streamlining the optimization process for real-time applications and exploring feed-forward strategies for scene reconstruction without extensive test-time optimization.

Conclusion

This paper presents a significant advancement in 3D reconstruction from monocular videos, especially concerning dynamic scenes. The approach effectively consolidates different data inputs to produce coherent and persistent 4D motion trajectories. It opens pathways for enhanced real-time scene understanding applications, potentially transforming areas like autonomous systems, virtual reality, and robotics.