NoPose-NeuS: Jointly Optimizing Camera Poses with Neural Implicit Surfaces for Multi-view Reconstruction
(2312.15238)Abstract
Learning neural implicit surfaces from volume rendering has become popular for multi-view reconstruction. Neural surface reconstruction approaches can recover complex 3D geometry that are difficult for classical Multi-view Stereo (MVS) approaches, such as non-Lambertian surfaces and thin structures. However, one key assumption for these methods is knowing accurate camera parameters for the input multi-view images, which are not always available. In this paper, we present NoPose-NeuS, a neural implicit surface reconstruction method that extends NeuS to jointly optimize camera poses with the geometry and color networks. We encode the camera poses as a multi-layer perceptron (MLP) and introduce two additional losses, which are multi-view feature consistency and rendered depth losses, to constrain the learned geometry for better estimated camera poses and scene surfaces. Extensive experiments on the DTU dataset show that the proposed method can estimate relatively accurate camera poses, while maintaining a high surface reconstruction quality with 0.89 mean Chamfer distance.
Overview
-
The paper addresses 3D geometry reconstruction from multi-view images and challenges with non-Lambertian surfaces and thin objects.
-
Camera poses are crucial for 3D reconstruction, but precise poses are hard to achieve with casually taken images.
-
The proposed method jointly optimizes camera poses and scene geometry using Neural Implicit Surface models, with additional loss functions for accuracy.
-
Tests on the DTU dataset demonstrate the method's accuracy in camera pose estimation and surface reconstruction without premapped camera parameters.
-
The success suggests future research directions in minimizing dependence on initial camera pose initialization and intrinsic camera parameter optimization.
Introduction to Neural Implicit Surfaces
The task of reconstructing 3D geometry from multi-view images presents significant challenges, particularly when dealing with complex structures such as non-Lambertian surfaces and thin objects. Classical Multi-view Stereo (MVS) methods have limitations in handling areas lacking clear texture, while recent neural rendering approaches impose stringent requirements on camera parameter knowledge.
Camera Pose Challenges and Innovations
In 3D geometry reconstruction, a typical condition is to have precise camera poses, which is often unachievable with casually captured images. Some innovative methods have addressed this by optimizing camera parameters alongside rendering. The paper under consideration extends this idea to Neural Implicit Surface (NeuS) models by proposing a method for simultaneous camera pose and scene geometry optimization. This approach introduces additional loss functions that cater to reinforcing the accuracy of the estimated camera positions and the fidelity of the scene's surfaces.
Methodology and Contributions
The key aspects of the proposed method involve encoding camera poses into a multi-layer perceptron (MLP), then refining these poses through multi-view feature consistency and depth losses. Qualitatively and quantitatively evaluated on the DTU dataset, the method showcases impressive accuracy in both camera pose estimation and surface reconstruction, displaying an ability to maintain high reconstruction quality while eschewing reliance on pre-determined camera parameters.
Findings and Further Potential
Demonstrated results offer a comprehensive analysis, confirming the method's proficiency in estimations and reconstructions. Although the camera pose initializations bear significant weight on the outcome, future research angles could involve refining this dependency or optimizing intrinsic camera parameters for a more complete calibration. The path paved by this method opens possibilities for diverse applications, particularly in scenarios where access to precise camera parameters is impractical.