Detector-Free Structure from Motion (2306.15669v1)

Abstract: We propose a new structure-from-motion framework to recover accurate camera poses and point clouds from unordered images. Traditional SfM systems typically rely on the successful detection of repeatable keypoints across multiple views as the first step, which is difficult for texture-poor scenes, and poor keypoint detection may break down the whole SfM system. We propose a new detector-free SfM framework to draw benefits from the recent success of detector-free matchers to avoid the early determination of keypoints, while solving the multi-view inconsistency issue of detector-free matchers. Specifically, our framework first reconstructs a coarse SfM model from quantized detector-free matches. Then, it refines the model by a novel iterative refinement pipeline, which iterates between an attention-based multi-view matching module to refine feature tracks and a geometry refinement module to improve the reconstruction accuracy. Experiments demonstrate that the proposed framework outperforms existing detector-based SfM systems on common benchmark datasets. We also collect a texture-poor SfM dataset to demonstrate the capability of our framework to reconstruct texture-poor scenes. Based on this framework, we take $\textit{first place}$ in Image Matching Challenge 2023.

• The paper presents a detector-free SfM approach that bypasses traditional keypoint detection challenges in texture-poor conditions.
• It employs a two-stage process starting with a coarse reconstruction that is iteratively refined with attention-based multi-view matching and geometry updates.
• The method outperforms conventional SfM systems on benchmark datasets, winning first place in the Image Matching Challenge 2023.

The paper "Detector-Free Structure from Motion" presents an innovative approach to structure-from-motion (SfM) by bypassing the traditional reliance on keypoint detection, which is often problematic in texture-poor scenes. Traditional SfM systems depend heavily on detecting repeatable keypoints across multiple images to establish correspondences for 3D reconstruction. This reliance can lead to system breakdowns when dealing with scenes lacking sufficient texture.

To address these challenges, the authors propose a detector-free SfM framework that leverages recent advancements in detector-free matchers. Their approach avoids the early commitment to specific keypoints, thereby mitigating the risk of poor keypoint detection affecting the overall SfM process. The proposed framework follows a two-stage process:

1. Coarse Reconstruction: The system initially constructs a coarse SfM model using quantized matches from detector-free techniques. This stage lays the groundwork for the refinement process.
2. Iterative Refinement Pipeline: The initial coarse model is refined through an iterative process that consists of two main modules:
   • Attention-based Multi-view Matching Module: This module focuses on refining feature tracks across multiple views using an attention mechanism.
   • Geometry Refinement Module: This module enhances the accuracy of the reconstruction by refining the geometric structure of the model.

The iterative nature of this pipeline allows the system to progressively improve the accuracy of the reconstructed model.

The authors validate their framework through a series of experiments, demonstrating its superiority over traditional detector-based SfM systems on standard benchmark datasets. Additionally, they introduce a new dataset specifically designed for texture-poor scenes to showcase the robust performance of their method in challenging conditions.

One of the notable achievements of the framework is achieving the first place in the Image Matching Challenge 2023, highlighting its effectiveness and robustness.

In summary, this paper provides a significant advancement in SfM techniques by introducing a detector-free methodology that not only addresses the limitations of traditional keypoint detection but also offers a robust solution for texture-poor environments.