General Purpose Image Encoder DINOv2 for Medical Image Registration (2402.15687v1)

Abstract: Existing medical image registration algorithms rely on either dataset specific training or local texture-based features to align images. The former cannot be reliably implemented without large modality-specific training datasets, while the latter lacks global semantics thus could be easily trapped at local minima. In this paper, we present a training-free deformable image registration method, DINO-Reg, leveraging a general purpose image encoder DINOv2 for image feature extraction. The DINOv2 encoder was trained using the ImageNet data containing natural images. We used the pretrained DINOv2 without any finetuning. Our method feeds the DINOv2 encoded features into a discrete optimizer to find the optimal deformable registration field. We conducted a series of experiments to understand the behavior and role of such a general purpose image encoder in the application of image registration. Combined with handcrafted features, our method won the first place in the recent OncoReg Challenge. To our knowledge, this is the first application of general vision foundation models in medical image registration.

• The paper presents DINO-Reg, a novel training-free deformable image registration method leveraging DINOv2’s robust feature extraction.
• The approach integrates DINOv2-encoded features with a discrete optimizer to overcome local minima challenges in traditional methods.
• The paper validates DINO-Reg’s effectiveness in the OncoReg Challenge, demonstrating superior performance by combining global semantics with local texture details.

The paper "General Purpose Image Encoder DINOv2 for Medical Image Registration" tackles the challenge of medical image registration, which is essential for various clinical applications such as disease tracking, surgical planning, and treatment monitoring. Traditional methods in this domain either rely on dataset-specific training or local texture-based features, each having notable limitations. Dataset-specific approaches require substantial, modality-specific training datasets, which are not always feasible to obtain. On the other hand, local texture-based methods often get trapped in local minima due to the lack of global semantic understanding.

The researchers introduce DINO-Reg, a novel training-free deformable image registration method. At the heart of DINO-Reg is the general purpose image encoder DINOv2, which was originally trained on the ImageNet dataset of natural images. The key advantage of this method is that it utilizes this pretrained encoder without requiring any finetuning.

Here's a breakdown of the main contributions and findings from the paper:

1. General Purpose Encoder: DINOv2, a state-of-the-art general purpose image encoder, is leveraged for extracting relevant features from medical images. This encoder was trained on a diverse set of natural images, which enables the method to capture broad visual features.
2. Training-Free Approach: Unlike conventional methods necessitating large, specific datasets, DINO-Reg operates in a training-free manner. This innovation mitigates the extensive need for modality-specific data collection and manual annotations.
3. Discrete Optimization: The method integrates DINOv2 encoded features into a discrete optimizer to derive the optimal deformable registration field. This integration ensures robust registration by leveraging the rich, hierarchical features captured by DINOv2.
4. Experiments and Results: A comprehensive series of experiments were conducted to evaluate the behavior and efficacy of utilizing such a general purpose image encoder in medical image registration. DINO-Reg was assessed in the OncoReg Challenge and emerged as the top performer, showcasing its competitive edge over traditional handcrafted feature-based methods.
5. Global Semantics: By combining the extracted features from DINOv2 with traditional handcrafted features, DINO-Reg benefits from both global semantic insight and local texture details. This dual approach allows the method to avoid local minima traps that purely local methods suffer from, offering more reliable and accurate registrations.

The application of DINOv2 in this context marks a pioneering use of general vision foundation models in the sphere of medical image registration. The researchers demonstrate that general purpose models trained on non-medical datasets can effectively translate to specialized medical applications, providing a new perspective on leveraging existing models for tasks that traditionally required domain-specific solutions. This achievement could pave the way for more widespread adoption of versatile, pretrained encoders in various specialized fields beyond their original training domains.