nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance (2309.16967v3)

Abstract: Automatic segmentation of medical images is crucial in modern clinical workflows. The Segment Anything Model (SAM) has emerged as a versatile tool for image segmentation without specific domain training, but it requires human prompts and may have limitations in specific domains. Traditional models like nnUNet perform automatic segmentation during inference and are effective in specific domains but need extensive domain-specific training. To combine the strengths of foundational and domain-specific models, we propose nnSAM, integrating SAM's robust feature extraction with nnUNet's automatic configuration to enhance segmentation accuracy on small datasets. Our nnSAM model optimizes two main approaches: leveraging SAM's feature extraction and nnUNet's domain-specific adaptation, and incorporating a boundary shape supervision loss function based on level set functions and curvature calculations to learn anatomical shape priors from limited data. We evaluated nnSAM on four segmentation tasks: brain white matter, liver, lung, and heart segmentation. Our method outperformed others, achieving the highest DICE score of 82.77% and the lowest ASD of 1.14 mm in brain white matter segmentation with 20 training samples, compared to nnUNet's DICE score of 79.25% and ASD of 1.36 mm. A sample size study highlighted nnSAM's advantage with fewer training samples. Our results demonstrate significant improvements in segmentation performance with nnSAM, showcasing its potential for small-sample learning in medical image segmentation.

• The paper introduces nnSAM, a hybrid model that fuses SAM's feature extraction with nnU-Net's adaptive workflow to improve segmentation under limited training data.
• It demonstrates significant performance gains in few-shot learning scenarios, addressing a critical challenge in medical image analysis.
• The study highlights the potential for hybrid architectures to become a new standard in medical imaging and to extend applications beyond segmentation.

nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance

The paper "nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance" presents an innovative approach to the domain of medical image segmentation by merging the capabilities of the Segment Anything Model (SAM) and the nnU-Net framework. The paper primarily targets the enhancement of segmentation performance in scenarios characterized by limited training data, a common challenge in medical imaging.

Background and Rationale

Medical image segmentation is a critical process in clinical workflows, aiding in disease diagnosis and treatment planning. Traditional segmentation methodologies demand significant expertise and manual effort, but the advent of deep learning-based models has notably streamlined this process. The U-Net architecture, and its derivatives like TransUNet and UNet++, have been at the forefront, leveraging combinatory architectures to balance precision and contextual awareness in segmentation tasks.

nnU-Net, particularly, offers a flexible framework by eliminating the need for novel architectures through its automated configuration process, which includes comprehensive preprocessing and postprocessing stages. It ensures optimal performance across diverse medical segmentation tasks without the necessity for custom network design.

Introduction of nnSAM

nnSAM emerges as a fusion model that integrates the image encoding capabilities of SAM with the adaptive architecture of nnU-Net, addressing the challenge of performing robust medical image segmentation with few-shot learning. The paper argues that this combination enables the generation of a versatile latent space representation, enhancing segmentation accuracy and efficiency even with sparse training data.

Method and Evaluation

The authors provide a comprehensive evaluation of nnSAM under various conditions, specifically focusing on its performance with different sample sizes. The results indicate that nnSAM consistently outperforms in few-shot learning scenarios, hinting at its potential to become a new standard in medical image segmentation. The integration of SAM's feature extraction capabilities and nnU-Net's dynamic configuration mechanisms allows for a model that can generalize more effectively across datasets with varying characteristics.

Implications and Future Directions

The introduction of nnSAM has significant implications for both the practical and theoretical aspects of medical imaging. Practically, nnSAM presents a viable tool for institutions facing data scarcity, ensuring quality segmentation without the need for large annotated datasets. Theoretically, this work reinforces the potential of hybridized model architectures, which blend the strengths of diverse frameworks to tackle specific challenges.

Future research could explore the application of nnSAM beyond medical image segmentation, perhaps extending to fields that also suffer from a lack of large, labeled datasets. Additionally, investigations could deepen into refining the model's adaptability and efficiency, potentially integrating it with other emerging deep learning architectures or foundation models.

In summary, nnSAM represents a methodical advance in medical imaging technology, offering improved performance through a thoughtful integration of existing models. This paper will likely serve as a reference point for future studies endeavoring to enhance segmentation capabilities under constrained conditions.