Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts
(2404.00741)Abstract
The goal of interactive image segmentation is to delineate specific regions within an image via visual or language prompts. Low-latency and high-quality interactive segmentation with diverse prompts remain challenging for existing specialist and generalist models. Specialist models, with their limited prompts and task-specific designs, experience high latency because the image must be recomputed every time the prompt is updated, due to the joint encoding of image and visual prompts. Generalist models, exemplified by the Segment Anything Model (SAM), have recently excelled in prompt diversity and efficiency, lifting image segmentation to the foundation model era. However, for high-quality segmentations, SAM still lags behind state-of-the-art specialist models despite SAM being trained with x100 more segmentation masks. In this work, we delve deep into the architectural differences between the two types of models. We observe that dense representation and fusion of visual prompts are the key design choices contributing to the high segmentation quality of specialist models. In light of this, we reintroduce this dense design into the generalist models, to facilitate the development of generalist models with high segmentation quality. To densely represent diverse visual prompts, we propose to use a dense map to capture five types: clicks, boxes, polygons, scribbles, and masks. Thus, we propose SegNext, a next-generation interactive segmentation approach offering low latency, high quality, and diverse prompt support. Our method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS, both quantitatively and qualitatively.
Overview
-
Introduces SegNext, a novel model for interactive image segmentation with low latency and high-quality results.
-
SegNext integrates dense representation and fusion of visual and language prompts within a generalist framework.
-
Achieves superior segmentation quality on benchmarks like HQSeg-44K and DAVIS datasets, while maintaining efficiency.
-
Identified limitations include resource-intensive dense representation and challenges with text prompts and complex scenes.
Rethinking Interactive Image Segmentation: Introducing SegNext for Low Latency, High-Quality Results with Diverse Prompts
Introduction to Interactive Image Segmentation
Interactive image segmentation aims to delineate specific regions within an image, utilizing visual or language prompts. This task has gained importance with advancements in camera technology and the need for high-resolution image processing. Traditional models fall into two categories: specialist and generalist models. Specialist models are designed for specific tasks but suffer from high latency due to their need to recompute the image with each prompt update. Generalist models, on the other hand, offer prompt diversity and efficiency but lag behind in segmentation quality.
The SegNext Approach
The paper introduces SegNext, a model designed to tackle the limitations of current interactive segmentation methods. By integrating dense representation and fusion of visual prompts, previously limited to specialist models, into a generalist framework, SegNext achieves low latency and high-quality interactive segmentation.
Visual Prompts Representation
Visual prompts, including clicks, boxes, polygons, scribbles, and masks, are encoded using a three-channel dense map, preserving the detailed spatial attributes critical for high-quality segmentation.
Fusion of Visual and Language Prompts
SegNext encodes visual prompts using convolutional layers, with embeddings fused to image embeddings via element-wise addition. This approach allows detailed spatial information to be maintained. For language prompts, SegNext utilizes the CLIP model to encode text into vectors, which are then queried against the image embedding for mask generation.
Training and Implementation Details
The model is trained with clicks as the primary prompt due to their generalizability to other prompt types. Training is conducted on the COCO+LVIS dataset, with fine-tuning on HQSeg-44K. SegNext employs ViT-Base as the image encoder and a lightweight segmentation decoder, ensuring efficient training and inference.
Experimental Evaluation
SegNext has been extensively evaluated on HQSeg-44K and DAVIS datasets, outperforming state-of-the-art methods in terms of segmentation quality while maintaining competitive latency. Additionally, the model shows promising generalizability in out-of-domain evaluations on medical datasets and can seamlessly handle diverse prompt types without specific training.
Limitations and Future Directions
The dense representation approach, while effective, is more resource-intensive compared to sparse representations. Furthermore, the model's handling of text prompts and its performance in capturing thin structures or dealing with cluttered scenes require further research. Future work may explore more powerful backbones or larger datasets to unlock SegNext's full potential.
Conclusion
SegNext represents a significant advance in interactive image segmentation, offering a versatile solution that combines the benefits of both specialist and generalist models. Its ability to efficiently process diverse prompts without sacrificing quality positions it as a promising tool for real-world applications, from enhanced user experiences in image editing to medical image analysis.
Create an account to read this summary for free: