Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation (2404.04231v1)

Abstract: This paper addresses text-supervised semantic segmentation, aiming to learn a model capable of segmenting arbitrary visual concepts within images by using only image-text pairs without dense annotations. Existing methods have demonstrated that contrastive learning on image-text pairs effectively aligns visual segments with the meanings of texts. We notice that there is a discrepancy between text alignment and semantic segmentation: A text often consists of multiple semantic concepts, whereas semantic segmentation strives to create semantically homogeneous segments. To address this issue, we propose a novel framework, Image-Text Co-Decomposition (CoDe), where the paired image and text are jointly decomposed into a set of image regions and a set of word segments, respectively, and contrastive learning is developed to enforce region-word alignment. To work with a vision-LLM, we present a prompt learning mechanism that derives an extra representation to highlight an image segment or a word segment of interest, with which more effective features can be extracted from that segment. Comprehensive experimental results demonstrate that our method performs favorably against existing text-supervised semantic segmentation methods on six benchmark datasets.

• The paper proposes the CoDe framework, aligning image regions with word segments to bridge the gap between training and testing in semantic segmentation.
• It employs contrastive learning and learnable prompts to effectively map decomposed image and text components, enhancing feature extraction.
• Experimental results on six benchmarks show superior performance, particularly in zero-shot scenarios, validating the framework's innovative approach.

Towards Accurate Text-Supervised Semantic Segmentation via Image-Text Co-Decomposition

Introduction

Semantic segmentation has been a critical task enabling various applications in computer vision, such as autonomous driving, robotic navigation, and medical image analysis. Traditionally, fully supervised methods have reigned supreme, primarily hindered by the high costs associated with obtaining pixel-level annotations. In response, text-supervised semantic segmentation has emerged as an appealing avenue, leveraging the abundant image-text pairs available on the internet to overcome the annotation bottleneck. This work introduces a novel framework, CoDe (Co-Decomposition), that fundamentally addresses the limitations inherent in existing text-supervised methods by facilitating an alignment between image regions and word segments to significantly enhance semantic segmentation performance.

Methodology

Image-Text Co-Decomposition (CoDe) Framework

CoDe establishes a direct correspondence between image and text by decomposing them into semantically relevant regions and word segments, respectively. This decomposition is achieved by employing a visual-LLM tailored for segmenting an image into regions and segregating a text into word segments based on selected keywords (nouns). This direct mapping between decomposed image regions and corresponding word segments drastically reduces the discrepancy between training and testing phases, peculiar to prior methods.

• Image and Text Segmenters: Functioning as the core components, these segmenters are designed for deducing semantically coherent regions from images and word segments from text, respectively, facilitated by the selection of nouns from the text.
• Region-Word Alignment Module: By leveraging contrastive learning, a consensus between the image and word segments is enforced, strategically reducing the gap prevalent in earlier methods between the training objective and the actual task of semantic segmentation.

Highlighted Region and Word Prompts

Given the challenge of applying vision-LLMs to masked inputs—stemming from the practice of highlighting certain segments by masking others—CoDe introduces learnable prompts that serve to maintain effective feature extraction capabilities. These prompts essentially provide a consistent input structure to the vision-LLM, thereby enhancing the alignment between the highlighted regions and words.

Experimental Validation

An extensive evaluation of the CoDe framework was carried out on six benchmark datasets, demonstrating its superior performance over existing text-supervised semantic segmentation methods. Notably, CoDe achieved favorable performance against state-of-the-art methods in zero-shot semantic segmentation, further supported by comprehensive ablation studies highlighting the effectiveness of its individual components and the overall approach.

Implications and Future Directions

The CoDe framework marks a significant step forward in text-supervised semantic segmentation, resolving critical discrepancies that have historically limited the efficacy of such methods. By establishing a robust mechanism for aligning image regions with word segments, CoDe not only enhances the segmentation accuracy but also opens up new possibilities for zero-shot and few-shot learning in the context of semantic segmentation. Future work could explore the integration of CoDe's principles with other tasks in vision-language processing, potentially leading to more generalized models capable of understanding and processing complex visual-textual relationships across a broader spectrum of applications.

Conclusion

The introduction of the CoDe framework represents a pivotal advancement in the field of text-supervised semantic segmentation, addressing longstanding challenges through innovative methodology. By fostering a more accurate and direct alignment between image regions and textual descriptors, CoDe sets a new standard for the development of efficient and robust semantic segmentation models, significantly broadening the horizons for research and applications in computer vision and beyond.