CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentation (2310.02296v2)

Abstract: Generalized Zero-shot Semantic Segmentation aims to segment both seen and unseen categories only under the supervision of the seen ones. To tackle this, existing methods adopt the large-scale Vision LLMs (VLMs) which obtain outstanding zero-shot performance. However, as the VLMs are designed for classification tasks, directly adapting the VLMs may lead to sub-optimal performance. Consequently, we propose CLIP-ZSS (Zero-shot Semantic Segmentation), a simple but effective training framework that enables any image encoder designed for closed-set segmentation applied in zero-shot and open-vocabulary tasks in testing without combining with VLMs or inserting new modules. CLIP-ZSS consists of two key modules: Global Learning Module (GLM) and Pixel Learning Module (PLM). GLM is proposed to probe the knowledge from the CLIP visual encoder by pulling the CLS token and the dense features from the image encoder of the same image and pushing others apart. Moreover, to enhance the ability to discriminate unseen categories, PLM consisting of pseudo labels and weight generation is designed. To generate semantically discriminated pseudo labels, a multi-scale K-Means with mask fusion working on the dense tokens is proposed. In pseudo weight generation, a synthesizer generating pseudo semantic features for the unannotated area is introduced. Experiments on three benchmarks show large performance gains compared with SOTA methods.