Learning What Not to Segment: A New Perspective on Few-Shot Segmentation (2203.07615v2)

Abstract: Recently few-shot segmentation (FSS) has been extensively developed. Most previous works strive to achieve generalization through the meta-learning framework derived from classification tasks; however, the trained models are biased towards the seen classes instead of being ideally class-agnostic, thus hindering the recognition of new concepts. This paper proposes a fresh and straightforward insight to alleviate the problem. Specifically, we apply an additional branch (base learner) to the conventional FSS model (meta learner) to explicitly identify the targets of base classes, i.e., the regions that do not need to be segmented. Then, the coarse results output by these two learners in parallel are adaptively integrated to yield precise segmentation prediction. Considering the sensitivity of meta learner, we further introduce an adjustment factor to estimate the scene differences between the input image pairs for facilitating the model ensemble forecasting. The substantial performance gains on PASCAL-5i and COCO-20i verify the effectiveness, and surprisingly, our versatile scheme sets a new state-of-the-art even with two plain learners. Moreover, in light of the unique nature of the proposed approach, we also extend it to a more realistic but challenging setting, i.e., generalized FSS, where the pixels of both base and novel classes are required to be determined. The source code is available at github.com/chunbolang/BAM.

• The paper introduces a two-learner model that explicitly excludes irrelevant base-class regions to reduce overfitting in few-shot segmentation.
• It employs an ensemble scheme with scene difference estimation using the Gram matrix to adjust meta learner predictions dynamically.
• The proposed method achieves a 4.71% mIoU improvement on PASCAL-5i and extends its approach to generalized few-shot segmentation scenarios.

A New Perspective on Few-Shot Segmentation

The paper "Learning What Not to Segment: A New Perspective on Few-Shot Segmentation" introduces a novel approach to alleviating the inherent bias in Few-Shot Segmentation (FSS) models, particularly those based on meta-learning frameworks. Traditionally, these models suffer from a tendency to overfit to the seen classes, thereby diminishing their ability to generalize to novel categories. This paper proposes an innovative two-learner model consisting of both a base learner and a meta learner, designed to explicitly identify and disregard regions of base classes that do not require segmentation.

Key Contributions

The main contributions of this work can be outlined as follows:

1. Two-Learner Model Architecture: The paper introduces an additional branch, referred to as the base learner, alongside the conventional meta learner. This branch is responsible for explicitly predicting which regions in the query images belong to base classes, thereby simplifying the segmentation task for the meta learner.
2. Ensemble Scheme and Scene Differences Estimation: The authors propose a method that assesses the scene differences between image pairs using the Gram matrix of low-level features. This evaluation aids in the model ensemble process, by adjusting predictions from the meta learner based on these differences.
3. Generalized Few-Shot Segmentation (GFSS): The framework is extended to a more comprehensive setting termed GFSS, where the model must determine pixels for both base and novel classes simultaneously. This extension does not merely scaffold the existing framework but adapts it creatively to handle more challenging scenarios.

Experimental Results

The approach sets a new benchmark in mIoU scores on standard datasets such as PASCAL-5^i and COCO-20^i, illustrating substantial improvements in performance even when employing plain learners. The model achieves a 4.71% improvement in mIoU over the previous state-of-the-art on PASCAL-5^i with a VGG16 backbone. Noteworthy gains are also recorded under the 5-shot setting.

Practical and Theoretical Implications

This paper offers a practical methodology to address the critical challenge of bias towards seen classes in FSS, which is instrumental in advancing the application spectrum of FSS models in real-world tasks, such as medical imaging, where novel class segmentation is paramount. Theoretically, the introduction of a base learner to segregate base class regions signifies a shift from traditional feature extraction methods to more specialized, class-discriminative frameworks.

Speculation on Future Developments

The promising results of applying a two-learner setup in FSS suggest potential applications in other few-shot learning paradigms, such as few-shot classification and detection. Furthermore, the concept of using scene difference estimation could be explored in unsupervised domain adaptation to enhance model generalization. There is also the possibility of developing more generalized frameworks that integrate multi-modal data, leveraging the differences in data modalities to further improve the robustness and accuracy of segmentation tasks.

In conclusion, the paper presents a strategically advantageous approach to the Few-Shot Segmentation problem, with significant implications for future research in AI segmentation models. The proposed methodological innovations are not only theoretically intriguing but also dramatically enhance performance, underscoring the potential of adaptive ensemble methodologies in machine learning.