USD: Unknown Sensitive Detector Empowered by Decoupled Objectness and Segment Anything Model (2306.02275v1)

Abstract: Open World Object Detection (OWOD) is a novel and challenging computer vision task that enables object detection with the ability to detect unknown objects. Existing methods typically estimate the object likelihood with an additional objectness branch, but ignore the conflict in learning objectness and classification boundaries, which oppose each other on the semantic manifold and training objective. To address this issue, we propose a simple yet effective learning strategy, namely Decoupled Objectness Learning (DOL), which divides the learning of these two boundaries into suitable decoder layers. Moreover, detecting unknown objects comprehensively requires a large amount of annotations, but labeling all unknown objects is both difficult and expensive. Therefore, we propose to take advantage of the recent Large Vision Model (LVM), specifically the Segment Anything Model (SAM), to enhance the detection of unknown objects. Nevertheless, the output results of SAM contain noise, including backgrounds and fragments, so we introduce an Auxiliary Supervision Framework (ASF) that uses a pseudo-labeling and a soft-weighting strategies to alleviate the negative impact of noise. Extensive experiments on popular benchmarks, including Pascal VOC and MS COCO, demonstrate the effectiveness of our approach. Our proposed Unknown Sensitive Detector (USD) outperforms the recent state-of-the-art methods in terms of Unknown Recall, achieving significant improvements of 14.3\%, 15.5\%, and 8.9\% on the M-OWODB, and 27.1\%, 29.1\%, and 25.1\% on the S-OWODB.

• The paper introduces USD, a framework that decouples objectness and classification to improve detection of both known and unknown objects.
• It employs the Segment Anything Model with an auxiliary supervision framework to filter noisy outputs using pseudo-labeling and soft-weighting techniques.
• Experimental results demonstrate significant improvements in Unknown Recall, outperforming state-of-the-art methods on benchmarks like Pascal VOC and MS COCO.

Overview of "USD: Unknown Sensitive Detector Empowered by Decoupled Objectness and Segment Anything Model"

This paper addresses the challenging task of Open World Object Detection (OWOD), which requires models to continuously learn and detect not only known objects but also previously unseen, unknown objects. The authors propose a novel framework called the Unknown Sensitive Detector (USD), which is empowered by two main strategies: Decoupled Objectness Learning (DOL) and the use of the Segment Anything Model (SAM) to tackle the issue of unknown object annotation.

Key Contributions and Methodology

1. Decoupled Objectness Learning (DOL): The paper identifies a fundamental conflict in existing OWOD methods, which attempt to learn objectness and classification boundaries simultaneously. This conflict arises because objectness and classification tasks require different semantic computations and have opposing objectives. To address this, DOL segregates these tasks into separate decoder layers within the model. Specifically, the initial layer focuses on objectness, aiming to localize all potential objects, while subsequent layers refine the results through category-specific classification and bounding box regression. This decoupling allows for improved detection and learning of both known and unknown objects.
2. Utilization of the Segment Anything Model (SAM): Given the difficulty and expense of annotating unknown objects, the authors leverage SAM, a large visual model capable of class-agnostic segmentation in an open-world context. However, SAM's outputs often include noisy elements such as fragments or background regions. To mitigate this, the paper introduces an Auxiliary Supervision Framework (ASF), which enhances the detection of unknown objects using pseudo-labeling and soft-weighting techniques. These strategies filter noisy outputs from SAM, thereby providing more reliable supervision for unknown object detection.
3. Comprehensive Evaluation: The effectiveness of USD is evaluated rigorously on popular benchmarks, including Pascal VOC and MS COCO datasets, using metrics such as Unknown Recall (U-Recall) for unknown objects and mean Average Precision (mAP) for known objects. Experimental results demonstrate that USD significantly outperforms state-of-the-art methods in terms of U-Recall, achieving improvements by margin points of 14.3\%, 15.5\%, and 8.9\% on the M-OWODB, and 27.1\%, 29.1\%, and 25.1\% on the S-OWODB datasets.

Implications and Future Work

The research presented in this paper has both practical and theoretical implications for the development of OWOD systems. Practically, enhancing the detection of unknown objects can significantly aid applications in autonomous driving and robotic systems where operational environments are highly dynamic and cannot be entirely forecasted. Theoretically, decoupling the learning of objectness and classification boundaries offers a new perspective that could inspire further advancements in multi-task learning frameworks.

Going forward, incorporating advanced techniques for dynamically adjusting to new classes and enhancing the scalability of OWOD systems in broader and more varied environments remains a critical research direction. Additionally, leveraging other varieties of large visual models beyond SAM to address data scarcity can continue to unlock potential advancements in this domain. The balance between computational efficiency and detection accuracy in rapidly evolving scenarios also continues to be a central challenge that demands ongoing attention.