3x2: 3D Object Part Segmentation by 2D Semantic Correspondences

3-by-2: 3D Object Part Segmentation by 2D Semantic Correspondences

In the paper titled "3-by-2: 3D Object Part Segmentation by 2D Semantic Correspondences," the authors introduce a novel, training-free method for 3D object part segmentation called "3-By-2." This method harnesses the power of 2D semantic correspondences derived from feature representations of pretrained foundation models, achieving state-of-the-art (SOTA) performance on various low-shot segmentation benchmarks. This paper addresses the challenges faced in 3D part segmentation, particularly the high cost and scarcity of annotated 3D datasets, by leveraging richly annotated 2D datasets to transfer part labels to 3D objects.

Method Overview

The paper details the 3-By-2 method which consists of three primary steps: 1) rendering multiple 2D views of a 3D object, 2) performing 2D part segmentation on each view using semantic correspondences, and 3) aggregating the 2D predictions into a coherent 3D segmentation using a mask-consistency module. The core innovation lies in utilizing features from image diffusion models and integrating these with a class-agnostic segmentation model like SAM (Segment Anything Model) to achieve precise part label transfer.

Key Contributions

1. Training-Free Methodology: The 3-By-2 method eliminates the need for extensive labeled 3D training data by leveraging 2D annotated datasets, significantly reducing annotation costs and complexities involved in traditional 3D segmentation tasks.
2. Non-Overlapping Mask Generation: The authors propose a method to generate non-overlapping 2D masks, refining the output of SAM to more accurately reflect part boundaries and improve segmentation fidelity.
3. Mask-Level Label Transfer and Consistency: By transferring labels at the mask level and enforcing consistency across multiple views, the method ensures high-quality segmentation across various parts and object categories.

Experimental Analysis

The paper provides a comprehensive evaluation of the 3-By-2 method on multiple datasets including PartNet-Ensembled (PartNetE) and PartNet with level-3 annotations. The results demonstrate that 3-By-2 achieves superior performance on both zero-shot and few-shot settings compared to existing methods.

• Few-Shot Setting: On PartNetE, 3-By-2 achieved an average mIoU of 0.642 across 45 categories, outperforming both fully-supervised and few-shot baseline methods. Specifically, it improved performance by up to 10% on certain categories compared to fully-supervised methods.
• Zero-Shot Setting: Using the PACO dataset for 2D labels, the method showed substantial improvements over other baselines like PartSLIP and SAMPro3D, achieving a notable performance boost on challenging categories with fine-grained annotations.
• PartNet with Level-3 Annotations: The method demonstrated competitiveness with MvDeCor, a model pretrained and finetuned on PartNet data, emphasizing the robustness and flexibility of 3-By-2 in handling highly granular part annotations without additional training.

Theoretical and Practical Implications

The study showcases the effectiveness of leveraging 2D semantic correspondences for 3D segmentation tasks, shedding light on the broader applicability of 2D vision models in 3D contexts. The flexibility of the 3-By-2 method in handling various part taxonomies and finely-grained segmentation tasks highlights a significant advance in the field.

Practically, this approach can be highly beneficial in domains where collecting 3D annotations is prohibitively expensive or logistically challenging, such as robotics, AR/VR, and graphics. The ability to perform accurate 3D part segmentation using abundantly available 2D data opens new avenues for rapid prototyping and deployment in these applications.

Future Directions

Future research may focus on optimizing the feature extraction and mask generation components to further enhance performance. The application of 3-By-2 to dynamic or deformable objects, as well as exploring transfer learning capabilities across even more diverse object categories, could provide additional insights into the robustness and scalability of the method.

Conclusion

The 3-By-2 method represents a significant step forward in 3D object part segmentation by innovatively leveraging 2D annotated datasets. Its training-free nature, combined with the robust label transfer and aggregation mechanisms, positions this method as a highly effective tool for a wide range of 3D vision applications.