How to Benchmark Vision Foundation Models for Semantic Segmentation? (2404.12172v2)

Abstract: Recent vision foundation models (VFMs) have demonstrated proficiency in various tasks but require supervised fine-tuning to perform the task of semantic segmentation effectively. Benchmarking their performance is essential for selecting current models and guiding future model developments for this task. The lack of a standardized benchmark complicates comparisons. Therefore, the primary objective of this paper is to study how VFMs should be benchmarked for semantic segmentation. To do so, various VFMs are fine-tuned under various settings, and the impact of individual settings on the performance ranking and training time is assessed. Based on the results, the recommendation is to fine-tune the ViT-B variants of VFMs with a 16x16 patch size and a linear decoder, as these settings are representative of using a larger model, more advanced decoder and smaller patch size, while reducing training time by more than 13 times. Using multiple datasets for training and evaluation is also recommended, as the performance ranking across datasets and domain shifts varies. Linear probing, a common practice for some VFMs, is not recommended, as it is not representative of end-to-end fine-tuning. The benchmarking setup recommended in this paper enables a performance analysis of VFMs for semantic segmentation. The findings of such an analysis reveal that pretraining with promptable segmentation is not beneficial, whereas masked image modeling (MIM) with abstract representations is crucial, even more important than the type of supervision used. The code for efficiently fine-tuning VFMs for semantic segmentation can be accessed through the project page at: https://tue-mps.github.io/benchmark-vfm-ss/.

• The paper introduces a comprehensive benchmarking setup that evaluates VFMs on semantic segmentation by varying key evaluation settings.
• It demonstrates that end-to-end fine-tuning and advanced decoders like Mask2Former yield superior performance compared to simpler linear setups.
• Results underscore the importance of multi-dataset evaluations and testing domain shifts to accurately assess model adaptability and generalization.

Vision Foundation Models for Semantic Segmentation: Evaluating Benchmarking Strategies

Introduction to the Study

This experimental paper evaluates various benchmarking methods to understand how Vision Foundation Models (VFMs) perform on the task of semantic segmentation. By analyzing settings that impact performance rankings, the research aims to propose an effective and representative benchmarking setup for VFMs. This setup focuses on variations in evaluation settings such as decoder types, model scales, patch sizes, training datasets, and the handling of domain shifts, using a diverse selection of VFMs.

Experiment Design and Methodology

Models were selected based on alignment with the latest techniques and variety in pretraining data sources, objectives, and architectures. All models used the Vision Transformer (ViT) framework with modifications to accommodate semantic segmentation tasks. A controlled set of experiments tested the impact of:

• Encoder freezing vs. end-to-end fine-tuning
• Different decoders: linear probing and Mask2Former
• Model scale variations: ViT-B versus ViT-L
• Patch sizes: $16\times16$ vs $8\times8$
• Variations in training datasets: ADE20K, PASCAL VOC, and Cityscapes
• Domain shift adaptation: training on synthetic vs. real-world images

Impact Analysis of Settings

Freezing the Encoder: Results showed significant performance drops with frozen encoders across models, suggesting that end-to-end fine-tuning is crucial for leveraging pretrained capabilities on new semantic segmentation tasks.

Decoder Variations: The use of the Mask2Former decoder generally improved performance but did not affect the relative rankings of models significantly, indicating that simpler linear decoders could suffice in benchmark settings for efficiency.

Model Scaling: Up-scaling model sizes offered performance gains, but similarly did not alter performance rankings markedly. For benchmark efficiency, smaller models are suggested unless the application necessitates larger sizes.

Patch Size Variations: Results were mixed across models when changing patch sizes. Since smaller patches only marginally improved outcomes, larger default patch sizes are recommended for efficiency.

Training Dataset Variations: Performance rankings varied more distinctively across different datasets. This emphasizes the importance of multi-dataset benchmarking to better understand model adaptability and generalization.

Domain Shifts: Introducing domain shifts, particularly synthetic-to-real transitions, generally led to performance degradation. This underscores the need for including such evaluations in benchmarks, especially for applications expected to operate across varied domains.

Conclusions and Recommendations

The paper has led to the proposal of a benchmarking setup that optimally balances efficiency and representativeness. Recommended setup includes:

• Using ViT-B with a $16\times16$ patch size and linear decoder.
• Involving multiple datasets in evaluations to capture performance across diverse scenarios.
• Implementing end-to-end fine-tuning rather than relying solely on linear probing to ensure models fully adapt to new tasks.

Moreover, the research questions the efficacy of promptable segmentation pretraining, suggesting instead the potential superiority of masked image modeling (MIM) with abstract representations for semantic segmentation.

Future Implications

The findings highlight the value of comprehensive and varied benchmark settings to truly gauge the performance of VFMs in semantic segmentation. Future work might expand upon integration of emerging VFM architectures, explore deeper into the interactions between benchmark settings, or establish more expansive sets of evaluation metrics and datasets to further refine the recommended benchmarks. Such efforts can ensure that benchmarks evolve in step with advancements in model development and training methodologies.

The paper stands as a significant contributor to the ongoing refinement of benchmarks that shape the development and evaluation of VFMs for semantic segmentation, providing a clear roadmap for both present and future research endeavors.