Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Prompting Diffusion Representations for Cross-Domain Semantic Segmentation (2307.02138v1)

Published 5 Jul 2023 in cs.CV

Abstract: While originally designed for image generation, diffusion models have recently shown to provide excellent pretrained feature representations for semantic segmentation. Intrigued by this result, we set out to explore how well diffusion-pretrained representations generalize to new domains, a crucial ability for any representation. We find that diffusion-pretraining achieves extraordinary domain generalization results for semantic segmentation, outperforming both supervised and self-supervised backbone networks. Motivated by this, we investigate how to utilize the model's unique ability of taking an input prompt, in order to further enhance its cross-domain performance. We introduce a scene prompt and a prompt randomization strategy to help further disentangle the domain-invariant information when training the segmentation head. Moreover, we propose a simple but highly effective approach for test-time domain adaptation, based on learning a scene prompt on the target domain in an unsupervised manner. Extensive experiments conducted on four synthetic-to-real and clear-to-adverse weather benchmarks demonstrate the effectiveness of our approaches. Without resorting to any complex techniques, such as image translation, augmentation, or rare-class sampling, we set a new state-of-the-art on all benchmarks. Our implementation will be publicly available at \url{https://github.com/ETHRuiGong/PTDiffSeg}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
  2. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In CVPR, 2021.
  3. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
  4. Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
  5. Transadapt: A transformative framework for online test time adaptive semantic segmentation. In ICASSP, 2023.
  6. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  7. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  8. Normalization perturbation: A simple domain generalization method for real-world domain shifts. arXiv preprint arXiv:2211.04393, 2022.
  9. Francois Fleuret et al. Uncertainty reduction for model adaptation in semantic segmentation. In CVPR, 2021.
  10. Masked autoencoders are scalable vision learners. In CVPR, 2022.
  11. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
  12. Denoising diffusion probabilistic models. In NeurIPS, 2020.
  13. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
  14. Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In CVPR, 2022.
  15. Fsdr: Frequency space domain randomization for domain generalization. In CVPR, 2021.
  16. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. In NeurIPS, 2021.
  17. C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation. In CVPR, 2023.
  18. Wedge: web-image assisted domain generalization for semantic segmentation. arXiv preprint arXiv:2109.14196, 2021.
  19. Panoptic feature pyramid networks. In CVPR, 2019.
  20. Uncertainty modeling for out-of-distribution generalization. arXiv preprint arXiv:2202.03958, 2022.
  21. Universal style transfer via feature transforms. In NeurIPS, 2017.
  22. Intra-source style augmentation for improved domain generalization. In WACV, 2023.
  23. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
  24. Source-free domain adaptation for semantic segmentation. In CVPR, 2021.
  25. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
  26. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022.
  27. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
  28. The norm must go on: dynamic unsupervised domain adaptation by normalization. In CVPR, 2022.
  29. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963, 2020.
  30. Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV, 2018.
  31. Semantic-aware domain generalized segmentation. In CVPR, 2022.
  32. Global and local texture randomization for synthetic-to-real semantic segmentation. TIP, 2021.
  33. Augco: augmentation consistency-guided self-training for source-free domain adaptive semantic segmentation. arXiv preprint arXiv:2107.10140, 2021.
  34. Learning transferable visual models from natural language supervision. In ICML, 2021.
  35. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  36. Master of all: Simultaneous generalization of urban-scene segmentation to all adverse weather conditions. In ECCV, 2022.
  37. Playing for data: Ground truth from computer games. In ECCV, 2016.
  38. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  39. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.
  40. Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. In ICCV, 2019.
  41. Map-guided curriculum domain adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. TPAMI, 2020.
  42. Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In ICCV, 2021.
  43. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
  44. Augmentation-based domain generalization for semantic segmentation. arXiv preprint arXiv:2304.12122, 2023.
  45. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017.
  46. Dacs: Domain adaptation via cross-domain mixed sampling. In WACV, 2021.
  47. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In CVPR Workshops, 2018.
  48. Learning to adapt structured output space for semantic segmentation. In CVPR, 2018.
  49. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR, 2017.
  50. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.
  51. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726, 2020.
  52. Deep high-resolution representation learning for visual recognition. TPAMI, 2020.
  53. Continual test-time domain adaptation. In CVPR, 2022.
  54. Dannet: A one-stage domain adaptation network for unsupervised nighttime semantic segmentation. In CVPR, 2021.
  55. Siamdoge: Domain generalizable semantic segmentation using siamese network. In ECCV, 2022.
  56. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
  57. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04803, 2023.
  58. Fda: Fourier domain adaptation for semantic segmentation. In CVPR, 2020.
  59. Source data-free unsupervised domain adaptation for semantic segmentation. In ACM MM, 2021.
  60. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In ICCV, 2019.
  61. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.02153, 2023.
  62. Style-hallucinated dual consistency learning for domain generalized semantic segmentation. In ECCV, 2022.
  63. Scene parsing through ade20k dataset. In CVPR, 2017.
  64. Domain generalization with mixstyle. arXiv preprint arXiv:2104.02008, 2021.
  65. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
  66. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV, 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Rui Gong (17 papers)
  2. Martin Danelljan (96 papers)
  3. Han Sun (31 papers)
  4. Julio Delgado Mangas (1 paper)
  5. Luc Van Gool (570 papers)
Citations (17)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. ETHRuiGong/PTDiffSeg · GitHub (1 star)