Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation (2404.03645v1)

Published 4 Apr 2024 in cs.CV

Abstract: Referring video segmentation relies on natural language expressions to identify and segment objects, often emphasizing motion clues. Previous works treat a sentence as a whole and directly perform identification at the video-level, mixing up static image-level cues with temporal motion cues. However, image-level features cannot well comprehend motion cues in sentences, and static cues are not crucial for temporal perception. In fact, static cues can sometimes interfere with temporal perception by overshadowing motion cues. In this work, we propose to decouple video-level referring expression understanding into static and motion perception, with a specific emphasis on enhancing temporal comprehension. Firstly, we introduce an expression-decoupling module to make static cues and motion cues perform their distinct role, alleviating the issue of sentence embeddings overlooking motion cues. Secondly, we propose a hierarchical motion perception module to capture temporal information effectively across varying timescales. Furthermore, we employ contrastive learning to distinguish the motions of visually similar objects. These contributions yield state-of-the-art performance across five datasets, including a remarkable $\textbf{9.2%}$ $\mathcal{J&F}$ improvement on the challenging $\textbf{MeViS}$ dataset. Code is available at https://github.com/heshuting555/DsHmp.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. A closer look at referring expressions for video object segmentation. Multimedia Tools and Applications, 2022.
  2. End-to-end referring video object segmentation with multimodal transformers. In CVPR, 2022.
  3. A simple framework for contrastive learning of visual representations. In ICML, 2020.
  4. Multi-attention network for compressed video referring object segmentation. In ACM MM, 2022.
  5. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
  6. PhraseClick: toward achieving flexible interactive segmentation by phrase and click. In ECCV, 2020.
  7. Vision-language transformer and query generation for referring segmentation. In ICCV, 2021a.
  8. MeViS: A large-scale benchmark for video segmentation with motion expressions. In ICCV, 2023a.
  9. MOSE: A new dataset for video object segmentation in complex scenes. In ICCV, 2023b.
  10. VLT: Vision-language transformer and query generation for referring segmentation. IEEE TPAMI, 2023c.
  11. Progressive multimodal interaction network for referring video object segmentation. The 3rd Large-scale Video Object Segmentation Challenge, 2021b.
  12. Language-bridged spatial-temporal interaction for referring video object segmentation. In CVPR, 2022.
  13. Encoder fusion network with co-attention embedding for referring image segmentation. In CVPR, 2021.
  14. Actor and action video segmentation from a sentence. In CVPR, 2018.
  15. Html: Hybrid temporal-scale multimodal learning framework for referring video object segmentation. In ICCV, 2023.
  16. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
  17. Vita: Video instance segmentation via object token association. In NeurIPS, 2022.
  18. Segmentation from natural language expressions. In ECCV, 2016.
  19. Beyond one-to-one: Rethinking the referring image segmentation. In ICCV, 2023.
  20. Minvis: A minimal video instance segmentation framework without video-based training. In NeurIPS, 2022.
  21. Towards understanding action recognition. In ICCV, 2013.
  22. Hard negative mixing for contrastive learning. NeurIPS, 2020.
  23. Video object segmentation with language referring expressions. In ACCV, 2018.
  24. Restr: Convolution-free referring image segmentation using transformers. In CVPR, 2022.
  25. Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
  26. You only infer once: Cross-modal meta-transfer for referring video object segmentation. In AAAI, 2022.
  27. Referring image segmentation via recurrent refinement networks. In CVPR, 2018.
  28. Transformer-based visual segmentation: A survey. arXiv preprint arXiv:2304.09854, 2023a.
  29. Robust referring video object segmentation with cyclic structural consensus. In ICCV, 2023b.
  30. Clawcranenet: Leveraging object-level relation for text-based video segmentation. arXiv preprint arXiv:2103.10702, 2021.
  31. Recurrent multimodal interaction for referring image segmentation. In ICCV, 2017.
  32. Instance-specific feature propagation for referring segmentation. IEEE TMM, 2022a.
  33. GRES: Generalized referring expression segmentation. In CVPR, 2023a.
  34. Multi-modal mutual attention and iterative interaction for referring image segmentation. IEEE TIP, 32, 2023b.
  35. Polyformer: Referring image segmentation as sequential polygon generation. In CVPR, 2023c.
  36. Cross-modal progressive comprehension for referring segmentation. IEEE TPAMI, 2022b.
  37. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  38. Video swin transformer. In CVPR, 2022c.
  39. Decoupled weight decay regularization. In ICLR, 2019.
  40. Soc: Semantic-assisted object cluster for referring video object segmentation. In NeurIPS, 2023.
  41. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
  42. Dynamic multimodal instance segmentation guided by natural language queries. In ECCV, 2018.
  43. Spectrum-guided multi-granularity referring video object segmentation. In ICCV, 2023.
  44. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017.
  45. Learning to segment every referring object point by point. In CVPR, 2023.
  46. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pages 70–80, 2015.
  47. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In ECCV, 2020.
  48. Contrastive grouping with transformer for referring image segmentation. In CVPR, 2023a.
  49. Temporal collection and distribution for referring video object segmentation. In ICCV, 2023b.
  50. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
  51. Attention is all you need. In NeurIPS, 2017.
  52. Asymmetric cross-guided attention network for actor and action video segmentation from natural language query. In ICCV, 2019.
  53. Cris: Clip-driven referring image segmentation. In CVPR, 2022.
  54. Multi-level representation learning with semantic alignment for referring video object segmentation. In CVPR, 2022a.
  55. Onlinerefer: A simple online baseline for referring video object segmentation. In ICCV, 2023a.
  56. Language as queries for referring video object segmentation. In CVPR, 2022b.
  57. Towards open vocabulary learning: A survey. IEEE TPAMI, 2024.
  58. Advancing referring expression segmentation beyond single image. In CVPR, 2023b.
  59. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
  60. Lavt: Language-aware vision transformer for referring image segmentation. In CVPR, 2022.
  61. Cross-modal self-attention network for referring image segmentation. In CVPR, 2019.
  62. Modeling context in referring expressions. In ECCV, 2016.
  63. Mattnet: Modular attention network for referring expression comprehension. In CVPR, 2018.
  64. Zero-shot referring image segmentation with global-local context features. In CVPR, 2023.
  65. Detection and tracking meet drones challenge. IEEE TPAMI, 2022.
  66. Generalized decoding for pixel, image, and language. In CVPR, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Shuting He (23 papers)
  2. Henghui Ding (87 papers)
Citations (17)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now