Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (2401.00901v2)

Abstract: Video grounding aims to localize a spatio-temporal section in a video corresponding to an input text query. This paper addresses a critical limitation in current video grounding methodologies by introducing an Open-Vocabulary Spatio-Temporal Video Grounding task. Unlike prevalent closed-set approaches that struggle with open-vocabulary scenarios due to limited training data and predefined vocabularies, our model leverages pre-trained representations from foundational spatial grounding models. This empowers it to effectively bridge the semantic gap between natural language and diverse visual content, achieving strong performance in closed-set and open-vocabulary settings. Our contributions include a novel spatio-temporal video grounding model, surpassing state-of-the-art results in closed-set evaluations on multiple datasets and demonstrating superior performance in open-vocabulary scenarios. Notably, the proposed model outperforms state-of-the-art methods in closed-set settings on VidSTG (Declarative and Interrogative) and HC-STVG (V1 and V2) datasets. Furthermore, in open-vocabulary evaluations on HC-STVG V1 and YouCook-Interactions, our model surpasses the recent best-performing models by $4.88$ m_vIoU and $1.83\%$ accuracy, demonstrating its efficacy in handling diverse linguistic and visual concepts for improved video understanding. Our codes will be publicly released.

• The paper introduces an open-vocabulary approach that uses a DETR-like architecture with temporal aggregation to overcome limitations of closed-set models.
• It integrates pre-trained spatial grounding from extensive image-text data, enabling effective fusion of visual and linguistic features.
• Experiments demonstrate superior performance over state-of-the-art methods on HC-STVG V1 and YouCook-Interactions benchmarks.

Introduction

Spatio-temporal video grounding plays a critical role in interpreting and linking visual content with descriptive natural language. Traditional models in this domain have primarily operated under a closed-set setting, relying on precise training datasets with a pre-defined and limited vocabulary. However, these models often falter when exposed to new visual and conceptual variations beyond the scope of their training data, a phenomenon frequently observed in real-world applications.

Open-Vocabulary Spatio-Temporal Video Grounding

To tackle the limitations posed by the closed vocabulary in existing spatio-temporal video grounding methods, a novel paradigm has been introduced that embraces the concept of open-vocabulary video grounding. This methodology aims to enable training models on a set of base categories and to facilitate their generalization to unseen objects and actions. By incorporating pre-trained representations from spatial grounding models engineered on extensive image-text datasets, this new approach exhibits remarkable generalization capabilities, successfully performing in scenarios where traditional models would typically underperform.

Model Architecture

The proposed model adopts a DETR-like architecture complemented with temporal aggregation modules further enhancing its capabilities. The spatial modules derive their initialization from a pre-trained foundational image model, ensuring the retention of nuanced representation aids in the model's generalization prowess. The architecture encompasses vision and text encoders, a cross-modality spatio-temporal encoder that fuses spatial, temporal, and modal information, and language-guided query selection to initialize cross-modal queries efficiently. A decoder then processes these queries, predicting bounding boxes and corresponding temporal tubes, leveraging a rich set of features extracted from the vision and text encoders.

Advancements and Contributions

The proposed video grounding model achieves significant advancements by delivering remarkable performance in both closed-set and open-vocabulary settings. The model consistently surpasses state-of-the-art methods across multiple benchmarks. Notably, it performs impressively when evaluated in an open-vocabulary setting on HC-STVG V1 and YouCook-Interactions benchmarks, showcasing improvement over recent best-performing models. These achievements underscore the efficacy of the approach in handling diverse linguistic and visual concepts, leading to an enhanced understanding of videos.

Conclusion

The paper's contribution to the field of video grounding is manifold, including a pioneering evaluation of spatio-temporal video grounding models in an open-vocabulary setting and a novel model that merges spatial grounding strengths with video-specific adaptability. These enhancements allow the model to not only exceed current closed-set standards but also to bravely navigate open-vocabulary challenges, signifying a promising step forward in the ever-evolving landscape of video understanding.