VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-domain Generalization (2404.19652v4)

Abstract: Text spotting, a task involving the extraction of textual information from image or video sequences, faces challenges in cross-domain adaption, such as image-to-image and image-to-video generalization. In this paper, we introduce a new method, termed VimTS, which enhances the generalization ability of the model by achieving better synergy among different tasks. Typically, we propose a Prompt Queries Generation Module and a Tasks-aware Adapter to effectively convert the original single-task model into a multi-task model suitable for both image and video scenarios with minimal additional parameters. The Prompt Queries Generation Module facilitates explicit interaction between different tasks, while the Tasks-aware Adapter helps the model dynamically learn suitable features for each task. Additionally, to further enable the model to learn temporal information at a lower cost, we propose a synthetic video text dataset (VTD-368k) by leveraging the Content Deformation Fields (CoDeF) algorithm. Notably, our method outperforms the state-of-the-art method by an average of 2.6% in six cross-domain benchmarks such as TT-to-IC15, CTW1500-to-TT, and TT-to-CTW1500. For video-level cross-domain adaption, our method even surpasses the previous end-to-end video spotting method in ICDAR2015 video and DSText v2 by an average of 5.5% on the MOTA metric, using only image-level data. We further demonstrate that existing Large Multimodal Models exhibit limitations in generating cross-domain scene text spotting, in contrast to our VimTS model which requires significantly fewer parameters and data. The code and datasets will be made available at the https://VimTextSpotter.github.io.

• The paper introduces a unified architecture that integrates text detection, recognition, and tracking for both images and videos.
• It employs a Prompt Queries Generation Module and Task-aware Adapter to dynamically manage word-level and line-level spotting with minimal overhead.
• Empirical results demonstrate a 2.6% H-mean boost on static images and a 5.5% MOTA improvement on video benchmarks, enhancing overall generalization.

Understanding VimTS: Enhancing Cross-Domain Generalization in Text Spotting

Introduction

In the evolving landscape of text spotting technologies, particularly for applications such as automated subtitling, reading road signs, and real-time translation, the challenge of effectively processing text across various domains remains significant. Traditional models often perform well within the domains they are trained on but falter when applied to new, unseen datasets or formats.

A novel approach presented in the text spotting domain is the VimTS (Video and Image Text Spotter), which aims to address these challenges by improving model generalization across different domains, such as transitioning from static images to dynamic video inputs.

Core Contributions of VimTS

The main advancements brought by VimTS can be categorized into the following:

1. Unified Multi-task Architecture: VimTS introduces a sophisticated architecture that integrates detection, recognition, and tracking into a single framework. This unification allows the model to leverage commonalities between these tasks, enhancing performance and efficiency.
2. Prompt Queries Generation Module (PQGM) and Task-aware Adapter: These components are crucial for the model's adaptability, allowing it to dynamically switch between tasks like detecting word-level or line-level text and adapting from static images to videos. The PQGM helps in generating context-specific queries which are essential for the model to focus on the relevant task, while the Task-aware Adapter optimizes feature selection across different tasks with minimal parameter overhead.
3. Synthetic Video Text Dataset (VTD-368k): VimTS incorporates a novel dataset created using a technique called Content Deformation Fields (CoDeF). This dataset is specifically designed to train the model on video data without the extensive costs typically associated with video annotation.

Empirical Performance

VimTS has shown remarkable performance improvements over existing state-of-the-art models. Specifically:

• On static image benchmarks, it improves by an average of 2.6% in H-mean score across six different benchmarks.
• In video-level adaptations, VimTS outperforms prior video text spotters by an average of 5.5% on the MOTA metric.

These results are indicative not only of the model's robustness but also of its generalization capability across diverse text spotting scenarios.

Practical Implications

The improvements VimTS brings are beneficial for a range of real-world applications:

• Automotive and Navigation Systems: Enhanced text spotting can lead to better recognition of road signs and navigation aids in real-time.
• Surveillance and Security: Accurate text spotting in video feeds can be crucial for security and monitoring applications.
• Media and Entertainment: From automated subtitling to more immersive augmented reality experiences, VimTS could significantly enhance media consumption technologies.

Future Directions

While VimTS presents a significant step forward, several areas could be explored further:

• Reduction in Computational Overhead: While the Task-aware Adapter reduces parameter needs, exploring more efficient architectures could further enhance deployment on edge devices.
• Robustness to Environmental Variants: Text spotting in adverse weather conditions or in poorly lit environments remains challenging and could be an area of future enhancement.

Conclusion

VimTS sets a new benchmark for cross-domain text spotting with its innovative architecture and synthetic training dataset. By effectively bridging the gap between static image and video text spotting, and between different text formats, it opens new avenues for research and application in automated text recognition technologies. As with all AI models, continuous refinement and adaptation will be key to maintaining relevance as new challenges and datasets emerge.