Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM

Abstract: Towards open-ended Video Anomaly Detection (VAD), existing methods often exhibit biased detection when faced with challenging or unseen events and lack interpretability. To address these drawbacks, we propose Holmes-VAD, a novel framework that leverages precise temporal supervision and rich multimodal instructions to enable accurate anomaly localization and comprehensive explanations. Firstly, towards unbiased and explainable VAD system, we construct the first large-scale multimodal VAD instruction-tuning benchmark, i.e., VAD-Instruct50k. This dataset is created using a carefully designed semi-automatic labeling paradigm. Efficient single-frame annotations are applied to the collected untrimmed videos, which are then synthesized into high-quality analyses of both abnormal and normal video clips using a robust off-the-shelf video captioner and a LLM. Building upon the VAD-Instruct50k dataset, we develop a customized solution for interpretable video anomaly detection. We train a lightweight temporal sampler to select frames with high anomaly response and fine-tune a multimodal LLM to generate explanatory content. Extensive experimental results validate the generality and interpretability of the proposed Holmes-VAD, establishing it as a novel interpretable technique for real-world video anomaly analysis. To support the community, our benchmark and model will be publicly available at https://holmesvad.github.io.

• The paper introduces Holmes-VAD, a framework that integrates a temporal sampler and multi-modal LLM for unbiased and interpretable anomaly detection.
• The study presents the VAD-Instruct50k dataset—a robust benchmark generated with semi-automated annotations and LLM-driven insights for detailed anomaly analysis.
• Results demonstrate significant improvements in AUC and AP on benchmarks like UCF-Crime, underscoring enhanced accuracy and transparency in anomaly detection.

Holmes-VAD: A Novel Approach to Video Anomaly Detection

Introduction

The paper introduces "Holmes-VAD" (2406.12235), a framework developed to address the current limitations in Video Anomaly Detection (VAD) systems, particularly bias and lack of interpretability. The authors propose a system that leverages precise temporal supervision and multi-modal instructions, enabling not only accurate anomaly localization but also comprehensive anomaly explanations. The core contribution of the work is the creation of a large-scale multimodal VAD instruction-tuning benchmark named VAD-Instruct50k. This dataset is unique in its semi-automatic labeling approach and the incorporation of LLMs to synthesize structured, high-quality anomaly analyses.

Key Contributions

1. Holmes-VAD Framework: The framework employs a novel architecture that combines a temporal sampler, a multi-modal LLM, and video encoders to predict anomaly scores and provide interpretable explanations for detected anomalies (Figure 1).
2. VAD-Instruct50k Dataset: This benchmark dataset consists of single-frame annotations for untrimmed videos, constructed using automated and semi-automated techniques. The dataset introduces a new labeling paradigm that leverages LLM to generate instructive conversation data, enhancing data reliability and richness (Figure 2).
3. Enhanced Anomaly Detection: Holmes-VAD integrates a lightweight temporal sampler to select frames of interest, thus enabling efficient processing of the visual data. This method significantly reduces the bias in anomaly detection, increasing the effectiveness of the model across varying scenarios.
4. Interpretability: Unlike traditional VAD systems, Holmes-VAD provides detailed explanations for anomalies, offering insights into the nature and context of detected aberrations, effectively bridging the gap between machine predictions and human comprehension.

Dataset Construction and Methodology

The VAD-Instruct50k dataset is a cornerstone of this research. It is built by collecting and enhancing existing anomaly datasets with single-frame annotations and fine-grained event clips. The data engine used is a combination of manual efforts and foundation models, ensuring the balance between annotation efficiency and quality.

Figure 2: Data engine for the proposed VAD-Instruct50k.

The framework employs an architecture (Holmes-VAD), where the temporal sampler efficiently selects video frames with high anomaly scores, which are then processed by a multi-modal LLM for generating explanations (Figure 1).

Figure 1: Overview of Holmes-VAD.

Results and Evaluation

Holmes-VAD demonstrates superior performance over existing VAD methods, achieving significant improvements in metrics such as Area Under the Curve (AUC) and Average Precision (AP). On datasets like UCF-Crime and XD-Violence, Holmes-VAD outperforms state-of-the-art models, highlighting its efficacy in both detecting anomalies and providing interpretations for those anomalies.

Qualitative analyses show promising interpretability results, where Holmes-VAD provides coherent and contextually relevant explanations for anomalies, also proven through human evaluation metrics like Judgement Accuracy, Content Perception, and Anomaly Explanatory capacities (Figure 3).

Figure 3: Qualitative results demonstrating interpretability.

Conclusion

Holmes-VAD represents a comprehensive step toward unbiased and interpretable VAD, contributing significantly to the fields of machine learning and computer vision. The integration of the VAD-Instruct50k dataset acts as an enabler for robust and scalable anomaly detection solutions, providing a benchmark for future research in multi-modal video analysis.

Future Work and Limitations

Despite these advancements, the paper acknowledges challenges in further refining anomaly explanations and scaling the model's performance on long-term videos. Future research directions may focus on enhancing data quality and exploring methods to improve LLM's understanding of video context over extended durations.