Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition
(2404.10904)Abstract
Human communication is multi-modal; e.g., face-to-face interaction involves auditory signals (speech) and visual signals (face movements and hand gestures). Hence, it is essential to exploit multiple modalities when designing machine learning-based facial expression recognition systems. In addition, given the ever-growing quantities of video data that capture human facial expressions, such systems should utilize raw unlabeled videos without requiring expensive annotations. Therefore, in this work, we employ a multitask multi-modal self-supervised learning method for facial expression recognition from in-the-wild video data. Our model combines three self-supervised objective functions: First, a multi-modal contrastive loss, that pulls diverse data modalities of the same video together in the representation space. Second, a multi-modal clustering loss that preserves the semantic structure of input data in the representation space. Finally, a multi-modal data reconstruction loss. We conduct a comprehensive study on this multimodal multi-task self-supervised learning method on three facial expression recognition benchmarks. To that end, we examine the performance of learning through different combinations of self-supervised tasks on the facial expression recognition downstream task. Our model ConCluGen outperforms several multi-modal self-supervised and fully supervised baselines on the CMU-MOSEI dataset. Our results generally show that multi-modal self-supervision tasks offer large performance gains for challenging tasks such as facial expression recognition, while also reducing the amount of manual annotations required. We release our pre-trained models as well as source code publicly
Overview
-
The paper introduces ConCluGen, a multi-task, multi-modal self-supervised learning framework for Facial Expression Recognition (FER) using video, audio, and text without manual annotations.
-
ConCluGen employs several learning objectives: multi-modal contrastive loss, multi-modal clustering loss, and reconstruction loss, using individual encoders for different data modalities to enhance feature extraction and representation.
-
The model was tested on multiple FER-specific datasets and demonstrated superior performance against current self-supervised and fully supervised methods, excelling particularly on the MOSEI dataset.
-
The research suggests potential extensions of the ConCluGen framework to other modalities and tasks, highlighting its versatility and effectiveness in interpreting human emotions in machines.
ConCluGen: Advancing Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition
Introduction
Facial Expression Recognition (FER) serves as a cornerstone in enhancing human-computer interaction by mirroring human-like understanding in systems. Despite significant advancements through deep learning techniques, the challenge intensifies when models are required to interpret expressions 'in the wild', with data that is not only massive but unlabeled. To address these challenges, the paper presented introduces ConCluGen, a model leveraging a multi-task, multi-modal self-supervised learning framework. This method uniquely combines multi-modal contrastive loss, multi-modal clustering loss, and reconstruction loss to learn from video, audio, and textual data without manual annotations.
Methodology
The ConCluGen framework employs separate encoders for video, text, and audio modalities to project input data into a shared latent space, facilitating the fusion of modal information. The methodology can be broken down as follows:
- Feature Extraction and Representation: Initial features are extracted using state-of-the-art models (2D and 3D ResNet for video, DAVENet for audio, and DistilBERT for text) and are then processed to a uniform temporal resolution.
- Multi-Task Learning Objectives:
- Multi-Modal Contrastive Loss: Minimizes distance between different modal representations of the same instance while maximizing the distance between representations of different instances.
- Multi-Modal Clustering Loss: Clusters embeddings from the same instance across modalities, enhancing intra-class compactness and inter-class separability.
- Reconstruction Loss: Aims to reconstruct the original input from its embedded representation, serving as a regularizing effect and helping the model capture a generalized feature set.
Experiments and Analysis
The ConCluGen model was evaluated against several benchmarks on three FER-specific datasets, emphasizing its effectiveness through superior performance metrics compared to existing multi-modal, self-supervised, and fully supervised methods. Highlights include:
- Datasets Used: Large-scale datasets like VoxCeleb2 for pretraining and CMU-MOSEI, CAER, and MELD for fine-tuning and testing.
- Performance Metrics: Weighted Accuracy, F1 Score, Precision, and Recall were considered, accounting for class imbalances present in the real-world datasets.
- Comparative Analysis: ConCluGen not only outperformed other self-supervised models but also showed competitive or superior results to fully supervised methods, particularly on the MOSEI dataset.
Implications and Future Directions
The integration of multi-task learning with multi-modal self-supervision as proposed in ConCluGen presents a significant step forward in utilizing unlabeled data effectively for complex tasks like FER. The model's ability to leverage inherent multi-modal data correlations without requiring explicit annotation is particularly valuable in scenarios where acquiring labeled data is costly or impractical.
The paper suggests several avenues for future research:
- Expansion to Additional Modalities: Incorporating other data types such as facial landmarks could potentially enhance the model's understanding and interpretation of expressions.
- Application to Other Tasks: Exploring the effectiveness of the ConCluGen framework on related tasks like action unit detection and sentiment analysis could broaden the model's utility.
Conclusion
This work successfully demonstrates the potential of multi-task multi-modal self-supervised learning in handling the complexities of facial expression recognition in uncontrolled environments. With ConCluGen, the research paves the way for more sophisticated, accurate, and practical FER systems, fostering advancements in how machines understand and interact with human emotions.
The full implementation of this model, along with the pre-trained weights, is made openly accessible for ongoing research and development, encouraging further exploration and adaptation of the proposed methods within the scientific community.
Create an account to read this summary for free: