Large Multi-modal Encoders for Recommendation (2310.20343v2)

Abstract: In recent years, the rapid growth of online multimedia services, such as e-commerce platforms, has necessitated the development of personalised recommendation approaches that can encode diverse content about each item. Indeed, modern multi-modal recommender systems exploit diverse features obtained from raw images and item descriptions to enhance the recommendation performance. However, the existing multi-modal recommenders primarily depend on the features extracted individually from different media through pre-trained modality-specific encoders, and exhibit only shallow alignments between different modalities - limiting these systems' ability to capture the underlying relationships between the modalities. In this paper, we investigate the usage of large multi-modal encoders within the specific context of recommender systems, as these have previously demonstrated state-of-the-art effectiveness when ranking items across various domains. Specifically, we tailor two state-of-the-art multi-modal encoders (CLIP and VLMo) for recommendation tasks using a range of strategies, including the exploration of pre-trained and fine-tuned encoders, as well as the assessment of the end-to-end training of these encoders. We demonstrate that pre-trained large multi-modal encoders can generate more aligned and effective user/item representations compared to existing modality-specific encoders across three multi-modal recommendation datasets. Furthermore, we show that fine-tuning these large multi-modal encoders with recommendation datasets leads to an enhanced recommendation performance. In terms of different training paradigms, our experiments highlight the essential role of the end-to-end training of large multi-modal encoders in multi-modal recommendation systems.

• The paper demonstrates that integrating large multi-modal encoders such as CLIP and VLMo with recommender systems significantly enhances item representations by aligning visual and textual modalities.
• It employs various training paradigms—including pre-training, fine-tuning, and end-to-end training—to evaluate performance improvements on Amazon datasets using metrics like Recall@K and NDCG@K.
• The results indicate that dual-stream architectures, particularly through end-to-end training, outperform traditional modality-specific methods by reducing conflicts in feature extraction and alignment.

Large Multi-modal Encoders for Recommendation

Introduction

The paper "Large Multi-modal Encoders for Recommendation" (2310.20343) explores the integration of large multi-modal (LMM) encoders, specifically CLIP and VLMo, into recommender systems to enhance the representation of items through a deeper alignment of multiple modalities. Traditional recommendation systems rely primarily on user-item interactions, failing to exploit the complex interdependencies among visual and textual data. The paper proposes employing LMM encoders to better align and integrate these diverse data sources.

Architecture and Methodology

The paper investigates two state-of-the-art encoders: CLIP, a dual-stream architecture designed for image-text retrieval tasks, and VLMo, a unified encoder handling multi-modal data with a mixture-of-modality-experts (MoME) transformer. CLIP independently processes images and text using Vision Transformers (ViT) and LLMs like GPT-2, aligning them through a contrastive loss. In contrast, VLMo combines both image and text data within a joint network, coordinating using modality-specific experts in its architecture.

Figure 1: An illustration of different feature extraction methods.

The integration of these encoders into existing recommendation models is examined under multiple paradigms: using pre-trained encoders, fine-tuning them on specific datasets, and incorporating them in an end-to-end training framework with respective recommendation models. The implementation assesses capabilities across variably demanding computational paradigms to suit different application requirements.

Experiments and Results

Experiments are conducted on three Amazon review datasets: Sports, Clothing, and Baby categories, leveraging visual elements and textual product descriptions. The paper performs a comprehensive evaluation across multiple recommendation tasks to determine efficacy improvements facilitated by LMM encoder integration. The evaluation metrics, including Recall@K and NDCG@K, indicate significant performance enhancements using LMM encoders versus traditional modality-specific encoders.

The findings reveal distinct advantages of LMM encoder strategies:

• Pre-trained and Fine-tuned Encoders: Fine-tuning LMM encoders typically yielded enhanced recommendation accuracy, though some encoder-model combinations like LATTICE experienced performance conflicts owing to overlapping feature extraction and alignment objectives.
• End-to-end Training: This paradigm generally proved superior, markedly with dual-stream LMMs (CLIP), underscoring its potential to refine existing multi-modal networks by aligning representations more robustly via recommendation losses.

Modality Contribution Analysis

The analysis of the contribution of each modality reveals that integrating both visual and textual modalities consistently performs better than utilizing individual ones. This reinforces the argument for incorporating diverse data types in recommender systems, and the alignment achieved by LMM encoders significantly surpasses traditional separate encoding methods.

Conclusions

The utilization of LMM encoders, particularly through fine-tuning and end-to-end training, demonstrates significant potential in enhancing multi-modal recommender system performance. By better aligning modalities such as text and images through advanced encoding methods like CLIP and VLMo, these systems can achieve richer, more contextually relevant user and item representations. The paper's findings set a precedent for leveraging large-scale pre-trained models in recommendation tasks, suggesting directions for future research, including optimizing model architectures to eliminate conflicts in inherent learning objectives.