CELA: Cost-Efficient Language Model Alignment for CTR Prediction

CELA: A Cost-Efficient Solution for CTR Prediction Using Language Models

Introduction

Understanding and predicting user behavior is at the heart of modern recommender systems, and an essential part of this is Click-Through Rate (CTR) prediction. Traditionally, industry has leaned on ID-based models for this task. These models encode user and item features into sparse one-hot vectors and transform them into dense embeddings, which are then fed through sophisticated feature interaction layers to predict CTR. Despite their success, ID-based models struggle with two main issues: dependency on historical data and limited feature scope.

To address these challenges, researchers have proposed integrating Pre-trained Language Models (PLMs) into recommender systems. The paper "CELA: Cost-Efficient Language Model Alignment for CTR Prediction" introduces an innovative approach, blending the robustness of PLMs with the practical strengths of ID-based models.

The CELA Framework

CELA, or Cost-Efficient Language Model Alignment, is designed to merge the advantages of textual feature utilization and ID-based models. It leverages PLMs to mitigate cold-start issues and broaden feature scope, all while maintaining a scalable and efficient system. The framework is a three-phase process:

1. Domain-Adaptive Pre-training (DAP):

   • Adapt the PLM to domain-specific texts from the dataset.
   • Use techniques like Masked Language Model (MLM) loss and SimCSE to ensure the PLM understands the domain and has effective, uniformly distributed embeddings.
   • Train an ID-based model separately to serve as a reference point.

2. Recommendation-Oriented Modal Alignment (ROMA):

   • Align the textual representations of items from the PLM with item-side feature embeddings from the ID-based model using contrastive learning.
   • This alignment is performed at the item level, significantly reducing training overhead.

3. Multi-Modal Feature Fusion (MF²):

   • Integrate aligned text representations with non-textual features in a new ID-based model.
   • This stage ensures collaborative filtering effectiveness while enriching the model with semantic knowledge.

These stages are designed to be iteratively refined, making the system progressively more accurate and efficient.

Key Contributions and Findings

The proposed approach delivers several notable contributions:

• Efficient Integration: Integrates textual features with ID-based models in a model-agnostic way, requiring minimal changes to existing architectures.
• Item-Level Alignment: Reduces training overhead by aligning item text representations at the item level rather than the interaction level, maintaining low latency during inference.
• Empirical Success: Comprehensive experiments on public and industrial datasets, including an A/B test in a real-world app store scenario, demonstrate CELA's effectiveness. It achieved notable improvements in both offline metrics (AUC, Logloss) and real-world metrics (eCPM, Download-Through Rate).

Practical Implications

The practical implications of this research are substantial:

• Scalability: Efficient training and inference make CELA suitable for real-world applications where scalability is critical.
• Cold-Start Problem: By leveraging PLMs, CELA effectively addresses the cold-start problem, making it robust in scenarios with sparse interactions.
• Enhanced Accuracy: The integration of textual features captured by PLMs enriches the model's understanding and prediction capabilities, leading to higher accuracy.

Future Directions

The research opens several intriguing avenues for future work:

1. Exploration of Larger PLMs: Investigate the use of even larger and more sophisticated PLMs, while finding ways to manage the increased complexity and overhead.
2. Cross-Domain Applications: Adapt and apply CELA to different domains and types of recommender systems beyond CTR prediction.
3. Real-Time Adaptation: Develop techniques for real-time learning and adaptation to dynamically changing datasets, ensuring the model remains up-to-date and accurate.

Conclusion

The CELA framework represents a significant step forward in CTR prediction, combining the strengths of traditional ID-based models with the rich semantic capabilities of PLMs. It is efficient, scalable, and effective, making it a promising solution for modern recommender systems.