A Unified Search and Recommendation Framework Based on Multi-Scenario Learning for Ranking in E-commerce

A Unified Search and Recommendation Framework for E-commerce

Introduction

Let's dive into a paper proposing a unified approach to Search and Recommendation (S&R) in e-commerce, put forward by a team of researchers at JD.com. Online platforms heavily rely on search and recommendation systems to enhance user experiences by tailoring content to individual preferences. This paper acknowledges the mutual benefits of these two scenarios and presents a novel framework that aims to optimally integrate them.

Traditional vs. Proposed Approach

The conventional multi-scenario learning models have been relying on either shared parameter methodologies or task-specific parameters to address different scenarios independently. While these methods offer a certain level of effectiveness, they fall short in two significant ways:

1. They miss the finer details of each scenario.
2. They do not leverage the full potential of the global label space.

To address these issues, the team at JD.com came up with a framework called Unified Search and Recommendation (USR), designed to enhance both search and recommendation functionalities through a more integrated and granular approach.

Key Components of the USR Framework

Here’s a breakdown of the main components of the USR framework:

1. S&R Views User Interest Extractor Layer (IE)

This layer isolates and refines user interests specific to search and recommendation scenarios, rather than generalizing user behaviors across scenarios. By applying dynamic weight techniques and gating mechanisms, the framework can better differentiate between search and recommendation interests. Additionally, contrastive learning is utilized to highlight these differences further.

2. S&R Views Feature Generator Layer (FG)

This layer applies scenario-specific scaling to features that are generally across both S&R scenarios. In simpler terms, it adjusts the importance of each feature based on the specific scenario it is being applied to— either search or recommendation—thereby generating more relevant insights.

3. Global Label Space Multi-Task Layer (GLMT)

Rather than treating the scenarios in isolation, this layer jointly models the primary task (click-through rate or purchase rate) along with auxiliary tasks using conditional probability, ensuring the model benefits from the entire label space across S&R scenarios.

Performance Observations

Strong Numerical Results

The paper reports extensive experimental evaluations on real-world e-commerce datasets, especially from JD.com's 7Fresh App. The results are quite impressive:

Significant improvements were noted across multiple Multi-Scenario Learning (MSL) methods:

• For instance, when the MMoE model was augmented with USR, the AUC (Area Under the Curve) for CTR (Click-Through Rate) improved from 0.8250 to 0.8270.
• Similar gains were noted in CTCVR (Click-Through Conversion Rate) tasks, demonstrating the framework’s robustness across different metrics.

The improvements can be summarized as follows:

• A boost in the accuracy of predicting clicks and purchases.
• Enhanced representation of user interests and scenario-agnostic features.
• Effective use of global labels to improve auxiliary tasks.

Results and Discussion

The method demonstrated clear benefits when applied to various MSL methods such as SharedBottom, MMoE, PLE, and more, consistently showing improvements in both CTR and CTCVR across both scenarios of search and recommendation. Beyond offline evaluations, online A/B testing in the 7Fresh App showed substantial performance gains in user conversion rates and click-through rates, underscoring the practical utility of the proposed framework.

Implications and Future Perspectives

Practical Implications

• Enhanced User Experience: By fine-tuning recommendations and search results, users are more likely to find relevant products quickly, increasing overall satisfaction and engagement.
• Operational Efficiency: With a unified model, e-commerce platforms can reduce redundancy and complexity, potentially leading to cost savings and more efficient use of computational resources.

Theoretical Implications

• Improved Generalization: The use of scenario-specific layers and global label spaces suggests a pathway toward generalizing multi-task learning models to other paired scenarios beyond just search and recommendation.
• Contrastive Learning Applications: The innovative use of contrastive learning to differentiate between scenarios provides a rich avenue for further research.

Future Developments

Looking ahead, the incorporation of more advanced techniques, such as reinforcement learning or deeper neural networks, could push the boundaries of what’s possible with unified S&R systems. Also, expanding the framework to handle more complex and varied datasets from different e-commerce contexts could further validate and refine the model's effectiveness.

In summary, the USR framework offers a robust, detailed, and integrated approach to simultaneously enhancing search and recommendation systems on e-commerce platforms. By addressing the fine nuances between these scenarios, it opens up new avenues for both theoretical research and practical applications.