LLMs for User Interest Exploration in Large-scale Recommendation Systems (2405.16363v2)

Abstract: Traditional recommendation systems are subject to a strong feedback loop by learning from and reinforcing past user-item interactions, which in turn limits the discovery of novel user interests. To address this, we introduce a hybrid hierarchical framework combining LLMs and classic recommendation models for user interest exploration. The framework controls the interfacing between the LLMs and the classic recommendation models through "interest clusters", the granularity of which can be explicitly determined by algorithm designers. It recommends the next novel interests by first representing "interest clusters" using language, and employs a fine-tuned LLM to generate novel interest descriptions that are strictly within these predefined clusters. At the low level, it grounds these generated interests to an item-level policy by restricting classic recommendation models, in this case a transformer-based sequence recommender to return items that fall within the novel clusters generated at the high level. We showcase the efficacy of this approach on an industrial-scale commercial platform serving billions of users. Live experiments show a significant increase in both exploration of novel interests and overall user enjoyment of the platform.

• The paper presents a hybrid hierarchical framework that leverages LLMs to predict novel interest clusters and expand user discovery, overcoming typical feedback loops.
• It deploys a two-level recommendation system where high-level LLM policies generate interest clusters while low-level models ground recommendations to specific items.
• Live experiments on a commercial platform demonstrate significant improvements in user exploration, content engagement, and overall user growth.

Summary of "LLMs for User Interest Exploration in Large-scale Recommendation Systems"

The paper presents an innovative framework for enhancing user interest exploration within large-scale recommendation systems by merging LLMs with established recommendation algorithms. This hybrid hierarchical framework seeks to circumvent the typical feedback loops found in recommendation systems, which prioritize previous user behavior over potentially novel content, thereby limiting the breadth of user discovery. Through the effective integration of LLMs, the framework generates novel interest clusters and confines classic recommendation models to recommend items within these clusters.

Hybrid Hierarchical Framework

High-Level Language Policy

The high-level language policy employs LLMs to examine a user's previous interactions, represented via interest clusters. By using cluster descriptions, LLMs predict new interests, enabling users to explore novel content based on their historical consumption. The LLM controls the granularity of recommendations and ensures predictions remain within predefined interest clusters.

Figure 1: LLM-powered hybrid hierarchical planning diagram for user interest exploration.

Practical deployment involves pre-computed lookup operations, leveraging the clustering and inference strategy to handle vast item corpuses efficiently. This is achieved by reducing the effective recommendation space to cluster pairs, performing offline inference, and maintaining a lightweight online serving mechanism.

Low-Level Item Policy

This component grounds language-generated interests to individual items, utilizing existing recommendation models such as sequential transformer models coupled with restriction policies. These models return items within the generated interest clusters, ensuring recommendations remain relevant and personalized while enhancing user exploration.

Controlled generation is vital, allowing LLMs to produce recommendations that align with cluster definitions effectively. Fine-tuning facilitates a balance between global knowledge and user-specific data, crucial for aligning LLM outputs with actual user behaviors.

Fine-Tuning for User Behavior Alignment

Fine-tuning LLMs with real user interaction data targets controlled generation, ensuring outputs match specific interest clusters and align with user behaviors. This involves diverse data curation to achieve balanced clusters and mitigate biased generation frequencies.

Figure 2: Prompt for Novel Interest Prediction when K=2.

Figure 3: Label (i.e., generated by fine-tuned LLM) Distribution: X-axis represents label frequency; Y-axis represents the percentage of labels within each frequency range.

Control generation and behavioral alignment are assessed through metrics such as match rate and recall, ensuring fine-tuned models generate relevant outputs consistent with user interests, aligning experimental results with observed user behaviors.

Live Experiments

Experiments conducted on a commercial recommendation platform demonstrate significant gains in user exploration and content engagement metrics. The framework effectively expands user interests, facilitating longer platform engagement and increased user growth.

Figure 4: (a) Model Finetuning Process. (b) and (c) Comparison between different recommenders in live experiments.

Figure 5: The proposed method drives user growth.

Novelty and Quality

The proposed method excels in introducing novel content while maintaining recommendation quality, outperforming existing exploration-oriented models. Novelty and user engagement metrics confirm its success in diversifying user consumption patterns.

User Interest Exploration

The framework significantly increases the diversity of user interest consumption (UCI metric), encouraging exploration across various interest levels, which demonstrates its efficacy in broadening user discovery.

User Growth

Real-world user engagement metrics reveal enhanced platform usage, attributed to the novel, high-quality recommendations provided by the proposed framework. Significant increases in active users further affirm its potential in driving growth.

Conclusion

This hybrid hierarchical framework offers a methodical approach to integrate LLMs within large-scale recommendation systems, overcoming traditional limitations by facilitating novel interest exploration. The methodology provides a seamless user experience with implications for broadening content engagement and enhancing user satisfaction. Future improvements will explore long-term planning considerations to further refine hierarchical recommendation processes.