Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialogue (2310.14626v2)

Abstract: E-commerce pre-sales dialogue aims to understand and elicit user needs and preferences for the items they are seeking so as to provide appropriate recommendations. Conversational recommender systems (CRSs) learn user representation and provide accurate recommendations based on dialogue context, but rely on external knowledge. LLMs generate responses that mimic pre-sales dialogues after fine-tuning, but lack domain-specific knowledge for accurate recommendations. Intuitively, the strengths of LLM and CRS in E-commerce pre-sales dialogues are complementary, yet no previous work has explored this. This paper investigates the effectiveness of combining LLM and CRS in E-commerce pre-sales dialogues, proposing two collaboration methods: CRS assisting LLM and LLM assisting CRS. We conduct extensive experiments on a real-world dataset of Ecommerce pre-sales dialogues. We analyze the impact of two collaborative approaches with two CRSs and two LLMs on four tasks of Ecommerce pre-sales dialogue. We find that collaborations between CRS and LLM can be very effective in some cases.

• The paper introduces two integration approaches—LLM assisting CRS and CRS assisting LLM—that synergize to boost recommendation accuracy in pre-sales scenarios.
• The evaluation on the U-NEED dataset using metrics like precision, recall, f1-score, and hit rates demonstrates improved dialogue understanding and recommendation quality.
• It highlights future research directions, including user sentiment analysis and scaling with larger LLMs to achieve more personalized and dynamic e-commerce interactions.

Conversational Recommender System and LLM Integration for E-commerce Pre-sales Dialogue

Introduction

The exploration of integrating Conversational Recommender Systems (CRS) with LLMs is an intriguing focus due to their complementary strengths in E-commerce pre-sales dialogues. CRS are adept at learning user preferences and making recommendations based on dialogue context but rely heavily on external knowledge sources. Conversely, LLMs, with vast pre-trained knowledge, handle natural language generation effectively but lack domain-specific expertise pivotal for detailed recommendations.

Integration Approaches

The research introduces two main methods for integrating CRS with LLMs: LLM assisting CRS and CRS assisting LLM. These approaches are evaluated across four key tasks in pre-sales dialogues, illustrating varied effectiveness depending on the specific task and collaboration direction.

LLM Assisting CRS

When implementing LLM to assist CRS, responses generated by the LLM are appended to the CRS inputs. This collaboration method is highly effective in enhancing the recommendation task by leveraging LLM outputs to enrich user representation within the CRS framework.

Figure 1: An example of collaboration between CRS and LLM on the user needs elicitation task, demonstrating enhanced data input and output processes.

CRS Assisting LLM

Alternatively, when CRS assist LLM, the predictions made by CRS are appended to LLM inputs. For recommendation tasks, CRS outputs contribute directly to refining the LLM's decision-making process, thereby optimizing the alignment between user needs and model recommendations.

Figure 2: A comparison of the collaboration between CRS and LLM, focusing on understanding task enhancements through joint integration strategies.

Evaluation and Results

The integration strategies were tested using the U-NEED dataset, encompassing pre-sales dialogues in multiple categories including Beauty, Phones, and Electronics. The evaluation metrics included precision, recall, and f1-score for dialogue understanding and generation, as well as hit rates for recommendation tasks.

Dialogue Understanding

The collaborations largely improved dialogue understanding tasks, where CRS-assisted LLM configurations typically outperformed standalone configurations, especially in tasks demanding nuanced understanding of user preferences.

Recommendation Quality

Significant improvements were noted in recommendation tasks where LLM-assisted CRS configurations annotated recommendation lists more accurately than individual models. This demonstrates the strategic advantage of using LLM to contextualize CRS predictions.

Figure 3: Example illustrating the CRS and LLM collaboration in enhancing dialogue generation tasks, with emphasized collaboration contents.

Implications and Future Research

This research opens pathways for more sophisticated and dynamically collaborative models in dialogue-based recommendation systems. The adaptability of LLM to incorporate domain-specific input refines the recommendation process, while CRS systems benefit from the comprehensive context generated by LLMs.

Future studies may explore extensions of these methods to even broader E-commerce scenarios, considering additional factors such as user sentiment and emotion recognition to further personalize interactions. Moreover, experimenting with newer, larger LLM models could yield additional insights into scaling these systems while maintaining efficiency.

Conclusion

The studied collaboration of CRS and LLMs marks an important step towards more intelligent, responsive, and user-aligned dialogue systems within E-commerce platforms. By strategically leveraging the strengths of both technologies, this integration enhances the overall quality and relevance of user interactions and recommendations, setting a new standard for future conversational AI systems.