Large Language Models Enhanced Collaborative Filtering (2403.17688v2)
Abstract: Recent advancements in LLMs have attracted considerable interest among researchers to leverage these models to enhance Recommender Systems (RSs). Existing work predominantly utilizes LLMs to generate knowledge-rich texts or utilizes LLM-derived embeddings as features to improve RSs. Although the extensive world knowledge embedded in LLMs generally benefits RSs, the application can only take limited number of users and items as inputs, without adequately exploiting collaborative filtering information. Considering its crucial role in RSs, one key challenge in enhancing RSs with LLMs lies in providing better collaborative filtering information through LLMs. In this paper, drawing inspiration from the in-context learning and chain of thought reasoning in LLMs, we propose the LLMs enhanced Collaborative Filtering (LLM-CF) framework, which distils the world knowledge and reasoning capabilities of LLMs into collaborative filtering. We also explored a concise and efficient instruction-tuning method, which improves the recommendation capabilities of LLMs while preserving their general functionalities (e.g., not decreasing on the LLM benchmark). Comprehensive experiments on three real-world datasets demonstrate that LLM-CF significantly enhances several backbone recommendation models and consistently outperforms competitive baselines, showcasing its effectiveness in distilling the world knowledge and reasoning capabilities of LLM into collaborative filtering.
- Better Fine-Tuning by Reducing Representational Collapse. ArXiv abs/2008.03156 (2020).
- TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation. Proceedings of the 17th ACM Conference on Recommender Systems (2023).
- TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2023).
- A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402 (2023).
- Leveraging Large Language Models for Pre-trained Recommender Systems. arXiv:2308.10837 [cs.IR]
- Deep Neural Networks for YouTube Recommendations. Proceedings of the 10th ACM Conference on Recommender Systems (2016).
- Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, Toronto, Canada, 4005–4019.
- Uncovering ChatGPT’s Capabilities in Recommender Systems. Proceedings of the 17th ACM Conference on Recommender Systems (2023). https://api.semanticscholar.org/CorpusID:258461170
- How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition. ArXiv abs/2310.05492 (2023).
- A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022).
- Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5). In Proceedings of the 16th ACM Conference on Recommender Systems (Seattle, WA, USA) (RecSys ’22). Association for Computing Machinery, New York, NY, USA, 299–315. https://doi.org/10.1145/3523227.3546767
- DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (Melbourne, Australia) (IJCAI’17). AAAI Press, 1725–1731.
- Leveraging Large Language Models for Sequential Recommendation. Proceedings of the 17th ACM Conference on Recommender Systems (2023). https://api.semanticscholar.org/CorpusID:261823711
- Ruining He and Julian McAuley. 2016. Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering. In Proceedings of the 25th International Conference on World Wide Web (Montréal, Québec, Canada) (WWW ’16). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 507–517. https://doi.org/10.1145/2872427.2883037
- Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020).
- Balázs Hidasi and Alexandros Karatzoglou. 2018. Recurrent Neural Networks with Top-k Gains for Session-based Recommendations. In CIKM. ACM, 843–852.
- Distilling the Knowledge in a Neural Network. arXiv:1503.02531 [stat.ML]
- LoRA: Low-Rank Adaptation of Large Language Models. In ICLR. OpenReview.net.
- NEFTune: Noisy Embeddings Improve Instruction Finetuning. arXiv preprint arXiv:2310.05914 (2023).
- Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM). IEEE, 197–206.
- Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction. ArXiv abs/2305.06474 (2023). https://api.semanticscholar.org/CorpusID:258615591
- Reformer: The Efficient Transformer. In International Conference on Learning Representations.
- XDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 1754–1763. https://doi.org/10.1145/3219819.3220023
- ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation. arXiv preprint arXiv:2308.11131 (2023).
- A First Look at LLM-Powered Generative News Recommendation. arXiv preprint arXiv:2305.06566 (2023).
- Kai Mei and Yongfeng Zhang. 2023. LightLM: A Lightweight Deep and Narrow Language Model for Generative Recommendation. arXiv:2310.17488 (2023).
- Marius Muja and David G. Lowe. 2009. Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration. In International Conference on Computer Vision Theory and Applications.
- Large Language Model Augmented Narrative Driven Recommendations. Proceedings of the 17th ACM Conference on Recommender Systems (2023).
- OpenAI. 2023. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023). https://api.semanticscholar.org/CorpusID:257532815
- Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 (2023).
- Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction. Proceedings of the 29th ACM International Conference on Information & Knowledge Management (2020).
- Retrieval & Interaction Machine for Tabular Data Prediction. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (2021).
- Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16.
- HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogeneous Memory. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS’20). Curran Associates Inc., Red Hook, NY, USA, Article 895, 13 pages.
- Ruifeng Ren and Yong Liu. 2023. In-context Learning with Transformer Is Really Equivalent to a Contrastive Learning Pattern. arXiv preprint arXiv:2310.13220 (2023).
- Large language models are competitive near cold-start recommenders for language-and item-based preferences. In Proceedings of the 17th ACM conference on recommender systems. 890–896.
- AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing, China) (CIKM ’19). Association for Computing Machinery, New York, NY, USA, 1161–1170. https://doi.org/10.1145/3357384.3357925
- Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023).
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
- Deep & Cross Network for Ad Click Predictions. In Proceedings of the ADKDD’17 (Halifax, NS, Canada) (ADKDD’17). Association for Computing Machinery, New York, NY, USA, Article 12, 7 pages. https://doi.org/10.1145/3124749.3124754
- DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-Scale Learning to Rank Systems. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Association for Computing Machinery, New York, NY, USA, 1785–1797. https://doi.org/10.1145/3442381.3450078
- Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022).
- LLMRec: Large Language Models with Graph Augmentation for Recommendation. ArXiv (2023).
- Robust Fine-Tuning of Zero-Shot Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7959–7971.
- Session-Based Recommendation with Graph Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (July 2019), 346–353. https://doi.org/10.1609/aaai.v33i01.3301346
- Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models. arXiv preprint arXiv:2306.10933 (2023).
- C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597 [cs.CL]
- Heterogeneous Knowledge Fusion: A Novel Approach for Personalized Recommendation via LLM. Proceedings of the 17th ACM Conference on Recommender Systems (2023). https://api.semanticscholar.org/CorpusID:260682982
- HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation. In Annual Meeting of the Association for Computational Linguistics.
- LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking. ArXiv abs/2311.02089 (2023). https://api.semanticscholar.org/CorpusID:265033634
- Generative job recommendations with large language model. arXiv preprint arXiv:2307.02157 (2023).
- Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 (2023).
- Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 1059–1068. https://doi.org/10.1145/3219819.3219823
- S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization. In CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020. ACM, 1893–1902.
- Filter-Enhanced MLP is All You Need for Sequential Recommendation. In Proceedings of the ACM Web Conference 2022 (Virtual Event, Lyon, France) (WWW ’22). Association for Computing Machinery, New York, NY, USA, 2388–2399. https://doi.org/10.1145/3485447.3512111
- Ensembled CTR Prediction via Knowledge Distillation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (Virtual Event, Ireland) (CIKM ’20). Association for Computing Machinery, New York, NY, USA, 2941–2958. https://doi.org/10.1145/3340531.3412704