Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback (2401.08959v1)
Abstract: Probabilistic learning to rank (LTR) has been the dominating approach for optimizing the ranking metric, but cannot maximize long-term rewards. Reinforcement learning models have been proposed to maximize user long-term rewards by formulating the recommendation as a sequential decision-making problem, but could only achieve inferior accuracy compared to LTR counterparts, primarily due to the lack of online interactions and the characteristics of ranking. In this paper, we propose a new off-policy value ranking (VR) algorithm that can simultaneously maximize user long-term rewards and optimize the ranking metric offline for improved sample efficiency in a unified Expectation-Maximization (EM) framework. We theoretically and empirically show that the EM process guides the leaned policy to enjoy the benefit of integration of the future reward and ranking metric, and learn without any online interactions. Extensive offline and online experiments demonstrate the effectiveness of our methods.
- Variational inference: A review for statisticians. JASA.
- Learning to rank using gradient descent. In ICML, 89–96.
- Learning to rank: from pairwise approach to listwise approach. In ICML.
- Top-K Off-Policy Correction for a REINFORCE Recommender System. In WSDM, 456–464.
- Doubly Robust Policy Evaluation and Learning. In ICML.
- Off-policy deep reinforcement learning without exploration. In ICML.
- Posterior regularization for structured latent variable models. JMLR.
- Reinforcement learning with deep energy-based policies. In ICML.
- Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML.
- Expectation propagation for approximate inference in. In UAI.
- Session-based recommendations with recurrent neural networks. In ICLR.
- Distilling the Knowledge in a Neural Network. stat, 1050: 9.
- Collaborative filtering for implicit feedback datasets. In ICDM.
- Recsim: A configurable simulation platform for recommender systems. arXiv.
- Self-attentive sequential recommendation. In ICDM, 197–206. IEEE.
- Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779.
- Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643.
- Variational autoencoders for collaborative filtering. In WWW, 689–698.
- Continuous control with deep reinforcement learning. arXiv.
- Safe and efficient off-policy reinforcement learning. In NIPS, 1054–1062.
- A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, 355–368. Springer.
- BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618.
- Rényi, A.; et al. 1961. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability.
- Robertson, S. E. 1977. The probability ranking principle in IR. Journal of documentation.
- Prioritized Experience Replay. In ICLR.
- Personalized top-n sequential recommendation via convolutional sequence embedding. In WSDM, 565–573.
- Q-learning. Machine learning.
- Listwise approach to learning to rank: theory and algorithm. In ICML.
- Learning How to Propagate Messages in Graph Neural Networks. In KDD, 1894–1903.
- Hierarchical neural variational model for personalized sequential recommendation. In WWW, 3377–3383.
- A General Offline Reinforcement Learning Framework for Interactive Recommendation. In AAAI, 4512–4520.
- Self-supervised reinforcement learning for recommender systems. In SIGIR, 931–940.
- Deep reinforcement learning for page-wise recommendations. In RecSys, 95–103.
- Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning. In KDD, 1040–1048.
- DRN: A Deep Reinforcement Learning Framework for News Recommendation. In WWW, 167–176.
- Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning. In SIGIR, 179–188.
- Pseudo Dyna-Q: A Reinforcement Learning Framework for Interactive Recommendation. In WSDM, 816–824.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.