Linear Recurrent Units for Sequential Recommendation (2310.02367v2)

Abstract: State-of-the-art sequential recommendation relies heavily on self-attention-based recommender models. Yet such models are computationally expensive and often too slow for real-time recommendation. Furthermore, the self-attention operation is performed at a sequence-level, thereby making low-cost incremental inference challenging. Inspired by recent advances in efficient LLMing, we propose linear recurrent units for sequential recommendation (LRURec). Similar to recurrent neural networks, LRURec offers rapid inference and can achieve incremental inference on sequential inputs. By decomposing the linear recurrence operation and designing recursive parallelization in our framework, LRURec provides the additional benefits of reduced model size and parallelizable training. Moreover, we optimize the architecture of LRURec by implementing a series of modifications to address the lack of non-linearity and improve training dynamics. To validate the effectiveness of our proposed LRURec, we conduct extensive experiments on multiple real-world datasets and compare its performance against state-of-the-art sequential recommenders. Experimental results demonstrate the effectiveness of LRURec, which consistently outperforms baselines by a significant margin. Results also highlight the efficiency of LRURec with our parallelized training paradigm and fast inference on long sequences, showing its potential to further enhance user experience in sequential recommendation.

• The paper introduces LRURec, which leverages efficient linear recurrent units to balance computational speed and recommendation quality.
• The model uses matrix diagonalization and optimizations like layer normalization to enhance parallel training and convergence.
• Extensive experiments show improved NDCG@10 and Recall@10, demonstrating LRURec's practical viability in real-time systems.

Linear Recurrent Units for Sequential Recommendation

The paper "Linear Recurrent Units for Sequential Recommendation" addresses the efficiency and effectiveness trade-offs in state-of-the-art sequential recommender systems, particularly those based on self-attentive architectures. While self-attention mechanisms have proven to be highly effective in capturing user-item interactions, these models are computationally intensive, making real-time recommendation challenging. The authors propose Linear Recurrent Units for Sequential Recommendation (LRURec), which, akin to Recurrent Neural Networks (RNNs), support rapid and incremental inference. LRURec aims to combine the computational efficiency characteristic of RNNs with the superior modeling capabilities of transformer-based architectures.

Key Contributions

1. Model Design: The core of LRURec lies in its efficient linear recurrent units that abandon the non-linearity to permit closed-form expression while maintaining effectiveness. The model applies matrix diagonalization to facilitate parallelizable training—a significant advantage over traditional RNNs which suffer from inefficiencies due to their lack of parallelism.
2. Optimization Techniques: The authors enhance the LRURec architecture by integrating various optimization techniques, such as layer normalization and position-wise feed-forward networks. These are known to bolster training dynamics and performance in deep learning architectures.
3. Experimental Validation: Extensive experiments on real-world datasets demonstrate that LRURec consistently outperforms existing state-of-the-art sequential recommenders in terms of both recommendation performance and computational efficiency. The model showcases superior training convergence and fast inference capabilities, highlighting its practical viability.

Technical Evaluation

• Algorithmic Efficiency: LRURec leverages recursive parallelization, which dramatically enhances its scalability by reducing the computational complexity of processing sequences. This aspect is crucial for handling longer sequences prevalent in real-world applications.
• Empirical Results: The empirical results support LRURec’s capability to outperform the baseline methods across various standard metrics, including NDCG@10 and Recall@10. The paper quantifies the improvements in recommendation accuracy, underscoring the model’s utility in sparse and dense interaction environments.

Implications and Future Directions

The introduction of LRURec signifies an impactful stride in designing recommender systems that need to handle both latency and accuracy constraints. The work challenges the prevailing notion that the self-attention mechanism is indispensable for high-performance recommendation models. By highlighting the efficacy of linear recurrence with enhanced architectural features, the research opens avenues for further exploration into lightweight yet effective architectures.

Practically, the deployment of LRURec can lead to significant resource savings in real-time systems without sacrificing recommendation quality. Theoretically, this work motivates additional studies into the roles of linear transformations within neural architectures, potentially influencing future developments in neural sequence models beyond recommender systems. Extending this framework to encompass multimodal or cross-domain sequential recommendation scenarios may also prove to be a fruitful direction for forthcoming research.

In summation, the proposed LRURec presents a compelling alternative to the heavy reliance on self-attention in current sequential models, providing a balanced approach that reconciles the constraints of performance and computational efficiency.