On Generative Agents in Recommendation (2310.10108v3)

Abstract: Recommender systems are the cornerstone of today's information dissemination, yet a disconnect between offline metrics and online performance greatly hinders their development. Addressing this challenge, we envision a recommendation simulator, capitalizing on recent breakthroughs in human-level intelligence exhibited by LLMs. We propose Agent4Rec, a user simulator in recommendation, leveraging LLM-empowered generative agents equipped with user profile, memory, and actions modules specifically tailored for the recommender system. In particular, these agents' profile modules are initialized using real-world datasets (e.g. MovieLens, Steam, Amazon-Book), capturing users' unique tastes and social traits; memory modules log both factual and emotional memories and are integrated with an emotion-driven reflection mechanism; action modules support a wide variety of behaviors, spanning both taste-driven and emotion-driven actions. Each agent interacts with personalized recommender models in a page-by-page manner, relying on a pre-implemented collaborative filtering-based recommendation algorithm. We delve into both the capabilities and limitations of Agent4Rec, aiming to explore an essential research question: ``To what extent can LLM-empowered generative agents faithfully simulate the behavior of real, autonomous humans in recommender systems?'' Extensive and multi-faceted evaluations of Agent4Rec highlight both the alignment and deviation between agents and user-personalized preferences. Beyond mere performance comparison, we explore insightful experiments, such as emulating the filter bubble effect and discovering the underlying causal relationships in recommendation tasks. Our codes are available at https://github.com/LehengTHU/Agent4Rec.

• The paper's main contribution is the creation of Agent4Rec, a simulator that uses LLM-powered generative agents to realistically emulate user behaviors in recommendation systems.
• It introduces a novel agent architecture with dedicated user profile, memory, and action modules that accurately capture user preferences and rating distributions.
• Agent4Rec is validated by its ability to assess various recommendation strategies, replicate filter bubble effects, and uncover causal relationships in user interactions.

On Generative Agents in Recommendation

Introduction

The paper "On Generative Agents in Recommendation" proposes a novel recommendation simulator, Agent4Rec, leveraging generative agents powered by LLMs. This simulator is designed to bridge the gap between offline metrics and online performance in recommender systems, a disconnect that has long hindered the field's advancements. By using LLMs to simulate user behavior in recommendation environments, the authors aim to create a realistic simulation platform that captures user intent and encodes human cognitive mechanisms (Figure 1).

Figure 1: The overall framework of Agent4Rec. Our simulator consists of two core facets: LLM-empowered Generative Agents and Recommendation Environment.

Agent Architecture

Agent4Rec harnesses LLMs to create generative agents with specialized modules: user profile, memory, and action modules, allowing the agents to simulate user behaviors comprehensively. The profile module is initialized using real-world datasets like MovieLens-1M, Steam, and Amazon-Book. It captures users' tastes and social traits, such as activity, conformity, and diversity. Memory modules store factual and emotional memories, with actions driven by taste or emotion. This intricate design enables these agents to interact authentically with recommendation systems.

Evaluation of Agent Alignment

The evaluation of Agent4Rec focuses on the alignment of agents with real user behaviors. Agents demonstrate high accuracy in emulating users’ preferences and rating distributions. This is evident as the agent-simulated ratings align closely with ground-truth distributions (Figure 2).

Figure 2: Comparison between ground-truth and agent-simulated rating distributions.

Social traits play a crucial role in ensuring that agents display diverse behaviors despite similar preferences. The evaluation confirms that these traits significantly influence agent actions, reflecting the simulator’s authenticity in capturing real-world user decision-making patterns (Figure 3).

Figure 3: Averaged scores of activity, conformity, and diversity among agent groups with varying degrees of social traits.

Recommendation Strategy Evaluation

Agent4Rec excels in evaluating different recommendation strategies, demonstrating trends consistent with real-world observations. More advanced algorithms such as LightGCN show higher satisfaction among agents compared to basic strategies like random or popularity-based recommendations. Such evaluations enable the simulator to potentially replace costly A/B testing by providing detailed insights into algorithm performance.

Simulation of Filter Bubble Effect

The filter bubble effect is a recognized challenge in recommendations where user exposure becomes increasingly homogeneous. Agent4Rec successfully replicates this phenomenon, showing reduced content diversity and increasing dominance of popular genres with more iterations (Figure 4).

Figure 4: The simulation performance of Agent4Rec to emulate the filter bubble effect.

Causal Discovery in Recommendations

Agent4Rec facilitates causal discovery by collecting comprehensive interaction data and applying causal inference techniques. The learned causal graph illustrates the relationships between movie quality, popularity, exposure, views, and ratings, shedding light on the intricate dynamics within recommendation systems (Figure 5).

Figure 5: Learned causal graph among movie quality, popularity, exposure rate, view number, and movie rating.

Conclusion

The development of Agent4Rec marks a significant step forward in recommendation systems, offering an innovative simulator that captures complex user behavior through LLM-empowered agents. The insights gained from the evaluation and experiments highlight the simulator's potential to drive future research directions, addressing persistent challenges like filter bubbles and uncovering causal dynamics in user preferences. As the field progresses, Agent4Rec stands poised to become an essential tool for researchers seeking to refine recommendation strategies and better understand user interactions in autonomous systems.