A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges

A Comprehensive Survey on Game Playing Agents Leveraging Large Models

Introduction to LMs in Game Playing Agents

The advent of Large Models (LMs) in the realm of game playing has ushered in a significant shift in how artificial intelligence can learn, reason, and engage within complex digital environments. Game Playing Agents (GPAs) encapsulate a critical area of research, providing insights into the capabilities of LMs and presenting a multitude of challenges and opportunities for future advancements. This survey aims to meticulously review the landscape of LM-based Agents (LMAs) in gaming, elucidating common methodologies, applications, challenges, and prospective research directions. The focus is primarily on the utilization of LMs across various stages of gameplay, encompassing perception, inference, behavior, and the overarching goal of improving authenticity and interaction within game scenarios.

Perception in Game Playing Agents

Perception serves as the foundation for GPAs, enabling them to interpret and react to the game environment. This includes:

• Semantic Understanding: Initial LMAs predominantly focused on textual information processing within games, necessitating a comprehensive understanding of semantics for effective gameplay. Advanced approaches now integrate visual and, potentially, auditory data to enhance the multi-modal perception capabilities of agents.
• Visual Perception: The incorporation of Multi-modal LLMs (MLLMs) is considered a substantial improvement, enabling agents to process visual cues more effectively, thus opening avenues for richer interaction and engagement within games.

The Role of Inference in Enhancing GPAs

Inference encompasses the cognitive processes that allow GPAs to make decisions based on perceived information. This involves:

• Learning and Reasoning: The application of learned knowledge to new scenarios is critical for the generalization capabilities of GPAs. LMs exhibit promising results in adapting learned behaviors to new tasks, showcasing potential in ongoing learning and adaptation.
• Decision-making and Reflection: Advanced LMs enable GPAs to undertake complex decision-making processes, involving multi-hop inference and long-term planning. Reflection mechanisms further allow for the evaluation and improvement of decisions based on feedback.

Behaviors and Actions of GPAs

Action execution in GPAs is pivotal for interacting with the game environment, characterized by:

• Generative Programming Techniques: These are employed for task execution, where iterative prompting and program generation based on language instructions play significant roles.
• Dialogue Interactions: Communication, either between agents or with human players, forms a crucial interaction channel within games.
• Consistency in Action: Ensuring that actions are coherent and contextually appropriate across diverse game scenarios is paramount for the viability of LMAs in gaming.

Addressing Challenges in Game Playing Agents

A myriad of challenges confront LMAs in gaming, such as:

• Hallucination: The propensity of LMs to generate outputs that may not align with the game's reality represents a critical hurdle.
• Error Correction: The ability of GPAs to identify and rectify errors autonomously is essential for sustained gameplay and learning.
• Generalization: The capacity to extend learned behaviors and strategies to new, unseen tasks remains a significant challenge, reflective of the broader goal of achieving AGI.
• Interpretability: Ensuring that the decision-making process of GPAs is transparent and understandable is crucial for debugging, improvement, and user trust.

Future Directions

The future trajectory of research in GPAs and LMs is oriented towards enhancing multi-modal perception, achieving authentic gaming experiences, leveraging external tools for comprehensive interaction, and excelling in real-time gaming scenarios. Further investigation into the application of LMs in game testing, level and story generation, and mechanics tuning presents a fertile ground for exploration.

Conclusion

The integration of LMs into game playing agents marks a substantial leap towards creating intelligent systems capable of navigating and interacting within complex digital environments. Despite the progress, the path towards fully autonomous, adaptable, and intelligent game agents presents numerous challenges and opportunities for innovation. As the field continues to evolve, the continued exploration of these avenues will undoubtedly yield significant advancements in artificial intelligence, game design, and beyond.