ExpeL: LLM Agents Are Experiential Learners (2308.10144v3)

Abstract: The recent surge in research interest in applying LLMs to decision-making tasks has flourished by leveraging the extensive world knowledge embedded in LLMs. While there is a growing demand to tailor LLMs for custom decision-making tasks, finetuning them for specific tasks is resource-intensive and may diminish the model's generalization capabilities. Moreover, state-of-the-art LLMs like GPT-4 and Claude are primarily accessible through API calls, with their parametric weights remaining proprietary and unavailable to the public. This scenario emphasizes the growing need for new methodologies that allow learning from agent experiences without requiring parametric updates. To address these problems, we introduce the Experiential Learning (ExpeL) agent. Our agent autonomously gathers experiences and extracts knowledge using natural language from a collection of training tasks. At inference, the agent recalls its extracted insights and past experiences to make informed decisions. Our empirical results highlight the robust learning efficacy of the ExpeL agent, indicating a consistent enhancement in its performance as it accumulates experiences. We further explore the emerging capabilities and transfer learning potential of the ExpeL agent through qualitative observations and additional experiments.

• The paper introduces an experiential learning framework that improves LLM agents by extracting insights from both successful and failed trajectories.
• Its methodology leverages reflective learning and task-similarity retrieval, boosting performance with a 39% success rate in HotpotQA over baselines.
• Moreover, ExpeL demonstrates effective transfer learning by achieving a 70% success rate on FEVER, highlighting its potential for adaptable decision-making.

ExpeL: LLM Agents Are Experiential Learners

The paper "ExpeL: LLM Agents Are Experiential Learners" presents a compelling approach to enhancing the capabilities of LLM agents through experiential learning. This framework, known as ExpeL, is crafted to autonomously gather experiences and extract insights in natural language from diverse tasks. The ExpeL agent utilizes these insights and recalls past successful examples during inference, thereby improving decision-making without parameter updates. This method is particularly beneficial given the proprietary nature of cutting-edge LLMs like GPT-4 and Claude, where access to parametric weights might not be possible.

Core Concept and Methodology

The ExpeL framework centers around two primary modes of learning: extracting insights from experience and recalling similar successful experiences as demonstrations. During the training phase, experiences are gathered through multiple trials, enabled by Reflexion—a framework that allows reflective learning based on past failures. This phase involves collecting both successful and failed trajectories across tasks. In the inference phase, the agent recalls these experiences by employing task similarity-based retrieval, providing specific examples as context for decision-making in new tasks.

Key Results

The empirical evaluation across various domains—HotpotQA, ALFWorld, and WebShop—demonstrated ExpeL's efficacy in consistently outperforming baseline models (ReAct and Act). Specifically, the ExpeL agent achieved a 39% success rate in HotpotQA tasks, a notable improvement over ReAct's 28%, which highlights the significant impact of insight extraction on reasoning tasks. In ALFWorld, where task completion relies on specific actions, the retrieval of successful trajectories from similar tasks showed marked improvements. These results underscore the synergistic effect of insight extraction and successful trajectory retrieval in enhancing performance across diverse environments.

Transfer Learning Potential

The paper also explores the transfer learning potential of ExpeL by applying insights gained from HotpotQA to the FEVER dataset. The agent successfully transferred knowledge, achieving a 70% success rate in FEVER tasks, surpassing other baselines. This indicates that ExpeL's experiential learning approach can be beneficial in scenarios where task distributions share common knowledge elements, even when direct retrieval of experiences from one domain might not be feasible.

Implications and Future Directions

The practical implications of ExpeL are significant in scenarios requiring adaptable and efficient decision-making processes. By facilitating cross-task learning and enabling agents to autonomously leverage their experiences, this approach enhances LLM agents without extensive data labeling or computational resources. The findings suggest potential applications in areas such as autonomous systems and interactive agents, where adaptability and incremental learning from diverse inputs are crucial.

Theoretical implications include offering a framework for integrating human-like experiential learning processes within LLM agents, potentially paving the way for more cognitively inspired AI systems. As foundation models and retrieval mechanisms continue to advance, ExpeL stands to benefit from these improvements, suggesting a naturally evolving enhancement pathway for LLM agents.

Future developments could examine the integration of vision and language modalities, refining how insights are dynamically retrieved and applied, or exploring further theoretical underpinnings to establish more optimal agent behaviors. Additionally, the exploration of public-domain models, alongside approaches for effective utilization of proprietary LLMs, may broaden the applicability of ExpeL across diverse domains.