Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models (2403.12881v1)

Abstract: Open-sourced LLMs have achieved great success in various NLP tasks, however, they are still far inferior to API-based models when acting as agents. How to integrate agent ability into general LLMs becomes a crucial and urgent problem. This paper first delivers three key observations: (1) the current agent training corpus is entangled with both formats following and agent reasoning, which significantly shifts from the distribution of its pre-training data; (2) LLMs exhibit different learning speeds on the capabilities required by agent tasks; and (3) current approaches have side-effects when improving agent abilities by introducing hallucinations. Based on the above findings, we propose Agent-FLAN to effectively Fine-tune LLMs for Agents. Through careful decomposition and redesign of the training corpus, Agent-FLAN enables Llama2-7B to outperform prior best works by 3.5\% across various agent evaluation datasets. With comprehensively constructed negative samples, Agent-FLAN greatly alleviates the hallucination issues based on our established evaluation benchmark. Besides, it consistently improves the agent capability of LLMs when scaling model sizes while slightly enhancing the general capability of LLMs. The code will be available at https://github.com/InternLM/Agent-FLAN.

• The paper introduces Agent-FLAN, a fine-tuning methodology that restructures agent training data and decomposes tasks to improve LLM agent capabilities.
• It employs differentiated training speeds and negative samples to mitigate hallucinations while achieving a 3.5% performance boost in experiments.
• The findings bridge the gap between open-sourced and API-based LLMs, paving the way for more adaptable and intelligent AI agents.

Enhancing Agent Abilities in LLMs with Agent-FLAN

Introduction to Agent-FLAN

The quest to imbue LLMs with robust agent capabilities has led to the development of Agent-FLAN, a fine-tuning methodology designed to effectively enhance LLMs' performance in agent tasks. The research stems from the observation that while open-sourced LLMs demonstrate exceptional proficiency in natural language understanding and generation, their ability to act as agents—making decisions based on environmental inputs and executing tasks—lags behind that of their API-based counterparts. Agent-FLAN (Fine-tuning LLMs for Agents) addresses this gap by refining the agent training corpus and introducing novel fine-tuning techniques tailored for agent tasks.

Key Observations and Methodology

The development of Agent-FLAN was guided by three pivotal observations, each highlighting specific challenges and opportunities in agent tuning:

1. Entanglement of Agent Training Data: The paper found that most agent training data mixes format adherence with agent reasoning, diverging significantly from the pre-training data distribution. This misalignment complicates the learning process for LLMs, constraining their ability to acquire agent-specific skills effectively.
2. Variable Learning Speeds: LLMs exhibit different learning velocities across various agent-related capabilities. This discrepancy suggests a need for tailored training approaches that account for the unique learning dynamics of each capability.
3. Side-Effects of Existing Approaches: Current strategies to enhance agent abilities in LLMs often lead to unintended consequences, such as the introduction of hallucinations—misleading, inaccurate, or irrelevant outputs.

To navigate these challenges, Agent-FLAN employs a multi-faceted approach:

• Alignment with Natural Language Domain: By restructuring agent training data to resemble natural conversations, Agent-FLAN mitigates the issue of data entanglement, facilitating more effective learning of agent abilities.
• Decomposition and Balanced Training: The methodology breaks down agent tasks into fundamental capabilities and adjusts the training focus according to the distinct learning rates of these capabilities.
• Mitigation of Hallucinations: Through the creation of an evaluation benchmark for hallucination and the incorporation of negative samples, Agent-FLAN significantly reduces the occurrence of hallucination in LLM outputs.

Empirical Validation and Results

Agent-FLAN's efficacy is demonstrated through a series of comprehensive experiments using the Llama2-7B model across various agent evaluation benchmarks. The approach achieved a 3.5\% improvement over previous works, showcasing its potential to significantly enhance the agent capabilities of LLMs. Additionally, Agent-FLAN was found to not only boost agent-specific abilities but also slightly improve the general capabilities of LLMs, underscoring the versatile benefits of the proposed fine-tuning methodology.

Implications and Future Directions

The success of Agent-FLAN in enhancing agent abilities of LLMs has several important implications:

• Bridging the Gap: The methodology represents a significant step toward narrowing the performance gap between open-sourced LLMs and API-based models in agent tasks.
• Flexible Learning: The differentiated learning strategies for various agent capabilities highlight the importance of adaptable training methods in maximizing LLMs' potential.
• Holistic Model Improvement: The positive impact of Agent-FLAN on both agent and general capabilities of LLMs suggests a pathway for developing more universally competent models.

Looking ahead, the insights gained from Agent-FLAN pave the way for further exploration in integrating effective agent functions into LLMs. Future research may delve into more granular training data decomposition, examine the scalability of Agent-FLAN across larger model sizes, and explore its applicability to a broader range of agent tasks.

In conclusion, Agent-FLAN offers a promising avenue for fortifying the agent capabilities of LLMs, marking an important advancement in the pursuit of more intelligent and versatile AI agents.