ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases (2306.05301v2)

Abstract: Enabling LLMs to utilize real-world tools effectively is crucial for achieving embodied intelligence. Existing approaches to tool learning have either primarily relied on extremely LLMs, such as GPT-4, to attain generalized tool-use abilities in a zero-shot manner, or utilized supervised learning to train limited scopes of tools on compact models. However, it remains uncertain whether smaller LLMs can achieve generalized tool-use abilities without tool-specific training. To address this question, this paper introduces ToolAlpaca, a novel framework designed to automatically generate a diverse tool-use corpus and learn generalized tool-use abilities on compact LLMs with minimal human intervention. Specifically, ToolAlpaca first automatically creates a highly diversified tool-use corpus by building a multi-agent simulation environment. The corpus contains 3938 tool-use instances from more than 400 real-world tool APIs spanning 50 distinct categories. Subsequently, the constructed corpus is employed to fine-tune compact LLMs, resulting in two models, namely ToolAlpaca-7B and ToolAlpaca-13B, respectively. Finally, we evaluate the ability of these models to utilize previously unseen tools without specific training. Experimental results demonstrate that ToolAlpaca achieves effective generalized tool-use capabilities comparable to those of extremely LLMs like GPT-3.5, demonstrating that learning generalized tool-use ability is feasible for compact LLMs.

• The paper presents ToolAlpaca, a novel framework that transfers generalized tool-use abilities to compact models using simulated training data.
• It leverages a diverse dataset of 3938 tool-use instances from over 400 APIs across 50 categories generated via multi-agent simulations.
• Experimental results reveal that ToolAlpaca-13B reaches a 70 accuracy on real-world tools, demonstrating competitive performance with larger models.

ToolAlpaca: Generalized Tool Learning for LLMs with 3000 Simulated Cases

The paper "ToolAlpaca: Generalized Tool Learning for LLMs with 3000 Simulated Cases" examines whether compact LLMs can effectively acquire generalized tool-use capabilities, an area traditionally dominated by extremely large models such as GPT-4. The authors introduce ToolAlpaca, a novel framework to enable compact LLMs to perform tool utilization without direct tool-specific training. This paper outlines a method that leverages simulated data generation within a multi-agent environment to fine-tune compact models like Vicuna.

ToolAlpaca addresses a fundamental gap in current AI research by focusing on transferring generalized tool-use capabilities to smaller models. It achieves this through an automatic generation of a diversified tool-use corpus from over 400 APIs across 50 distinct categories, yielding 3938 tool-use instances. The framework simulates diverse real-world scenarios using a multi-agent simulation environment that includes user, assistant, and tool executor agents. These simulated interactions generate a comprehensive dataset of actions, responses, and tool interactions to fine-tune models.

The empirical evaluation of ToolAlpaca focuses on the ability of two compact models, ToolAlpaca-7B and ToolAlpaca-13B, to utilize unforeseen tools. The models, trained on the simulated corpus, were evaluated against both simulated test environments and real-world API tools to assess their generalized tool-use ability. Remarkably, experimental results show that ToolAlpaca models achieve competitive performance levels with models like GPT-3.5. For instance, ToolAlpaca-13B achieved an overall accuracy of 70 in utilizing real-world tools, compared to 75 by GPT-3.5.

A key finding of this research lies in the impact of dataset diversity on tool-use generalization. Tests demonstrated that increasing the variety and complexity of the toolset significantly improved model performance, even when the number of instances remained constant. This insight underscores the importance of a diverse training corpus for developing broad generalization capabilities in LLMs.

The practical implications of ToolAlpaca's contributions are profound. It suggests a scalable approach to developing generalized capabilities in smaller models, potentially democratizing access to advanced AI capabilities without relying on exceptionally large models. Theoretically, it paves the way for future research in AI tool utilization, offering a new paradigm in which models are trained using diverse and simulated data rather than vast quantities of real-world data.

In conclusion, this paper provides compelling evidence that generalized tool-use ability can be effectively transferred to compact LLMs through simulated training, an achievement that traditionally required the computational expense of significantly larger models. As AI development progresses, the principles outlined in ToolAlpaca could influence broader AI applications, promoting efficiency and innovation in model training approaches.