ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (2307.16789v2)

Abstract: Despite the advancements of open-source LLMs, e.g., LLaMA, they remain significantly limited in tool-use capabilities, i.e., using external tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-art (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap, we introduce ToolLLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the construction can be divided into three stages: (i) API collection: we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruction generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop a novel depth-first search-based decision tree algorithm. It enables LLMs to evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of LLMs, we develop an automatic evaluator: ToolEval. Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction. Experiments show that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. Our ToolLLaMA also demonstrates strong zero-shot generalization ability in an out-of-distribution tool-use dataset: APIBench.

• The paper introduces ToolLLM, which enables LLMs to interact with 16,000+ real-world APIs by leveraging a novel instruction-tuning dataset and DFSDT algorithm.
• It presents ToolLLaMA, a fine-tuned LLaMA version that, aided by a neural API retriever, rivals closed-source models on unseen APIs and achieves strong zero-shot performance.
• An automated evaluation framework, ToolEval, validates the approach with high pass and win rates, offering a scalable benchmark for tool-use capabilities.

ToolLLM: Facilitating LLMs to Master 16000+ Real-world APIs

Introduction

The paper "ToolLLM: Facilitating LLMs to Master 16000+ Real-world APIs" focuses on enhancing the tool-use capabilities of open-source LLMs such as LLaMA. Despite the advancements in LLMs, these models still lack the ability to effectively interact with external tools (APIs) to fulfill complex human instructions, unlike their closed-source counterparts like GPT-4. This work introduces ToolLLM, a comprehensive framework that includes innovative approaches to data construction, model training, and evaluation to address this gap.

Figure 1: Architectural overview of ToolLLM showcasing its components and workflow.

ToolBench: Instruction-Tuning Dataset

A significant contribution of this work is the introduction of ToolBench, an instruction-tuning dataset specifically designed for tool-use tasks. ToolBench consists of automatically generated instructions using ChatGPT, which cover interactions with a vast collection of over 16,000 real-world RESTful APIs. The dataset construction involves three stages:

1. API Collection: Gathering 16,464 APIs spanning 49 categories from RapidAPI Hub.
2. Instruction Generation: Using ChatGPT to create diverse instructions for these APIs in both single-tool and multi-tool scenarios.
3. Solution Path Annotation: Employing a novel depth-first search-based decision tree algorithm to annotate valid solution paths for each instruction.

   Figure 2: Dataset construction stages illustrating API collection, instruction generation, and solution path annotation.

Decision Tree Algorithm for Enhanced Reasoning

The paper introduces a depth-first search-based decision tree (DFSDT) algorithm to bolster the reasoning capabilities of LLMs. By enabling the evaluation of multiple reasoning traces, DFSDT facilitates a broader expansion of the search space compared to conventional methods like Chain-of-Thought (CoT) and ReACT. This approach significantly improves the model's ability to annotate complex instructions efficiently, as demonstrated by its superior performance in generating valid solution paths where other methods fail.

ToolLLaMA and ToolEval

ToolLLaMA, a fine-tuned version of LLaMA, is developed using ToolBench and enhanced with a neural API retriever. This retriever aids in recommending appropriate APIs for each instruction, increasing the model's effectiveness in executing complex tasks. Experiments indicate that ToolLLaMA rivals the performance of state-of-the-art closed-source models like ChatGPT, particularly in scenarios involving unseen APIs, thereby showcasing strong zero-shot generalization.

Additionally, the paper introduces ToolEval, an automated evaluation framework to assess tool-use capabilities with two key metrics: pass rate and win rate. ToolEval, validated for high correlation with human evaluations, offers a scalable solution for benchmarking LLM performance in tool-use tasks.

Figure 3: ToolEval schematic illustrating the evaluation process for model's tool-use capabilities.

Experimental Results

The experiments reveal that ToolLLaMA exhibits remarkable capabilities in both single-tool and multi-tool instruction scenarios, outperforming models like Claude-2 and approaching the performance of ChatGPT. The introduction of DFSDT enhances decision-making strategies, leading to improved outcomes compared to ReACT. Furthermore, ToolLLaMA demonstrates strong generalization on an out-of-distribution dataset, APIBench, by effectively adapting to new APIs with minimal additional data.

Conclusion

This research marks significant progress in equipping open-source LLMs with advanced tool-use skills. By integrating novel data construction methods, enhanced reasoning algorithms, and robust evaluation metrics, ToolLLM effectively bridges the capability gap between open-source and closed-source models. The flexibility and generalization ability of ToolLLaMA open new avenues for AI systems that can dynamically interact with a diverse range of real-world applications, paving the way for future developments in autonomous decision-making and API-driven AI models.