Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
104 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
40 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning (2401.10727v3)

Published 19 Jan 2024 in cs.CV

Abstract: Recently, the astonishing performance of LLMs in natural language comprehension and generation tasks triggered lots of exploration of using them as central controllers to build agent systems. Multiple studies focus on bridging the LLMs to external tools to extend the application scenarios. However, the current LLMs' ability to perceive tool use is limited to a single text query, which may result in ambiguity in understanding the users' real intentions. LLMs are expected to eliminate that by perceiving the information in the visual- or auditory-grounded instructions. Therefore, in this paper, we propose MLLM-Tool, a system incorporating open-source LLMs and multi-modal encoders so that the learned LLMs can be conscious of multi-modal input instruction and then select the function-matched tool correctly. To facilitate the evaluation of the model's capability, we collect a dataset featuring multi-modal input tools from HuggingFace. Another essential feature of our dataset is that it also contains multiple potential choices for the same instruction due to the existence of identical functions and synonymous functions, which provides more potential solutions for the same query. The experiments reveal that our MLLM-Tool is capable of recommending appropriate tools for multi-modal instructions. Codes and data are available at https://github.com/MLLM-Tool/MLLM-Tool.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  3. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
  4. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023.
  5. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://lmsys.org/blog/2023-03-30-vicuna/, 2023.
  6. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  7. Glm: General language model pretraining with autoregressive blank infilling, 2022.
  8. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946–35958, 2022.
  9. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023.
  10. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778, 2021.
  11. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  12. Metatool benchmark: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128, 2023.
  13. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
  14. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a.
  15. Api-bank: A benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023b.
  16. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.
  17. Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support. arXiv preprint arXiv:2307.15810, 2023.
  18. Y Nakajima. Task-driven autonomous agent utilizing gpt-4, pinecone, and langchain for diverse applications. See https://yoheinakajima. com/task-driven-autonomous-agent-utilizing-gpt- 4-pinecone-and-langchain-for-diverse-applications (accessed 18 April 2023), 2023.
  19. OpenAI. Gpt-4v(ision) system card. https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023.
  20. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  21. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.
  22. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023.
  23. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023.
  24. Tptu: Task planning and tool usage of large language model-based ai agents. arXiv preprint arXiv:2308.03427, 2023.
  25. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
  26. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  27. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023.
  28. Restgpt: Connecting large language models with real-world applications via restful apis. arXiv preprint arXiv:2306.06624, 2023.
  29. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023.
  30. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128, 2023.
  31. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023.
  32. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078–10093, 2022.
  33. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  34. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  35. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  36. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023a.
  37. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023b.
  38. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504, 2023.
  39. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023a.
  40. Auto-gpt for online decision making: Benchmarks and additional opinions, 2023b.
  41. Gpt4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arXiv:2305.18752, 2023c.
  42. Glm-130b: An open bilingual pre-trained model, 2023.
  43. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023a.
  44. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF Intersnational Conference on Computer Vision, pages 3836–3847, 2023b.
  45. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
Citations (10)
View on Semantic Scholar

Summary

  • The paper introduces MLLM-Tool, a novel system that integrates multimodal encoders with LLMs to enhance tool selection accuracy.
  • The paper presents the ToolMMBench dataset with over 932 high-quality machine learning APIs for realistic multimodal instructions.
  • The paper demonstrates an 88.19% accuracy in tool selection and highlights the benefits of fine-tuning with LoRA for scalable performance.

Overview of "MLLM-Tool: A Multimodal LLM For Tool Agent Learning"

The paper "MLLM-Tool: A Multimodal LLM For Tool Agent Learning" addresses the limitation of current LLMs in comprehending and utilizing external tools based solely on text inputs. The authors propose MLLM-Tool, an innovative system that integrates multimodal encoders with open-source LLMs to process and understand instructions formed from diverse modalities, including visual and auditory inputs. This advancement aims to enhance the capability of LLMs in selecting appropriate tools when faced with tasks requiring more than textual input, thus reducing ambiguity and improving accuracy in understanding user intentions.

Key Contributions

  1. Integration of Multimodal Encoders and LLMs: The system allows LLMs to be conscious of and integrate inputs across various modalities, ensuring a broader understanding of the task at hand and more precise tool selection.
  2. ToolMMBench Dataset: The authors introduce a novel dataset, ToolMMBench, compiled from HuggingFace, including more than 932 high-quality machine learning APIs. The dataset not only encompasses multifaceted modalities but also includes numerous instances where single instructions are associated with multiple potential APIs, reflecting more realistic scenarios.
  3. Performance Metrics and Evaluation:
    • The authors establish evaluation metrics that consider the specifics of multimodal inputs, ambiguity types, and varied modality combinations to comprehensively assess the model's performance.
    • Extensive experiments reveal that MLLM-Tool achieves a tool selection accuracy of 88.19%, demonstrating its effectiveness in selecting the correct tools for multimodal instructions.
  4. Fine-tuning with Low-Rank Adaptation (LoRA): The paper employs LoRA to fine-tune LLMs efficiently, optimizing performance while minimizing overhead in parameters.

Findings and Implications

  • Accuracy and Ambiguity Resolution: MLLM-Tool shows superior accuracy in resolving ambiguities brought by multimodal instructions compared to traditional text-only instruction following, underlining the importance of incorporating visual and auditory information in task execution.
  • The Model Configuration: The exploration of multiple LLM configurations, including Vicuna, Llama, and Llama2, indicates that larger models (13B) generally outperform their smaller counterparts (7B) after adequate training, highlighting the scaling advantages in multimodal contexts.
  • Practical Implications: This development could provide substantial improvements in LLM-based systems, such as virtual assistants and autonomous agents, which require interaction with diverse data forms and external systems.

Future Directions

  • Extension to More Complex Scenarios: While MLLM-Tool deals with a defined set of APIs, its methodology could extend to explore open-domain tool learning, especially as LLMs continue to evolve with better interpretative layers for varied modalities.
  • Integration with Enhanced Interaction Techniques: Implementing Chain-of-Thought prompting and ensuring compatibility with multistep and interactive task processing could offer further sophistication to the system.
  • Increased Dataset Diversity: As Transformer-based models and APIs proliferate, additional integrations of APIs within other specialized fields could enhance the robustness of such systems across broader applications.

In conclusion, MLLM-Tool exemplifies a significant stride towards equipping LLM-based systems with comprehensive multimodal capabilities, bridging the gap between human-like understanding and computational efficiency in executing tasks across diverse platforms.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - Tool-LMM/Tool-LMM: Tool-LMM: A Large Multi-Modal Model for Tool Agent Learning (79 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets