Improved Baselines with Visual Instruction Tuning (2310.03744v2)

Abstract: Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.

• The paper introduces a novel MLP-based vision-language connector that improves data efficiency and sets new SoTA performance on 11 benchmarks.
• It demonstrates scalable training with high-resolution images and robust multilingual visual conversations using limited academic-task-oriented data.
• The study highlights advancements in reducing dataset dependency and overfitting, laying groundwork for universal multimodal systems.

Improved Baselines with Visual Instruction Tuning

Introduction

The research paper "Improved Baselines with Visual Instruction Tuning" (2310.03744) conducts a comprehensive paper focusing on the optimization of Large Multimodal Models (LMMs) through the LLaVA framework. This paper emphasizes systematic advancements in training methodologies for LMMs, aiming to streamline their efficiency and broaden their capabilities using visual instruction tuning.

Methodological Advancements

The paper introduces significant enhancements to the LLaVA framework, primarily through the incorporation of a fully-connected vision-language connector, highlighting its surprising efficacy in data-efficient training. This is achieved by substituting the original projection with a Multi-Layer Perceptron (MLP) connector, which, when combined with academic-task-oriented VQA datasets, establishes a robust baseline across 11 benchmarks. The model sets new state-of-the-art (SoTA) performance while utilizing only 1.2 million public data samples and requiring approximately one day of training on an 8-A100 GPU node.

Figure 1: LLaVA-1.5 achieves SoTA on a broad range of 11 tasks (Top), with high training sample efficiency (Left) and simple modifications to LLaVA (Right): an MLP connector and including academic-task-oriented data with response formatting prompts.

Explorations and Findings

The paper further explores several open problems related to LMMs, such as scaling input resolutions, enabling compositional capabilities, and reducing model hallucinations. Notably, the research demonstrates that LLaVA-1.5-HD, an extension facilitating high-resolution image inputs, achieves scalable performance without necessitating positional embedding interpolation. This involves dividing higher resolution images into grids, encoding them independently, and concatenating their features with a downsampled version for global context.

Figure 2: LLaVA-1.5-HD. Scaling LLaVA-1.5 to higher resolutions by splitting the image into grids and encoding them independently.

In terms of data efficiency and model scaling, the paper's ablation studies reveal that the model maintains robust performance even when training data is substantially reduced, indicating potential for dataset compression without sacrificing outcomes. The research also highlights the model's proficiency in multilingual visual conversations, thanks to training it with English-only visual data augmented by multilingual text from ShareGPT.

Implications and Future Directions

This work significantly contributes to the understanding and development of LMMs by providing a straightforward yet effective approach to visual instruction tuning, thereby facilitating the creation of advanced multimodal systems. The findings suggest potential pathways to further decrease data dependency and reduce overfitting on benchmark datasets through refined data integration strategies.

Figure 3: Ablation on LLM choices. Data points represent the relative performance of the best performing variant for each dataset.

The implications of this research extend into potential advancements in universal LMMs capable of comprehensive understanding across multiple modalities and languages. Future developments could focus on further enhancing resolution handling and expanding multimodal capabilities, particularly in understanding more complex language constructs, irrespective of their linguistic origin.

Conclusion

"Improved Baselines with Visual Instruction Tuning" presents a meticulous exploration into optimizing LMMs through the LLaVA framework, underscoring significant advancements in data efficiency and performance across multiple benchmarks. By addressing various open challenges in multimodal learning and providing a robust, reproducible baseline, this research sets the stage for future endeavors in enhancing the accessibility and applicability of state-of-the-art LMMs.