BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2301.12597v3)

Abstract: The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen LLMs. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen LLM. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.

• The paper introduces a two-stage bootstrapping strategy that aligns frozen image encoders with LLMs using a lightweight Q-Former.
• It achieves an 8.7% improvement in zero-shot VQA accuracy while excelling in image captioning and retrieval tasks.
• The approach reduces training parameters and costs, setting a new benchmark in multimodal reasoning for vision-language tasks.

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and LLMs

Introduction

BLIP-2 introduces a novel vision-language pre-training framework that promises efficiency and high performance by leveraging frozen pre-trained image encoders and LLMs. By employing a two-stage, bootstrapped learning strategy, it optimally aligns image representations with LLMs without the need to retrain extensive components fully. This approach is particularly essential in reducing the computational expense of vision-language pre-training, which has escalated with the increase in model scales.

Methodology

The core innovation of BLIP-2 is the use of a lightweight Querying Transformer (Q-Former) to bridge modality gaps, specifically the alignment between vision and language domains. This model undergoes a two-tier pre-training approach.

Figure 1: Overview of BLIP-2's framework. We pre-train a lightweight Querying Transformer following a two-stage strategy to bridge the modality gap.

Stage 1: Vision-Language Representation Learning

The first stage leverages a frozen image encoder to bootstrap the vision-language representation learning process. Within this phase, BLIP-2 employs a multi-objective optimization strategy to ensure the Q-Former can effectively extract visual features relevant to textual data. The objectives include Image-Text Contrastive Learning (ITC), Image-grounded Text Generation (ITG), and Image-Text Matching (ITM).

Figure 2: (Left) Model architecture of Q-Former and BLIP-2's first-stage vision-language representation learning objectives.

Stage 2: Vision-to-Language Generative Pre-training

In the second phase, Q-Former helps facilitate vision-to-language generative learning with a frozen LLM. Both decoder-based and encoder-decoder-based LLMs have been evaluated, further emphasizing zero-shot capability and efficiency in extracting image-relevant textual representations for LLMs.

Figure 3: BLIP-2's second-stage vision-to-language generative pre-training, leveraging frozen LLMs.

The inclusion of pre-trained, unimodal networks (both vision and language) aids this model in achieving a balanced alignment without catastrophic forgetting, thus allowing the learning of vision-language associations with minimal new parameter training.

Experimental Results

BLIP-2's framework shows impressive performance benchmarks across multiple tasks including visual question answering (VQA), image captioning, and image-text retrieval.

Visual Question Answering

For VQA, BLIP-2 demonstrates improved accuracy, especially in the zero-shot setting. It surpasses the previous state-of-the-art, Flamingo80B, by achieving a performance margin of 8.7% on zero-shot VQAv2, despite utilizing significantly fewer parameters. The system's constructive use of both frozen models enhances its flexibility and generalization ability across unseen data.

Figure 4: Effect of vision-language representation learning on vision-to-language generative learning.

Image Captioning

In structured evaluations, BLIP-2 exhibits robust capabilities in both in-domain and zero-shot transfer scenarios for image captioning tasks. Its performance on NoCaps and COCO Caption datasets validates its retrieval prowess and textual generation reliability. This is particularly noteworthy as the results show BLIP-2 outperforming more parameter-heavy models.

Image-Text Retrieval

BLIP-2's finetuned models for image-text retrieval achieve high retrieval accuracy. By tuning on COCO and evaluating on Flickr30K datasets, the model consistently bests competitors in image-to-text and text-to-image retrieval, marking a significant milestone in aligning visual and linguistic information with greater efficacy.

Conclusion

BLIP-2 presents a computationally efficient framework that judiciously employs pre-trained image encoders and LLMs to achieve state-of-the-art results in vision-language tasks with minimal parameter training. This method sets the stage for further research on multi-modal conversational AI systems, opening avenues for enhanced visual understanding and language generation models with broader applications. By reducing computational costs without sacrificing performance, BLIP-2 provides a strategic direction for future advancements in integrated AI systems capable of performing complex, multimodal reasoning tasks.