Chameleon: Mixed-Modal Early-Fusion Foundation Models

Abstract: We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.

• The paper introduces a uniform transformer that tokenizes text, images, and code into a single mixed-modal framework for seamless integration.
• It demonstrates state-of-the-art performance in image captioning and competitive results in long-form mixed-modal generation compared to specialized models.
• The study presents architectural innovations like query-key normalization, offering a robust foundation for unified multimodal reasoning and generation.

"Chameleon: Mixed-Modal Early-Fusion Foundation Models" (2405.09818)

Introduction

The paper introduces "Chameleon: Mixed-Modal Early-Fusion Foundation Models," a novel family of foundational models designed for multimodal tasks, integrating text, images, and code into a seamless token-based framework. Chameleon diverges from traditional multimodal models, which often use modality-specific encoders, by employing a uniform architecture that treats all input modalities as discrete tokens. This allows Chameleon to perform complex tasks like visual question answering, image captioning, and both text and image generation within a single model, without needing separate components tailored to each modality.

Figure 1: Chameleon represents all modalities --- images, text, and code, as discrete tokens and uses a uniform transformer-based architecture that is trained from scratch in an end-to-end fashion on $\sim$10T tokens of interleaved mixed-modal data. As a result, Chameleon can both reason over, as well as generate, arbitrary mixed-modal documents. Text tokens are represented in green and image tokens are represented in blue.

Architecture and Training

Chameleon's architecture is a fully tokenized transformer model that ensures all input modalities are integrated early in the processing pipeline. By converting images into discrete tokens similar to text, the model applies the same transformer layers across all data types. This uniform token representation simplifies the model’s design and enhances its capability to handle arbitrary sequences of mixed-modal data. Training stability in Chameleon is achieved through architectural innovations like query-key normalization and revised layer normalization placement, crucial for handling the different entropy levels across modalities.

Performance and Evaluation

Evaluations demonstrate that Chameleon excels across various benchmarks, achieving state-of-the-art results in image captioning and competitive performance on text-only tasks against specialized models like LLaMa-2. Notably, Chameleon's 34B variant surpasses existing models in long-form mixed-modal generation, a testament to its mixed-modal reasoning and generation prowess.

Figure 2: LLaMa-2-7B vs Chameleon-7B Architecture Training Curves over Mixed-Modal Data.

Implications and Future Work

The Chameleon model exemplifies a significant advancement toward integrated multimodal AI frameworks. Its ability to uniformly process and generate content across diverse inputs broadens its applicability in areas such as automated content creation, dynamic storytelling, and comprehensive document understanding. Future work could explore further scaling Chameleon’s architecture and improving inference strategies to enhance efficiency, especially in real-time applications.

Conclusion

Chameleon's development marks a pivotal step in multimodal AI research, offering a versatile, efficient framework that seamlessly bridges the gap between separate modality-specific models. Its innovations in training strategies and architectural design provide a robust foundation for future exploration in unified multimodal learning, potentially influencing various AI applications in interdisciplinary domains.