Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion (2401.13388v3)

Published 24 Jan 2024 in cs.CV

Abstract: Existing text-to-image diffusion models primarily generate images from text prompts. However, the inherent conciseness of textual descriptions poses challenges in faithfully synthesizing images with intricate details, such as specific entities or scenes. This paper presents UNIMO-G, a simple multimodal conditional diffusion framework that operates on multimodal prompts with interleaved textual and visual inputs, which demonstrates a unified ability for both text-driven and subject-driven image generation. UNIMO-G comprises two core components: a Multimodal LLM (MLLM) for encoding multimodal prompts, and a conditional denoising diffusion network for generating images based on the encoded multimodal input. We leverage a two-stage training strategy to effectively train the framework: firstly pre-training on large-scale text-image pairs to develop conditional image generation capabilities, and then instruction tuning with multimodal prompts to achieve unified image generation proficiency. A well-designed data processing pipeline involving language grounding and image segmentation is employed to construct multi-modal prompts. UNIMO-G excels in both text-to-image generation and zero-shot subject-driven synthesis, and is notably effective in generating high-fidelity images from complex multimodal prompts involving multiple image entities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset.
  2. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568.
  3. Subject-driven text-to-image generation via apprenticeship learning. arXiv preprint arXiv:2304.00186.
  4. Re-imagen: Retrieval-augmented text-to-image generator. arXiv preprint arXiv:2209.14491.
  5. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499.
  6. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618.
  7. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626.
  8. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851.
  9. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017.
  10. Segment anything. arXiv preprint arXiv:2304.02643.
  11. Generating images with multimodal language models. arXiv preprint arXiv:2305.17216.
  12. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941.
  13. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. arXiv preprint arXiv:2305.14720.
  14. Upainting: Unified text-to-image diffusion generation with cross-modal guidance. arXiv preprint arXiv:2210.16031.
  15. Photomaker: Customizing realistic human photos via stacked id embedding. arXiv preprint arXiv:2312.04461.
  16. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer.
  17. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.
  18. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. arXiv preprint arXiv:2307.11410.
  19. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741.
  20. Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992.
  21. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952.
  22. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3.
  23. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695.
  24. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer.
  25. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510.
  26. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494.
  27. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294.
  28. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565.
  29. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411.
  30. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
  31. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222.
  32. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848.
  33. M-vader: A model for diffusion with multimodal context. arXiv preprint arXiv:2212.02936.
  34. Fastcomposer: Tuning-free multi-subject image generation with localized attention. arXiv preprint arXiv:2305.10431.
  35. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
Citations (4)
View on Semantic Scholar

Summary

  • The paper introduces UNIMO-G, a multimodal conditional diffusion model that fuses text and visual cues to generate detailed, context-specific images.
  • It employs a two-stage training approach using a frozen MLLM and a conditional denoising diffusion network, ensuring robust perceptual encoding and refined image synthesis.
  • Evaluations on datasets like MS-COCO highlight UNIMO-G’s superior performance in subject-driven image generation and handling complex, multi-entity prompts.

Introduction

Advancements in text-to-image diffusion models like DALLE-2, Imagen, and Stable Diffusion have efficiently translated textual prompts into images, yet they encounter limitations when detailed and context-specific renditions are required. UNIMO-G, a multimodal conditional diffusion framework, emerges as a solution, uniquely leveraging both textual and visual inputs within prompts for image generation. This novel approach integrates a Multimodal LLM (MLLM) and a conditional denoising diffusion network to produce images that are not only text-aligned but also visually intricate and contextually specific.

Architecture and Training

UNIMO-G encompasses an in-house Chinese MLLM, similar to MiniGPT-4, with a significant 7 billion parameter size, pre-trained on a myriad of text-image pairs. The model is fine-tuned in two stages; initially through a text-image pair pre-training, thereby acquiring conditional image generation competence, followed by multimodal instruction tuning with multimodal prompts. This second phase is pivotal for the model’s ability to accurately translate complex prompts with multiple entities into high-fidelity images. Notably, throughout this training, the MLLM parameters remain frozen, ensuring that the model's perception capabilities are preserved.

Text-to-Image Pre-training

UNIMO-G's pre-training employs a technique akin to Latent Diffusion, relying on a perceptual compression framework to encode images into latents for a computationally efficient diffusion process. Extensive pre-training utilizes sizable Chinese text-image pairs, gradually improving the model through an incremental training regimen that begins with foundational knowledge acquisition from a smaller corpus and then advances to training with a large-scale, diverse dataset, culminating with a highly curated selection for refinement.

Multimodal Instruction Tuning

This phase is fundamental for adapting UNIMO-G to render images from multimodal inputs. By enhancing the cross-attention mechanism, the model is now capable of capitalizing on the visual characteristics of multimodal input, thus rendering images that correlate accurately with the specifics of the visual content in the prompts. The introduction of a novel loss component ensures the generated noisy latent space adequately emphasizes image regions correlating with the represented objects, thereby enforcing a visual congruence between the input prompt's visual elements and the synthesized image.

Evaluation and Results

UNIMO-G's proficiency extends to both text-to-image and zero-shot subject-driven image generation, as substantiated by evaluations on the MS-COCO and MultiBench datasets. Its performance is notably superior in scenarios involving subject-driven synthesis, proving its ability to process complex prompts involving multiple entities while maintaining both visual fidelity and textual relevance. Comparative human evaluation further underscores UNIMO-G's enhanced image quality against its contemporaries, spotlighting its strength in multi-entity scenarios.

Conclusion

In synthesis, UNIMO-G sets a benchmark in controlled image generation, adeptly managing the intricacies of multimodal prompts and demonstrating significant improvements over other models in terms of fidelity and relevance. The successful marriage of MLLM for encoding with a diffusion network for generation paves the way for more sophisticated text-to-image models capable of handling nuanced and composite visual prompts.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown