Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models (2312.02949v1)

Published 5 Dec 2023 in cs.CV

Abstract: With the recent significant advancements in large multi-modal models (LMMs), the importance of their grounding capability in visual chat is increasingly recognized. Despite recent efforts to enable LMMs to support grounding, their capabilities for grounding and chat are usually separate, and their chat performance drops dramatically when asked to ground. The problem is the lack of a dataset for grounded visual chat (GVC). Existing grounding datasets only contain short captions. To address this issue, we have created GVC data that allows for the combination of grounding and chat capabilities. To better evaluate the GVC capabilities, we have introduced a benchmark called Grounding-Bench. Additionally, we have proposed a model design that can support GVC and various types of visual prompts by connecting segmentation models with LLMs. Experimental results demonstrate that our model outperforms other LMMs on Grounding-Bench. Furthermore, our model achieves competitive performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K Entities. Our code will be released at https://github.com/UX-Decoder/LLaVA-Grounding .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Position-enhanced visual instruction tuning for multimodal large language models, 2023a.
  2. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv:2310.09478, 2023b.
  3. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023c.
  4. Transvg: End-to-end visual grounding with transformers, 2022.
  5. A unified mutual supervision framework for referring expression segmentation and generation, 2022.
  6. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918, 2021.
  7. Mdetr – modulated detection for end-to-end multi-modal understanding, 2021.
  8. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014.
  9. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016.
  10. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692, 2023.
  11. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023a.
  12. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023b.
  13. Grounded language-image pre-training. In CVPR, 2022.
  14. Microsoft COCO: Common objects in context. In ECCV, 2014.
  15. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23592–23601, 2023a.
  16. Improved baselines with visual instruction tuning, 2023b.
  17. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023c.
  18. Polyformer: Referring image segmentation as sequential polygon generation, 2023d.
  19. Llava-plus: Learning to use tools for creating multimodal agents, 2023e.
  20. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023f.
  21. Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond language, 2023g.
  22. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  23. Towards language-guided visual recognition via dynamic convolutions. International Journal of Computer Vision, pages 1–19, 2023.
  24. OpenAI. Gpt-4 technical report, 2023a.
  25. OpenAI. Gpt-4v(ision) system card. https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023b.
  26. OpenAI. Gpt-4 technical report, 2023c.
  27. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023.
  28. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015.
  29. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.
  30. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  31. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, 2023a.
  32. Cogvlm: Visual expert for pretrained language models. 2023b.
  33. Universal instance perception as object discovery and retrieval. In CVPR, 2023.
  34. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023.
  35. Unitab: Unifying text and box outputs for grounded vision-language modeling, 2022.
  36. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection, 2022.
  37. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment, 2023.
  38. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499, 2023.
  39. Ferret: Refer and ground anything anywhere at any granularity, 2023.
  40. Modeling context in referring expressions, 2016.
  41. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022a.
  42. Glipv2: Unifying localization and vision-language understanding. arXiv preprint arXiv:2206.05836, 2022b.
  43. A simple framework for open-vocabulary segmentation and detection. arXiv preprint arXiv:2303.08131, 2023a.
  44. Llama-adapter: Efficient fine-tuning of language models with zero-init attention, 2023b.
  45. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023c.
  46. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581, 2023.
  47. Seqtr: A simple yet universal network for visual grounding. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 598–615. Springer, 2022.
  48. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
Citations (47)
View on Semantic Scholar

Summary

  • The paper develops a high-quality 150K-instance GVC dataset that bridges the gap between visual grounding and conversational tasks.
  • The paper presents LLaVA-G, a novel model that integrates object- and pixel-level grounding with chat capabilities for diverse visual prompts.
  • The paper establishes the Grounding-Bench benchmark, demonstrating LLaVA-G's superior performance over existing models on accuracy and conversational grounding.

Insightful Overview of "LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models"

The paper "LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models" presents a significant advancement in the field of large multimodal models (LMMs) by addressing the challenges of grounded visual chat (GVC). The authors have meticulously identified and addressed two primary limitations in existing LMMs: the lack of datasets for grounded visual chat and the distinct separation between grounding and chat capabilities in contemporary models. Traditional models have struggled to effectively integrate these functionalities due to the absence of comprehensive data and suboptimal model designs.

Key Contributions

The paper details several key contributions that aim to revolutionize GVC:

  1. Creation of GVC Dataset: The authors have developed a high-quality GVC dataset that bridges the gap between visual grounding and conversational capabilities. They have leveraged human-labeled object detection data and enhanced it with GPT-4 to produce quality annotations, resulting in a dataset with 150K grounded visual chat instances.
  2. Novel Model Architecture: The paper introduces a novel model architecture named LLaVA-Grounding (LLaVA-G). This architecture seamlessly connects a large multimodal model with a grounding model, thereby enabling it to handle both object and pixel-level grounding. The LLaVA-G model can manage various types of visual prompts, such as marks, clicks, boxes, and scribbles.
  3. Introduction of Grounding-Bench Benchmark: A new benchmark, Grounding-Bench, has been established to evaluate models on their GVC capabilities. The benchmark tests models on their ability to perform in grounded conversations, detailed descriptions, and complex reasoning, providing a robust framework for evaluating GVC models.
  4. Strong Experimental Results: The experimental results presented in the paper demonstrate that LLaVA-G outperforms existing LMMs on the Grounding-Bench. The model shows competitive performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K Entities.

Experimental Evaluation

The experimental results are compelling, showcasing LLaVA-G's superior performance over other open-source models on the Grounding-Bench. LLaVA-G achieves a balance between grounding accuracy and chat performance, which has been a challenging feat for previous models. The model's design allows it to outperform on tasks involving both conversation and grounding without compromising on either. The paper emphasizes numerical results, leveraging metrics such as F1F_1 scores to substantiate the performance claims.

Implications and Future Directions

The implications of this research are far-reaching both in theory and practical application. The ability of a model to engage in contextually grounded chat has substantial potential in various real-world applications, such as interactive agents in customer support, educational tools, and enhanced accessibility features. Theoretical implications lie in the improved understanding and integration of multimodal data processing, particularly in the context of artificial intelligence and machine learning.

Future research could explore further expanding the semantic scope of the models to support open-vocabulary settings or fine-tuning the models for specific industry applications. Additionally, exploring the integration of other modalities and expanding the dataset annotation methodologies could provide pathways for future exploration.

In conclusion, "LLaVA-Grounding" effectively addresses the shortcomings of its predecessors and sets a robust framework for future advancements in the field of grounded visual chat. It stands as a testament to the collaborative potential of visual and textual modalities within large-scale models, paving the way for more sophisticated AI systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - UX-Decoder/LLaVA-Grounding (284 stars)