VisMin: Visual Minimal-Change Understanding (2407.16772v2)

Abstract: Fine-grained understanding of objects, attributes, and relationships between objects is crucial for visual-LLMs (VLMs). Existing benchmarks primarily focus on evaluating VLMs' capability to distinguish between two very similar captions given an image. In this paper, we introduce a new, challenging benchmark termed Visual Minimal-Change Understanding (VisMin), which requires models to predict the correct image-caption match given two images and two captions. The image pair and caption pair contain minimal changes, i.e., only one aspect changes at a time from among the following: object, attribute, count, and spatial relation. These changes test the models' understanding of objects, attributes (such as color, material, shape), counts, and spatial relationships between objects. We built an automatic framework using LLMs and diffusion models, followed by a rigorous 4-step verification process by human annotators. Empirical experiments reveal that current VLMs exhibit notable deficiencies in understanding spatial relationships and counting abilities. We also generate a large-scale training dataset to finetune CLIP and Idefics2, showing significant improvements in fine-grained understanding across benchmarks and in CLIP's general image-text alignment. We release all resources, including the benchmark, training data, and finetuned model checkpoints, at https://vismin.net/.

• The paper presents VisMin, a benchmark evaluating VLMs' ability to discern minuscule visual differences through automated synthesis and rigorous human verification.
• It reveals that while models like CLIP excel at object recognition, they struggle with spatial relations and counting in complex scenes.
• Fine-tuning with a 64K-example minimal-change dataset considerably improves performance, highlighting new pathways for enhancing VLM capabilities.

Overview of "VisMin: Visual Minimal-Change Understanding"

The paper "VisMin: Visual Minimal-Change Understanding" introduces a novel benchmark designed to probe the fine-grained understanding of Visual-LLMs (VLMs). Unlike conventional benchmarks that assess model performance by evaluating differences between similar captions given one image, VisMin evaluates the ability to discern minimal changes between two nearly identical images when provided with corresponding captions. This focus shifts to distinguishing between minor changes in object attributes, counts, and spatial relationships — essential skills for advanced VLMs.

Benchmark Construction and Methodology

The VisMin benchmark is curated through a sophisticated combination of automated tools and rigorous human verification steps:

1. Minimal-Change Pairs Synthesis: Using LLMs and diffusion models, the authors generated minimal-change pairs for testing. This involved creating image-caption pairs that differ by a single aspect (object, attribute, count, spatial relation) without affecting other image components.
2. Automated Filtering: This phase relied on a Visual Question Answering (VQA) system to ensure the generated images and captions were plausible and faithfully depicted the intended changes. The VQA system checked consistency by posing questions about the edited captions and verifying the coherence of responses based on the edited images.
3. Human Verification: To further ensure data quality, human annotators conducted a four-step verification process involving checks for naturalness, the sensical nature of captions, and the faithful representation of minimal changes. This step was crucial in maintaining the robustness of the VisMin benchmark.

This meticulous approach allowed the authors to create a benchmark set composed of complex real-world images, mainly sourced from the COCO dataset, enriched by synthetically generated minimal-change pairs that pose significant challenges to current VLM capabilities.

Key Findings and Insights

Empirical evaluations on the VisMin benchmark exposed notable deficiencies in existing VLMs, particularly in understanding spatial relationships and counting capabilities. For instance, foundational VLMs like CLIP and multimodal LLMs (MLLMs) such as Idefics2 showed robust performance in object and attribute understanding but struggled significantly with spatial relations, often performing below random chance.

Key findings include:

• Current VLM Performance: Models like CLIP exhibited superior performance in object recognition tasks but lagged in more complex scenarios involving spatial relations and counting.
• Relative Performance: Foundational models generally outperformed MLLMs, attributed to the latter's lack of training with multiple images and simple vertical concatenation methods that did not sufficiently parse visual signals for alignment.
• Comparison Across Models: Within the studied models, GPT-4 and Gemini demonstrated strong capabilities, underlining the potential of closed-source models in these nuanced tasks.

Enhancing Fine-Grained Understanding Through Fine-Tuning

To address the identified gaps in VLM performance, the authors generated a large-scale minimal-change dataset for additional fine-tuning of VLMs. This dataset, consisting of over 64,000 examples, was leveraged to fine-tune CLIP and Idefics2:

• Fine-Tuning CLIP: The fine-tuned CLIP (termed VisMin-CLIP) demonstrated marked improvements across most benchmark tasks, including substantial enhancements in multi-image understanding benchmarks such as Winoground and MMVP. This highlights the efficacy of minimal-change training data in bolstering fine-grained visual understanding.
• Fine-Tuning Idefics2: Fine-tuning Idefics2 using the VisMin dataset also resulted in significant performance boosts, especially in spatial relations, showcasing the transformative potential of this fine-tuning method.

Implications and Future Directions

The introduction of VisMin and the accompanying datasets has several critical implications for the field of AI and VLM development:

• Benchmarking and Evaluation: VisMin sets a new standard for evaluating the nuanced understanding capabilities of VLMs, ensuring that future models are rigorously tested for their ability to discern minimal changes in complex scenes.
• Model Training and Fine-Tuning: The demonstrated improvements from fine-tuning with minimal-change data indicate that future VLMs can benefit significantly from incorporating such data into their training regimens.
• Advancements in AI Research: Enhanced model capabilities in understanding fine-grained visual differences have far-reaching applications, from improving AI-driven content moderation to advancing autonomous systems that need to navigate dynamic environments.

In conclusion, the VisMin benchmark, with its emphasis on minimal visual changes, provides a crucial tool for advancing fine-grained visual understanding in VLMs. The benchmark, coupled with the substantial improvements seen in fine-tuning applications, sets the stage for future research aimed at overcoming current model limitations, particularly in spatial reasoning and counting, and fostering more capable AI systems.