Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs (2404.07449v1)

Abstract: Integration of LLMs into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks, particularly for visual question answering (VQA). However, existing V-LLMs (e.g. BLIP-2, LLaVA) demonstrate weak spatial reasoning and localization awareness. Despite generating highly descriptive and elaborate textual answers, these models fail at simple tasks like distinguishing a left vs right location. In this work, we explore how image-space coordinate based instruction fine-tuning objectives could inject spatial awareness into V-LLMs. We discover optimal coordinate representations, data-efficient instruction fine-tuning objectives, and pseudo-data generation strategies that lead to improved spatial awareness in V-LLMs. Additionally, our resulting model improves VQA across image and video domains, reduces undesired hallucination, and generates better contextual object descriptions. Experiments across 5 vision-language tasks involving 14 different datasets establish the clear performance improvements achieved by our proposed framework.

• The paper introduces LocVLM, which integrates image-coordinate objectives to improve spatial reasoning in visual-LLMs.
• It employs novel methodologies like LocPred, NegPred, and RevLoc to significantly boost performance in both image and video VQA tasks.
• The framework achieves state-of-the-art results on benchmarks such as GQA and VQAv2 while reducing object hallucination.

Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs

This paper explores improvements in spatial reasoning for visual-LLMs (V-LLMs), which integrates LLMs into visual tasks like visual question answering (VQA). Although current V-LLMs like BLIP-2 and LLaVA perform well in vision-language tasks, they show a significant limitation in spatial reasoning, such as differentiating left from right. This work introduces methods to enhance spatial awareness using image-space coordinate-based objectives. The proposed framework demonstrates improved performance on spatial awareness tasks and VQA, minimizing hallucination and enhancing contextual object descriptions.

Introduction

The authors identify a crucial gap in existing V-LLMs regarding spatial reasoning. Despite their advancement in answering complex queries about image content, these models struggle with simple spatial distinctions. The researchers propose integrating spatial localization by embedding image-space coordinates within language prompts, aiming to enhance V-LLMs' spatial reasoning capabilities and performance in tasks requiring spatial awareness.

Figure 1: The model's ability to use contextual region descriptions enhances spatial awareness in visual question answering.

Methodology

The methodology involves instruction fine-tuning objectives that explicitly incorporate spatial coordinates. The authors detail three main objectives: Location Prediction (LocPred), Negative Prediction (NegPred), and Reverse-Location Prediction (RevLoc). These are designed to embed spatial reasoning into V-LLMs, allowing them to process and generate coordinate-based information effectively.

The paper proposes a framework named LocVLM, which applies visual instruction tuning similar to LLaVA. It uses a visual encoder, an adapter layer, and an LLM simultaneously, enabling the V-LLM to extract meaningful coordinates and descriptions from images. The authors present a data-efficient pseudo-data generation strategy, leveraging pre-trained V-LLMs to enhance the training process without additional human annotations.

Figure 2: Model architecture inspired by LLaVa with modifications to incorporate spatial reasoning tasks.

Experimental Results

The framework's effectiveness is tested across multiple benchmarks. Spatial reasoning evaluations reveal near random performance from existing V-LLMs, while LocVLM exhibits marked improvements. The framework achieves state-of-the-art results in image and video VQA tasks and successfully reduces object hallucination.

1. Spatial Reasoning Results: LocVLM significantly outperforms BLIP-2 and LLaVA in distinguishing spatial relationships, showcasing enhanced spatial reasoning capabilities.
2. Image VQA: Tested on GQA and VQAv2, LocVLM shows clear improvements, highlighting the impact of integrated spatial localization on complex reasoning tasks.
3. Video Domain Operation: Adapted for video analysis, LocVLM further demonstrates superior performance in video VQA tasks like ActivityNet-QA, confirming its robust applicability across dynamic inputs.
4. Object Hallucination: LocVLM showcases reduced hallucination incidents, a prevalent issue in current V-LLMs, as evidenced by improved accuracy across proposed and existing datasets.

   Figure 3: Example images illustrating enhanced spatial reasoning capabilities in toy experiments.

Future Work and Implications

This research opens avenues for further exploration in spatial reasoning within V-LLMs, suggesting an integration framework that does not rely solely on increased training data but on strategic objective alignment. Future work could enhance temporal modeling for dynamic video understanding, embedding time-coordinate reasoning within the spatial framework.

The practical implications are profound, offering improved interaction in automated systems requiring spatial decision-making, such as robotics and autonomous navigation, where contextual awareness and accurate spatial interpretation are crucial.

Conclusion

The proposed LocVLM framework substantially enhances spatial reasoning in V-LLMs, highlighting its potential in bolstering VQA tasks. By integrating location-specific fine-tuning objectives, the framework overcomes inherent limitations in spatial reasoning, setting a new benchmark for V-LLMs in handling complex spatial tasks. These findings suggest a viable path for evolving AI towards more holistic visual and cognitive functionalities.

Figure 4: Demonstrating the model's ability to provide representative descriptions for specific queried image regions.