LLaRA: Supercharging Robot Learning Data for Vision-Language Policy (2406.20095v3)

Abstract: Vision LLMs (VLMs) have recently been leveraged to generate robotic actions, forming Vision-Language-Action (VLA) models. However, directly adapting a pretrained VLM for robotic control remains challenging, particularly when constrained by a limited number of robot demonstrations. In this work, we introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations and enables an efficient transfer of a pretrained VLM into a powerful VLA, motivated by the success of visual instruction tuning in Computer Vision. First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets, aligning robotic actions with image pixel coordinates. Further, we enhance this dataset in a self-supervised manner by defining six auxiliary tasks, without requiring any additional action annotations. We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control. Through experiments across multiple simulated and real-world tasks, we demonstrate that LLaRA achieves state-of-the-art performance while preserving the generalization capabilities of LLMs. The code, datasets, and pretrained models are available at https://github.com/LostXine/LLaRA.

• The paper presents a novel approach that formulates robot tasks as natural language instruction-response pairs for improved policy learning.
• It introduces an automated pipeline to generate diverse auxiliary data, enhancing both simulation and real-world robotic performance.
• Experiments show that leveraging vision-language models yields substantial improvements over traditional behavior cloning methods.

LLaRA: Supercharging Robot Learning Data for Vision-Language Policy

The paper "LLaRA: Supercharging Robot Learning Data for Vision-Language Policy" introduces a framework called LLaRA (Large Language and Robotics Assistant) that employs Vision LLMs (VLMs) to formulate robot action policies as conversation pairs. This methodology is posited to offer state-of-the-art performance in robot action policy by leveraging auxiliary data that complements the core policy learning data.

Core Contributions and Methodology

LLaRA capitalizes on the capabilities of LLMs and VLMs, such as those found in models like GPT-4, LLaVA, and their successors, which inherently possess extensive knowledge and advanced reasoning skills. The paper suggests that by fine-tuning these models with visually-grounded language data, one can formulate effective robot action policies.

Several key contributions are highlighted:

1. Formulating Robot Tasks as Instruction-Response Pairs: The framework effectively converts conventional robot manipulation tasks into formatted instruction-response pairs. These pairs are designed using natural language, which enables leveraging the linguistic understanding and reasoning capabilities of LLMs.
2. Automated Pipeline for Data Generation: A significant innovation presented in the paper is an automated pipeline for generating diverse and high-quality robotics instruction data from existing behavior cloning datasets. This pipeline ensures the creation of an extensive and varied dataset that is essential for robust model training.
3. Introduction of Auxiliary Data: Beyond transforming existing data, the framework also synthesizes additional auxiliary datasets that support policy learning in a self-supervised manner. This includes tasks like object localization, detection, future prediction, spatial relations, and temporal relations.

Experimental Validation

The paper reports extensive experiments conducted in both simulated environments (e.g., VIMA-Bench) and real-world settings which demonstrate the effectiveness and robustness of the proposed framework. Several versions of the dataset and model, including inBC, $ + Aux, and their combinations, show that LLaRA can outperform traditional behavior cloning methods, especially when auxiliary data is utilized.

Performance and Numerical Results

The numerical results presented emphasize substantial improvements in performance. For instance, on the VIMA-0.8k dataset, the inBC with auxiliary data configuration achieved marked improvements across various levels of difficulty when compared to other baselines. Particularly notable is the scalability of the model, with performance metrics positively correlating with the increase in training data size.

For the real-world experiments, even the zero-shot generalization performance of LLaRA outstripped baseline models. The framework's ability to adapt to real-world scenarios through joint training and fine-tuning phases illuminates its practicality and robustness in versatile robotic applications.

Theoretical Implications and Future Directions

The implications of this research are twofold:

1. Enhanced Scene Understanding: By training on auxiliary tasks such as object localization and spatial relations, the model develops a nuanced understanding of the scene, which is crucial for effective robot action policy.
2. Scalable Data Generation: The automated pipeline for generating instruction-tuning data underscores an important advancement towards efficient and scalable data curation in robotics.

Future developments in AI, particularly in robot learning, could greatly benefit from the integration of more sophisticated auxiliary tasks and cross-modal synergies. Expanding the dataset generation pipeline to handle more complex, multi-object environments or incorporating additional sensory data (beyond visual inputs) could further enhance model performance.

Conclusion

In conclusion, the LLaRA framework offers a significant stride in leveraging VLM-based models for robot learning tasks. The introduction of automated data generation and auxiliary datasets presents a comprehensive solution for enhancing robot learning through vision-language integration. This work sets a promising course for future exploration in the utilization of LLMs and VLMs in robotics, contributing valuable insights into the development of more intelligent and adaptive robotic systems.