iVideoGPT: Interactive VideoGPTs are Scalable World Models (2405.15223v3)

Abstract: World models empower model-based agents to interactively explore, reason, and plan within imagined environments for real-world decision-making. However, the high demand for interactivity poses challenges in harnessing recent advancements in video generative models for developing world models at scale. This work introduces Interactive VideoGPT (iVideoGPT), a scalable autoregressive transformer framework that integrates multimodal signals--visual observations, actions, and rewards--into a sequence of tokens, facilitating an interactive experience of agents via next-token prediction. iVideoGPT features a novel compressive tokenization technique that efficiently discretizes high-dimensional visual observations. Leveraging its scalable architecture, we are able to pre-train iVideoGPT on millions of human and robotic manipulation trajectories, establishing a versatile foundation that is adaptable to serve as interactive world models for a wide range of downstream tasks. These include action-conditioned video prediction, visual planning, and model-based reinforcement learning, where iVideoGPT achieves competitive performance compared with state-of-the-art methods. Our work advances the development of interactive general world models, bridging the gap between generative video models and practical model-based reinforcement learning applications. Code and pre-trained models are available at https://thuml.github.io/iVideoGPT.

• The paper introduces a compressive tokenization technique that reduces visual token lengths and enhances video prediction efficiency.
• The paper leverages a scalable autoregressive transformer pre-trained on over one million trajectories to integrate multimodal sensor inputs.
• The paper demonstrates improved performance in model-based reinforcement learning and visual planning on BAIR, RoboNet, and Meta-World benchmarks.

Overview of Interactive VideoGPT: Bridging Generative Video Models and Interactive World Models

The paper introduces Interactive VideoGPT (iVideoGPT), a scalable autoregressive transformer framework designed to address the challenges in utilizing generative video models for interactive world models. The primary contributions of the work are situated at the intersection of video generation, model-based reinforcement learning (MBRL), and multimodal integration of agents' sensory inputs. This framework facilitates an interactive experience for agents through next-token prediction, allowing them to imagine, reason, and plan within high-dimensional environments.

Key Contributions

1. Compressive Tokenization Technique: The proposed iVideoGPT employs a novel compressive tokenization mechanism, which discretizes complex visual observations into a manageable sequence of tokens. By conditionally encoding visual frames based on temporal context, this approach achieves a significant reduction in token sequence length, leading to more efficient training and generation processes.
2. Scalable Autoregressive Transformer: iVideoGPT leverages an autoregressive transformer architecture similar to models used in LLMs. This allows for flexible handling of various multimodal signals, including visual frames, actions, and rewards. The architecture's scalability enables pre-training on millions of trajectories, creating a broad foundation for interactive world models that can be adapted to a wide array of downstream tasks.
3. Comprehensive Pre-Training: The iVideoGPT framework is pre-trained on a diverse dataset of human and robotic manipulation trajectories, totaling over one million sequences. This extensive pre-training equips the model with generalizable knowledge about physical interactions, which can be fine-tuned for specific tasks such as action-conditioned video prediction, visual planning, and visual model-based reinforcement learning.

Numerical Results and Experimental Evaluation

The experimental results presented in the paper showcase the effectiveness of iVideoGPT across various metrics and datasets:

• Video Prediction:

iVideoGPT provides competitive results on established benchmarks like the BAIR robot pushing and RoboNet datasets, achieving significant improvements in FVD, PSNR, SSIM, and LPIPS metrics. The model's ability to integrate actions into video prediction workflows enhances its interactivity and performance, particularly notable under action-conditioned scenarios.

• Visual Planning:

The model's performance on the VP$^2$ benchmark, which evaluates video prediction models for visual model-predictive control, underscores its robustness. iVideoGPT outperforms many baselines in specific Robosuite and RoboDesk tasks, demonstrating its applicability in control tasks where accurate and realistic predictions are crucial.

• Visual Model-Based Reinforcement Learning:

The visual MBRL experiments on Meta-World tasks highlight the remarkable sample efficiency of iVideoGPT-enabled algorithms. The model-based approach, leveraging iVideoGPT for synthetic rollouts, outperforms model-free alternatives and achieves comparable results to state-of-the-art latent imagination methods like DreamerV3.

Practical and Theoretical Implications

The practical implications of this work are substantial. By enabling a more efficient and scalable approach to building interactive world models, iVideoGPT represents a significant step forward in the application of generative video models to real-world decision-making tasks:

• Model-Based Learning Efficiency:

The ability to pre-train on large, diverse datasets and fine-tune efficiently for specific tasks can drastically reduce the requirement for extensive data collection in new environments. This is particularly advantageous in robotics and autonomous systems where real-world trials can be costly and time-consuming.

• Generalization and Adaptation:

iVideoGPT's ability to generalize from human manipulation datasets to diverse robotic contexts highlights the model's potential for robust transfer learning. This capability is crucial for developing versatile agents that perform consistently across various environments and tasks.

Future Directions

The promising results from iVideoGPT pave the way for several future research directions:

• Scaling and Diverse Applications:

Further scaling of the architecture and pre-training on more diverse, Internet-scale datasets could enhance the model's generalizability and performance. This would be especially relevant for applications in complex, real-world scenarios such as autonomous driving and general-purpose robotics.

• Enhancements in Tokenization:

Investigating alternative tokenization strategies that maintain high fidelity while further reducing computational overhead could lead to even more efficient training and inference processes. Improvements in this area might also enhance the model's ability to handle higher-resolution inputs and more complex scenarios.

• Integration with Other Modalities:

Extending the multimodal capabilities of iVideoGPT to include additional sensory inputs such as audio and haptic feedback could broaden the range of applications and improve the model's performance in environments where multisensory integration is critical.

Overall, iVideoGPT represents a significant advancement in combining the strengths of autoregressive transformers, generative video models, and interactive world modeling, providing a robust foundation for future research and practical applications in MBRL and beyond.