Vision-based 3D occupancy prediction in autonomous driving: a review and outlook (2405.02595v2)

Abstract: In recent years, autonomous driving has garnered escalating attention for its potential to relieve drivers' burdens and improve driving safety. Vision-based 3D occupancy prediction, which predicts the spatial occupancy status and semantics of 3D voxel grids around the autonomous vehicle from image inputs, is an emerging perception task suitable for cost-effective perception system of autonomous driving. Although numerous studies have demonstrated the greater advantages of 3D occupancy prediction over object-centric perception tasks, there is still a lack of a dedicated review focusing on this rapidly developing field. In this paper, we first introduce the background of vision-based 3D occupancy prediction and discuss the challenges in this task. Secondly, we conduct a comprehensive survey of the progress in vision-based 3D occupancy prediction from three aspects: feature enhancement, deployment friendliness and label efficiency, and provide an in-depth analysis of the potentials and challenges of each category of methods. Finally, we present a summary of prevailing research trends and propose some inspiring future outlooks. To provide a valuable reference for researchers, a regularly updated collection of related papers, datasets, and codes is organized at https://github.com/zya3d/Awesome-3D-Occupancy-Prediction.

• The paper provides an in-depth review of vision-based 3D occupancy prediction by categorizing methods into feature enhancement, deployment-friendly, and label-efficient approaches.
• It demonstrates that multi-view and voxel-based techniques yield competitive mIoU scores and reduced computational costs, enabling more effective real-time performance.
• The paper outlines future directions that advocate for unified frameworks and self-supervised strategies to advance dynamic, 4D perception in autonomous driving.

Vision-based 3D Occupancy Prediction in Autonomous Driving: Insights and Developments

Vision-based 3D occupancy prediction has emerged as a pivotal perception task in autonomous driving aimed at predicting the spatial occupancy status and semantics of 3D voxel grids from image inputs. This paper provides an in-depth review of the progress, challenges, and future directions in the field. As an increasingly promising approach, vision-based 3D occupancy offers fine-grained spatial representation, critical for autonomous navigation and the detection of undefined, long-tail obstacles.

The paper categorically analyzes existing methods from three perspectives: feature enhancement, deployment friendliness, and label efficiency. This structured approach not only lays out a comprehensive landscape of the current methodologies but also presents a robust framework for comparing them. Each category addresses specific challenges; for instance, feature enhancement focuses on improving 3D feature extraction from 2D images, thereby enhancing the semantic and spatial accuracy of predictions.

Feature Enhancement Methods

In the domain of feature enhancement, BEV, TPV, and voxel-based representations constitute the central axis for development. Methods such as TPVFormer leverage multi-view representations to effectively bridge the gap between 2D and 3D spaces, while strategies like VoxFormer utilize learned queries for improved voxelization. Strong numerical results indicate that these methods yield significant improvements in semantic segmentation accuracy, as evidenced by mIoU scores reaching competitive levels on benchmark datasets like Occ3D-nuScenes.

Deployment-friendly Methods

Given the high computational costs of 3D processes, deployment-friendly methods, such as FlashOcc, aim to minimize memory and latency while maintaining performance. The application of perspective decomposition techniques and the coarse-to-fine learning paradigms provide promising avenues for efficient 3D occupancy computation. The focus on reducing computational costs without significant losses in accuracy is crucial for real-time applications in autonomous vehicles.

Label-efficient Methods

The paper also explores label-efficient methods that eschew dense annotations, which are often costly and impractical at scale. Techniques like UniOcc that employ neural rendering significantly reduce annotation dependency by leveraging 2D supervision. This initiative aligns with future goals of achieving robust self-supervised frameworks in 3D occupancy prediction.

Implications and Future Outlook

The authors underscore the significance of continuous development in generating more realistic driving scenarios through synthetic data and advancements in world model frameworks. The exploration of multi-agent collaborative perception paves the way for more holistic environmental awareness by integrating features from multiple vehicles.

Nevertheless, the paper identifies an overarching need for a unified framework encompassing feature enhancements, efficiency, and minimal labels, suggesting that future work should focus on amalgamating these aspects. Furthermore, the adaptation of open-vocabulary and dynamic (4D) perception capabilities stand out as critical areas, promising to enhance the depth and breadth of the autonomous vehicle's understanding of real-world environments.

The proposed outlook offers a coherent path forward, urging researchers to explore innovative solutions that intertwine theoretical advancements with practical applications. As the field progresses, vision-based 3D occupancy prediction remains an essential component in realizing the full potential of autonomous driving technologies.