SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2303.09551v2)

Abstract: 3D scene understanding plays a vital role in vision-based autonomous driving. While most existing methods focus on 3D object detection, they have difficulty describing real-world objects of arbitrary shapes and infinite classes. Towards a more comprehensive perception of a 3D scene, in this paper, we propose a SurroundOcc method to predict the 3D occupancy with multi-camera images. We first extract multi-scale features for each image and adopt spatial 2D-3D attention to lift them to the 3D volume space. Then we apply 3D convolutions to progressively upsample the volume features and impose supervision on multiple levels. To obtain dense occupancy prediction, we design a pipeline to generate dense occupancy ground truth without expansive occupancy annotations. Specifically, we fuse multi-frame LiDAR scans of dynamic objects and static scenes separately. Then we adopt Poisson Reconstruction to fill the holes and voxelize the mesh to get dense occupancy labels. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the superiority of our method. Code and dataset are available at https://github.com/weiyithu/SurroundOcc

• The paper introduces SurroundOcc, a novel multi-camera 3D occupancy framework that outperforms state-of-the-art methods with improved IoU and mIoU on nuScenes and SemanticKITTI.
• It employs a 3D U-Net-like architecture with spatial 2D-3D attention and decayed weighted losses to refine feature upsampling and fusion across multiple scales.
• The approach robustly infers occluded regions and adapts to diverse driving conditions, paving the way for real-time autonomous driving applications.

Multi-Camera 3D Occupancy Prediction for Autonomous Driving

The paper "SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving" presents a novel approach to enhance 3D scene understanding in vision-based autonomous driving systems. The authors propose "SurroundOcc," an innovative methodology for predicting 3D occupancy using multi-camera images. This work addresses the limitations of existing 3D object detection methods that struggle to accommodate objects with arbitrary shapes and a vast range of classes.

Methodology Overview

SurroundOcc introduces a comprehensive pipeline composed of several key steps:

1. Feature Extraction and Lifting: The method begins by extracting multi-scale features from each camera image. Spatial 2D-3D attention mechanisms are employed to lift these features into a 3D volumetric space, utilizing a 3D convolutional network to progressively upsample and refine these features across multiple levels.
2. Supervision and Consistency: To address the scarcity of dense occupancy annotations, the authors propose a pipeline that circumvents this limitation by employing multi-frame LiDAR scans combined with Poisson Reconstruction, allowing for dense occupancy prediction. This process effectively fills gaps in the visual data, converting them into a dense voxelized mesh that serves as a supervisory label.
3. Multi-Scale Network Architecture: SurroundOcc employs a 3D U-Net-like structure, with levels for feature upsampling and fusion, supported by decayed weighted losses for each level to propagate supervisory signals throughout the network.

Experimental Evaluation

The authors validate their model on extensive datasets including nuScenes and SemanticKITTI. SurroundOcc demonstrates superior performance in both quantitative measures and qualitative visualizations over state-of-the-art methods in 3D semantic occupancy prediction. Specifically, the model excels in outdoor environments, proving its robustness and effectiveness even in challenging scenarios such as rainy or night-time conditions. Key results include significant improvements in IoU and mIoU metrics, underscoring the model's capability to generalize across various driving environments.

Implications and Future Directions

This work opens several avenues for further research and practical applications:

• Enhanced 3D Scene Representation:

The adoption of 3D occupancy as a key representation format offers fine-grained scene modeling, which is crucial for downstream tasks like semantic segmentation and scene flow estimation.

• Robust to Occlusions:

Unlike depth map approaches, which are limited to visible surfaces, SurroundOcc can infer occluded regions, providing a more comprehensive scene understanding.

• Potential for Real-Time Applications:

The efficiency of the proposed method suggests feasibility for real-time applications within autonomous driving systems, enhancing situational awareness.

Conclusion

SurroundOcc introduces a highly effective approach to multi-camera 3D occupancy prediction, characterized by its use of advanced spatial attention mechanisms and robust supervisory techniques. The method represents a significant step toward achieving refined and dense 3D scene understanding in autonomous vehicles. Future developments may build upon this foundation to explore self-supervised learning strategies or extend the model's application to dynamic occupancy flow scenarios.