MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection

Abstract: Monocular 3D object detection has long been a challenging task in autonomous driving. Most existing methods follow conventional 2D detectors to first localize object centers, and then predict 3D attributes by neighboring features. However, only using local visual features is insufficient to understand the scene-level 3D spatial structures and ignores the long-range inter-object depth relations. In this paper, we introduce the first DETR framework for Monocular DEtection with a depth-guided TRansformer, named MonoDETR. We modify the vanilla transformer to be depth-aware and guide the whole detection process by contextual depth cues. Specifically, concurrent to the visual encoder that captures object appearances, we introduce to predict a foreground depth map, and specialize a depth encoder to extract non-local depth embeddings. Then, we formulate 3D object candidates as learnable queries and propose a depth-guided decoder to conduct object-scene depth interactions. In this way, each object query estimates its 3D attributes adaptively from the depth-guided regions on the image and is no longer constrained to local visual features. On KITTI benchmark with monocular images as input, MonoDETR achieves state-of-the-art performance and requires no extra dense depth annotations. Besides, our depth-guided modules can also be plug-and-play to enhance multi-view 3D object detectors on nuScenes dataset, demonstrating our superior generalization capacity. Code is available at https://github.com/ZrrSkywalker/MonoDETR.

• The paper introduces a novel depth-guided transformer framework that enhances monocular 3D object detection by integrating both visual and object-centric depth cues.
• It leverages parallel visual and depth encoders paired with a depth-guided decoder featuring a cross-attention layer for robust scene-level feature aggregation.
• The approach achieves state-of-the-art KITTI benchmark results, with improvements of +2.53%, +1.08%, and +0.85% in AP3D across easy, moderate, and hard settings.

Depth-guided Transformer for Monocular 3D Object Detection

The task of 3D object detection from monocular images in autonomous driving remains notably challenging. The paper under discussion introduces MonoDETR, a novel approach utilizing a depth-guided transformer framework for monocular 3D object detection. This approach addresses the limitations inherent in existing methods, which typically rely on local visual features for depth estimation, often insufficiently capturing broad spatial structures and inter-object depth relationships.

MonoDETR integrates a depth-aware dynamic into the Detection Transformer (DETR) framework. Traditional DETR workflows focus predominantly on visual features. In contrast, MonoDETR introduces depth awareness by enhancing the process with contextual depth cues. It comprises three main components: a visual encoder, a depth encoder, and a depth-guided decoder. This configuration allows for superior adaptation in estimating 3D attributes by leveraging depth-associated information from across the image rather than relying solely on localized data.

Technical Contributions

1. Depth Prediction and Encoding: MonoDETR introduces a foreground depth map predicted through a lightweight depth predictor, specialized to identify object-wise depth without requiring dense annotations. This approach offers efficiency and effectiveness in focusing on critical depth cues.
2. Parallel Depth and Visual Encoders: The architecture employs two parallel encoders, enhancing both visual and depth representations. This dual approach contributes to a better understanding of 3D spatial structures by capturing distinct visual appearances and depth geometries.
3. Depth-guided Decoder: A novel depth-guided decoder facilitates scene-level interactions by employing a depth cross-attention layer. This layer fosters robust feature aggregation across the entire image, enabling object queries to derive 3D attributes through enriched context.

Results and Implications

On benchmark datasets such as KITTI, MonoDETR achieves state-of-the-art performance, demonstrating substantial improvements in 3D object detection accuracy over traditional methods. Notably, it recorded gains of +2.53%, +1.08%, and +0.85% at easy, moderate, and hard difficulty levels, respectively, in terms of $AP_{3D}$.

The implications of these results are significant for both theoretical and practical applications. Theoretically, MonoDETR pushes the boundaries of monocular 3D object detection frameworks by fully integrating depth guidance mechanisms. Practically, its plug-and-play nature offers flexibility to enhance existing multi-view 3D detection systems, as evidenced by improved performance with minor adjustments in established models like PETRv2 and BEVFormer.

Future Directions

Future research may explore extending the capabilities of depth-guided transformers to multi-modal inputs, integrating additional sensors such as LiDAR or RADAR to further enhance depth perception and spatial comprehension. Additionally, optimizing computational efficiency while maintaining high detection accuracy could provide broader applicability in real-time data processing contexts within autonomous driving systems.

MonoDETR's depth-focused approach provides a promising avenue for advancing 3D object detection technology, particularly where resource constraints and data limitations pose significant challenges. This paper offers a compelling case for adopting depth guidance as a primary mechanism in monocular detection frameworks.