Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Bilateral Propagation Network for Depth Completion (2403.11270v2)

Published 17 Mar 2024 in cs.CV

Abstract: Depth completion aims to derive a dense depth map from sparse depth measurements with a synchronized color image. Current state-of-the-art (SOTA) methods are predominantly propagation-based, which work as an iterative refinement on the initial estimated dense depth. However, the initial depth estimations mostly result from direct applications of convolutional layers on the sparse depth map. In this paper, we present a Bilateral Propagation Network (BP-Net), that propagates depth at the earliest stage to avoid directly convolving on sparse data. Specifically, our approach propagates the target depth from nearby depth measurements via a non-linear model, whose coefficients are generated through a multi-layer perceptron conditioned on both \emph{radiometric difference} and \emph{spatial distance}. By integrating bilateral propagation with multi-modal fusion and depth refinement in a multi-scale framework, our BP-Net demonstrates outstanding performance on both indoor and outdoor scenes. It achieves SOTA on the NYUv2 dataset and ranks 1st on the KITTI depth completion benchmark at the time of submission. Experimental results not only show the effectiveness of bilateral propagation but also emphasize the significance of early-stage propagation in contrast to the refinement stage. Our code and trained models will be available on the project page.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Learning joint 2d-3d representations for depth completion. In ICCV, pages 10023–10032, 2019.
  2. Estimating depth from rgb and sparse sensing. In ECCV, pages 167–182, 2018.
  3. Learning depth with convolutional spatial propagation network. IEEE TPAMI, 42(10):2361–2379, 2019.
  4. Cspn++: Learning context and resource aware convolutional spatial propagation networks for depth completion. In AAAI, pages 10615–10622, 2020.
  5. Sparsity agnostic depth completion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5871–5880, 2023.
  6. An application of markov random fields to range sensing. NeurIPS, 18, 2005.
  7. Confidence propagation through cnns for guided sparse depth regression. IEEE TPAMI, 42(10):2423–2436, 2019.
  8. Image guided depth upsampling using anisotropic total generalized variation. In ICCV, pages 993–1000, 2013.
  9. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
  10. Guided image filtering. IEEE TPAMI, 35(6):1397–1409, 2012.
  11. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  12. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
  13. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  14. Penet: Towards precise and efficient image guided depth completion. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13656–13662. IEEE, 2021.
  15. Hms-net: Hierarchical multi-scale sparsity-invariant network for sparse depth completion. IEEE TIP, 29:3429–3441, 2019.
  16. Fusion of range and color images for denoising and resolution enhancement with a non-local filter. Computer vision and image understanding, 114(12):1336–1345, 2010.
  17. Depth coefficients for depth completion. In CVPR, pages 12438–12447. IEEE, 2019.
  18. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. pmlr, 2015.
  19. Joint bilateral upsampling. ACM TOG, 26(3):96–es, 2007.
  20. In defense of classical image processing: Fast depth completion on the cpu. In 2018 15th Conference on Computer and Robot Vision (CRV), pages 16–22. IEEE, 2018.
  21. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648, 2016.
  22. Joint image filtering with deep convolutional networks. IEEE TPAMI, 41(8):1909–1923, 2019.
  23. Dynamic spatial propagation network for depth completion. In AAAI, pages 1638–1646, 2022.
  24. Learning steering kernels for guided depth completion. IEEE TIP, 30:2850–2861, 2021.
  25. Learning affinity via spatial propagation networks. NeurIPS, 30, 2017.
  26. Graphcspn: Geometry-aware depth completion via dynamic gcns. In ECCV, pages 90–107. Springer, 2022.
  27. Decoupled weight decay regularization. In ICLR, 2018.
  28. Sparse-to-dense: Depth prediction from sparse depth samples and a single image. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4796–4803. IEEE, 2018.
  29. Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera. In 2019 International Conference on Robotics and Automation (ICRA), pages 3288–3295. IEEE, 2019.
  30. Learning ambidextrous robot grasping policies. Science Robotics, 4(26):eaau4984, 2019.
  31. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  32. Non-local spatial propagation network for depth completion. In ECCV, pages 120–136. Springer, 2020.
  33. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  34. Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image. In CVPR, pages 3313–3322, 2019.
  35. Depth completion via deep basis fitting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 71–80, 2020.
  36. Bayesian deep basis fitting for depth completion with uncertainty. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16147–16157, 2021.
  37. Guideformer: Transformers for image guided depth completion. In CVPR, pages 6250–6259, 2022.
  38. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  39. Structure-from-motion revisited. In CVPR, pages 4104–4113, 2016.
  40. Indoor segmentation and support inference from rgbd images. In ECCV, pages 746–760, 2012.
  41. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, pages 369–386. SPIE, 2019.
  42. Learning guided convolutional network for depth completion. IEEE TIP, 30:1116–1129, 2020.
  43. Bilateral filtering for gray and color images. In ICCV, pages 839–846. IEEE, 1998.
  44. Learning joint intensity-depth sparse representations. IEEE TIP, 23(5):2122–2132, 2014.
  45. Sparsity invariant cnns. In IEEE International Conference on 3D Vision (3DV), pages 11–20, 2017.
  46. Attention is all you need. 30, 2017.
  47. Lrru: Long-short range recurrent updating networks for depth completion. In ICCV, pages 9422–9432, 2023.
  48. Rigidfusion: Rgb-d scene reconstruction with rigidly-moving objects. In Computer Graphics Forum, pages 511–522. Wiley Online Library, 2021.
  49. Depth completion from sparse lidar data with depth-normal constraints. In ICCV, pages 2811–2820, 2019.
  50. Rignet: Repetitive image guided network for depth completion. In ECCV, pages 214–230. Springer, 2022.
  51. Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE TIP, 23(8):3443–3458, 2014.
  52. Dense depth posterior (ddp) from single image and sparse range. In CVPR, pages 3353–3362, 2019.
  53. Deep depth completion of a single rgb-d image. In CVPR, pages 175–185, 2018.
  54. Completionformer: Depth completion with convolutions and vision transformers. In CVPR, pages 18527–18536, 2023.
  55. Adaptive context-aware multi-modal network for depth completion. IEEE TIP, 30:5264–5276, 2021a.
  56. A surface geometry model for lidar depth completion. IEEE Robotics and Automation Letters, 6(3):4457–4464, 2021b.
  57. Bev@ dc: Bird’s-eye view assisted training for depth completion. In CVPR, pages 9233–9242, 2023.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now