Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Cooperative Visual-LiDAR Extrinsic Calibration Technology for Intersection Vehicle-Infrastructure: A review (2405.10132v1)

Published 16 May 2024 in cs.CV

Abstract: In the typical urban intersection scenario, both vehicles and infrastructures are equipped with visual and LiDAR sensors. By successfully integrating the data from vehicle-side and road monitoring devices, a more comprehensive and accurate environmental perception and information acquisition can be achieved. The Calibration of sensors, as an essential component of autonomous driving technology, has consistently drawn significant attention. Particularly in scenarios involving multiple sensors collaboratively perceiving and addressing localization challenges, the requirement for inter-sensor calibration becomes crucial. Recent years have witnessed the emergence of the concept of multi-end cooperation, where infrastructure captures and transmits surrounding environment information to vehicles, bolstering their perception capabilities while mitigating costs. However, this also poses technical complexities, underscoring the pressing need for diverse end calibration. Camera and LiDAR, the bedrock sensors in autonomous driving, exhibit expansive applicability. This paper comprehensively examines and analyzes the calibration of multi-end camera-LiDAR setups from vehicle, roadside, and vehicle-road cooperation perspectives, outlining their relevant applications and profound significance. Concluding with a summary, we present our future-oriented ideas and hypotheses.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (123)
  1. Sensor and sensor fusion technology in autonomous vehicles: A review. Sensors, 21(6):2140, 2021.
  2. Help or hindrance? the travel, energy and carbon impacts of highly automated vehicles. Transportation Research Part A: Policy and Practice, 86:1–18, 2016.
  3. The impact of autonomous vehicles on cities: A review. Journal of Urban Technology, 25(4):3–18, 2018.
  4. A review of sensor technologies for perception in automated driving. IEEE Intelligent Transportation Systems Magazine, 11(4):94–108, 2019.
  5. Multi-sensor fusion in automated driving: A survey. Ieee Access, 8:2847–2868, 2019.
  6. Exploiting key points supervision and grouped feature fusion for multiview pedestrian detection. Pattern Recognition, 131:108866, 2022.
  7. Sensors and sensor fusion in autonomous vehicles. In 2018 26th Telecommunications Forum (TELFOR), pages 420–425. IEEE, 2018.
  8. Blind-spot visualization via ar glasses using millimeter-wave v2x for safe driving. In 2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall), pages 1–5. IEEE, 2021.
  9. Fusion of 3d lidar and camera data for object detection in autonomous vehicle applications. IEEE Sensors Journal, 20(9):4901–4913, 2020.
  10. Online camera lidar fusion and object detection on hybrid data for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1632–1638. IEEE, 2018.
  11. Ece498sm: V2v lidar communication in occluded environments.
  12. A spatial calibration method for robust cooperative perception. IEEE Robotics and Automation Letters, 2024.
  13. Road-feature extraction using point cloud and 3d lidar sensor for vehicle localization. In 2017 14th international conference on ubiquitous robots and ambient intelligence (URAI), pages 891–892. IEEE, 2017.
  14. V2i based environment perception for autonomous vehicles at intersections. China Communications, 18(7):1–12, 2021.
  15. Infra2go: A mobile development platform for connected, cooperative and autonomous driving. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pages 1629–1636. IEEE, 2022.
  16. Songbin Chen. Intelligent perception system for vehicle-road cooperation. arXiv preprint arXiv:2208.14052, 2022.
  17. Yu-Jin Zhang. Camera calibration. In 3-D Computer Vision: Principles, Algorithms and Applications, pages 37–65. Springer, 2023.
  18. Extrinsic calibration of a 3d-lidar and a camera. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1765–1770. IEEE, 2020.
  19. Robert Pless. Using many cameras as one. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume 2, pages II–587. IEEE, 2003.
  20. Intrinsic and extrinsic active self-calibration of multi-camera systems. Machine vision and applications, 25(2):389–403, 2014.
  21. Multi-camera lidar inertial extension to the newer college dataset. arXiv preprint arXiv:2112.08854, 2021.
  22. Extrinsic calibration of multiple 3d lidar sensors by the use of planar objects. Sensors, 22(19):7234, 2022.
  23. Zhengyou Zhang. Camera calibration with one-dimensional objects. IEEE transactions on pattern analysis and machine intelligence, 26(7):892–899, 2004.
  24. A flexible method for zoom lens calibration and modeling using a planar checkerboard. Photogrammetric Engineering & Remote Sensing, 79(6):555–571, 2013.
  25. Design and recognition of three dimensional calibration target based on coded marker. In 2015 International Conference on Optical Instruments and Technology: Optoelectronic Imaging and Processing Technology, volume 9622, pages 285–292. SPIE, 2015.
  26. Roger Tsai. A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal on Robotics and Automation, 3(4):323–344, 1987.
  27. Zhengyou Zhang. A flexible new technique for camera calibration. IEEE Transactions on pattern analysis and machine intelligence, 22(11):1330–1334, 2000.
  28. Camera calibration method based on optimal polarization angle. Optics and Lasers in Engineering, 112:128–135, 2019.
  29. The double sphere camera model. In 2018 International Conference on 3D Vision (3DV), pages 552–560. IEEE, 2018.
  30. Accuracy and reliability of multi-gnss real-time precise positioning: Gps, glonass, beidou, and galileo. Journal of geodesy, 89(6):607–635, 2015.
  31. Open-source lidar time synchronization system by mimicking gnss-clock. In 2022 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control, and Communication (ISPCS), pages 1–5. IEEE, 2022.
  32. A low-cost system for high-rate, high-accuracy temporal calibration for lidars and cameras. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2219–2226, 2017.
  33. Automated extrinsic calibration of multi-cameras and lidar. volume 73, pages 1–12, 2024.
  34. Sst-calib: Simultaneous spatial-temporal parameter calibration between lidar and camera. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pages 2896–2902. IEEE, 2022.
  35. Demonstrations of cooperative perception: Safety and robustness in connected and automated vehicle operations. Sensors, 21(1):200, 2020.
  36. Multi-sensor fusion algorithm in cooperative vehicle-infrastructure system for blind spot warning. International Journal of Distributed Sensor Networks, 18(5):15501329221100412, 2022.
  37. Overview of autonomous vehicle sensors and systems. In International Conference on Operations Excellence and Service Engineering, volume 2015. sn, 2015.
  38. Lidar-enhanced connected infrastructures sensing and broadcasting high-resolution traffic information serving smart cities. Ieee Access, 7:79895–79907, 2019.
  39. Automatic extrinsic calibration method for lidar and camera sensor setups. IEEE Transactions on Intelligent Transportation Systems, 23(10):17677–17689, 2022.
  40. Accurate calibration of multi-lidar-multi-camera systems. Sensors, 18(7):2139, 2018.
  41. An analytical least-squares solution to the line scan lidar-camera extrinsic calibration problem. In 2013 IEEE International Conference on Robotics and Automation, pages 2943–2948. IEEE, 2013.
  42. Automatic extrinsic calibration for lidar-stereo vehicle sensor setups. In 2017 IEEE 20th international conference on intelligent transportation systems (ITSC), pages 1–6. IEEE, 2017.
  43. Sergiu Nedevschi et al. Online cross-calibration of camera and lidar. In 2017 13th IEEE International Conference on Intelligent Computer Communication and Processing (ICCP), pages 295–301. IEEE, 2017.
  44. Automatic extrinsic calibration between a camera and a 3d lidar using 3d point and plane correspondences. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 3906–3912. IEEE, 2019.
  45. Accurate calibration of lidar-camera systems using ordinary boxes. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 394–402, 2017.
  46. A novel calibration method between a camera and a 3d lidar with infrared images. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4963–4969. IEEE, 2020.
  47. A novel method for the extrinsic calibration of a 2d laser rangefinder and a camera. IEEE Sensors Journal, 18(10):4200–4211, 2018.
  48. Automatic calibration of multiple cameras and depth sensors with a spherical target. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, 2018.
  49. Extrinsic calibration of a 2d laser-rangefinder and a camera based on scene corners. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 3611–3616. IEEE, 2015.
  50. Automatic calibration of lidar and camera images using normalized mutual information. In Robotics and Automation (ICRA), 2013 IEEE International Conference on. Citeseer, 2013.
  51. General, single-shot, target-less, and automatic lidar-camera extrinsic calibration toolbox. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11301–11307. IEEE, 2023.
  52. Line-based automatic extrinsic calibration of lidar and camera. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9347–9353. IEEE, 2021.
  53. Pixel-level extrinsic self calibration of high resolution lidar and camera in targetless environments. IEEE Robotics and Automation Letters, 6(4):7517–7524, 2021.
  54. A roadside camera-radar sensing fusion system for intelligent transportation. In 2020 17th European Radar Conference (EuRAD), pages 282–285. IEEE, 2021.
  55. Providentia – a large scale sensing system for the assistance of autonomous vehicles. 2019.
  56. Laci: Low-effort automatic calibration of infrastructure sensors. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 3928–3933. IEEE, 2019.
  57. Intrinsic and extrinsic calibration of roadside lidar and camera. In 2022 China Automation Congress (CAC), pages 2367–2372, 2022.
  58. Regnet: Multimodal sensor registration using deep neural networks. In 2017 IEEE intelligent vehicles symposium (IV), pages 1803–1810. IEEE, 2017.
  59. Visual odometry driven online calibration for monocular lidar-camera systems. In 2016 23rd international conference on pattern recognition (icpr), pages 2848–2853. IEEE, 2016.
  60. Soic: Semantic online initialization and calibration for lidar and camera. 2020.
  61. Providentia–a large-scale sensor system for the assistance of autonomous vehicles and its evaluation. arXiv preprint arXiv:1906.06789, 2019.
  62. A taxonomy and analysis of camera calibration methods for traffic monitoring applications. IEEE Transactions on Intelligent Transportation Systems, 11(2):441–452, 2010.
  63. TCITS ETSI. Intelligent transport systems (its); vehicular communications; basic set of applications; part 2: Specification of cooperative awareness basic service. Draft ETSI TS, 20(2011):448–51, 2011.
  64. A digital mirror: A mobile edge computing service based on infrastructure sensors. In 25th ITS World Congress Copenhagen, 2018.
  65. Keypoint-based lidar-camera online calibration with robust geometric network. IEEE Transactions on Instrumentation and Measurement, 71:1–11, 2021.
  66. Calibnet: Geometrically supervised extrinsic calibration using 3d spatial transformer networks. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1110–1117. IEEE, 2018.
  67. Lccnet: Lidar and camera self-calibration using cost volume network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2894–2901, 2021.
  68. Rggnet: Tolerance aware lidar-camera online calibration with geometric deep learning and generative model. IEEE Robotics and Automation Letters, 5(4):6956–6963, 2020.
  69. Atop: An attention-to-optimization approach for automatic lidar-camera calibration via cross-modal object matching. IEEE Transactions on Intelligent Vehicles, 8(1):696–708, 2022.
  70. Cooperative perception of roadside unit and onboard equipment with edge artificial intelligence for driving assistance. 2021.
  71. Roadside edge sensed and fused three-dimensional localization using camera and lidar. In 2022 7th International Conference on Image, Vision and Computing (ICIVC), pages 762–766. IEEE, 2022.
  72. Points registration for roadside lidar sensors. Transportation research record, 2673(9):627–639, 2019.
  73. 3d point cloud registration for multiple roadside lidars with retroreflective reference. In 2020 IEEE International Conference on Networking, Sensing and Control (ICNSC), pages 1–6. IEEE, 2020.
  74. Revolution and rotation-based method for roadside lidar data integration. Optics & Laser Technology, 119:105571, 2019.
  75. Trajmatch: Toward automatic spatio-temporal calibration for roadside lidars through trajectory matching. IEEE Transactions on Intelligent Transportation Systems, 2023.
  76. A spatial alignment framework using geolocation cues for roadside multi-view multi-sensor fusion. In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pages 3633–3640. IEEE, 2023.
  77. Leveraging dynamic objects for relative localization correction in a connected autonomous vehicle network. arXiv preprint arXiv:2205.09418, 2022.
  78. A cooperative perception system robust to localization errors. In 2023 IEEE Intelligent Vehicles Symposium (IV), pages 1–6. IEEE, 2023.
  79. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 4812–4818. IEEE, 2023.
  80. Alignment of perception information for cooperative perception. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 1849–1854. IEEE, 2019.
  81. The normal distributions transform: A new approach to laser scan matching. In Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No. 03CH37453), volume 3, pages 2743–2748. IEEE, 2003.
  82. Vi-eye: semantic-based 3d point cloud registration for infrastructure-assisted autonomous driving. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, pages 573–586, 2021.
  83. Hpcr-vi: Heterogeneous point cloud registration for vehicle-infrastructure collaboration. In 2023 IEEE Intelligent Vehicles Symposium (IV), pages 1–6. IEEE, 2023.
  84. Fast global registration. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 766–782. Springer, 2016.
  85. Vips: Real-time perception fusion for infrastructure-assisted autonomous driving. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking, pages 133–146, 2022.
  86. Rendering-enhanced automatic image-to-point cloud registration for roadside scenes. arXiv preprint arXiv:2404.05164, 2024.
  87. An efficient and robust object-level cooperative perception framework for connected and automated driving. arXiv preprint arXiv:2210.06289, 2, 2022.
  88. Fast and robust iterative closest point. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3450–3466, 2021.
  89. A spectral technique for correspondence problems using pairwise constraints. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, volume 2, pages 1482–1489. IEEE, 2005.
  90. Cooper: Cooperative perception for connected autonomous vehicles based on 3d point clouds. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pages 514–524. IEEE, 2019.
  91. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pages 88–100, 2019.
  92. V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. In European conference on computer vision, pages 107–124. Springer, 2022.
  93. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 605–621. Springer, 2020.
  94. Learning to communicate and correct pose errors. In Conference on Robot Learning, pages 1195–1210. PMLR, 2021.
  95. Keypoints-based deep feature fusion for cooperative vehicle detection of autonomous driving. IEEE Robotics and Automation Letters, 7(2):3054–3061, 2022.
  96. V2vformer: Vehicle-to-vehicle cooperative perception with spatial-channel transformer. IEEE Transactions on Intelligent Vehicles, 2024.
  97. Spatio-temporal domain awareness for multi-agent collaborative perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23383–23392, 2023.
  98. Overcoming obstructions via bandwidth-limited multi-agent spatial handshaking. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2406–2413. IEEE, 2021.
  99. Mcot: Multi-modal vehicle-to-vehicle cooperative perception with transformers. In 2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS), pages 1612–1619. IEEE, 2023.
  100. Hm-vit: Hetero-modal vehicle-to-vehicle cooperative perception with vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 284–295, 2023.
  101. Cooperative perception for 3d object detection in driving scenarios using infrastructure sensors. IEEE Transactions on Intelligent Transportation Systems, 23(3):1852–1864, 2020.
  102. Fast and robust registration of partially overlapping point clouds. IEEE Robotics and Automation Letters, 7(2):1502–1509, 2021.
  103. V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4):10914–10921, 2022.
  104. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17252–17262, 2022.
  105. Dolphins: Dataset for collaborative perception enabled harmonious and interconnected self-driving. In Proceedings of the Asian Conference on Computer Vision, pages 4361–4377, 2022.
  106. Y Yuan and M Sester. Comap: A synthetic dataset for collective multi-agent perception of autonomous driving. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 43:255–263, 2021.
  107. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In 2022 International Conference on Robotics and Automation (ICRA), pages 2583–2589. IEEE, 2022.
  108. Deepaccident: A motion and accident prediction benchmark for v2x autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5599–5606, 2024.
  109. A9-dataset: Multi-sensor infrastructure-based dataset for mobility research. In 2022 IEEE Intelligent Vehicles Symposium (IV), pages 965–970. IEEE, 2022.
  110. Lucoop: Leibniz university cooperative perception and urban navigation dataset. In 2023 IEEE Intelligent Vehicles Symposium (IV), pages 1–8. IEEE, 2023.
  111. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21361–21370, 2022.
  112. Ips300+: a challenging multi-modal data sets for intersection perception system. In 2022 International Conference on Robotics and Automation (ICRA), pages 2539–2545. IEEE, 2022.
  113. V2x-seq: A large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5486–5495, 2023.
  114. Weakly supervised training of monocular 3d object detectors using wide baseline multi-view traffic camera data. arXiv preprint arXiv:2110.10966, 2021.
  115. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13712–13722, 2023.
  116. Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16. PMLR, 2017.
  117. Lidarsim: Realistic lidar simulation by leveraging the real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11167–11176, 2020.
  118. Daniel Krajzewicz. Traffic simulation with sumo–simulation of urban mobility. Fundamentals of traffic simulation, pages 269–293, 2010.
  119. Opencda: an open cooperative driving automation framework integrated with co-simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1155–1162. IEEE, 2021.
  120. Towards vehicle-to-everything autonomous driving: A survey on collaborative perception. arXiv preprint arXiv:2308.16714, 2023.
  121. Multistage fusion approach of lidar and camera for vehicle-infrastructure cooperative object detection. In 2022 5th World Conference on Mechanical Engineering and Intelligent Manufacturing (WCMEIM), pages 811–816. IEEE, 2022.
  122. Multi-modal virtual-real fusion based transformer for collaborative perception. In 2022 IEEE 13th International Symposium on Parallel Architectures, Algorithms and Programming (PAAP), pages 1–6. IEEE, 2022.
  123. Vehicle-infrastructure cooperative 3d object detection via feature flow prediction. arXiv preprint arXiv:2303.10552, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets