Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles (2302.11919v2)

Published 23 Feb 2023 in cs.RO and cs.AI

Abstract: Even though virtual testing of Autonomous Vehicles (AVs) has been well recognized as essential for safety assessment, AV simulators are still undergoing active development. One particularly challenging question is to effectively include the Sensing and Perception (S&P) subsystem into the simulation loop. In this article, we define Perception Error Models (PEM), a virtual simulation component that can enable the analysis of the impact of perception errors on AV safety, without the need to model the sensors themselves. We propose a generalized data-driven procedure towards parametric modeling and evaluate it using Apollo, an open-source driving software, and nuScenes, a public AV dataset. Additionally, we implement PEMs in SVL, an open-source vehicle simulator. Furthermore, we demonstrate the usefulness of PEM-based virtual tests, by evaluating camera, LiDAR, and camera-LiDAR setups. Our virtual tests highlight limitations in the current evaluation metrics, and the proposed approach can help study the impact of perception errors on AV safety.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Y. Shen, S. Jiang, Y. Chen, E. Yang, X. Jin, Y. Fan, and K. D. Campbell, “To explain or not to explain: A study on the necessity of explanations for autonomous vehicles,” arXiv:2006.11684, 2020.
  2. SAE, “J3016_201806 standard: Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles,” 2018.
  3. E. D. Marti, M. A. de Miguel, F. Garcia, and J. Perez, “A Review of Sensor Technologies for Perception in Automated Driving,” IEEE Intelligent Transportation Systems Magazine, 2019.
  4. NTSB, “Collison Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian.” 2019, highway Accident Report NTSB/HAR-19/03. Washington, DC.
  5. W. Young, A. Sobhani, M. G. Lenné, and M. Sarvi, “Simulation of safety: A review of the state of the art in road safety simulation modelling,” Accident Analysis & Prevention, vol. 66, pp. 89–103, 2014.
  6. A. Piazzoni, J. Cherian, M. Slavik, and J. Dauwels, “Modeling perception errors towards robust decision making in autonomous vehicles,” in Proc. of the 29th International Joint Conference on Artificial Intelligence, IJCAI-20, 7 2020, pp. 3494–3500, main track.
  7. F. Rosique, P. J. Navarro, C. Fernández, and A. Padilla, “A systematic review of perception system and simulators for autonomous vehicles research,” Sensors, vol. 19, no. 3, p. 648, 2019.
  8. S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a formal model of safe and scalable self-driving cars,” arXiv:1708.06374, 2017.
  9. S. Feng, Y. Feng, C. Yu, Y. Zhang, and H. X. Liu, “Testing scenario library generation for connected and automated vehicles, part i: Methodology,” IEEE Transactions on Intelligent Transportation Systems, 2020.
  10. A. Piazzoni, J. Cherian, M. Azhar, J. Y. Yap, J. L. W. Shung, and R. Vijay, “ViSTA: a framework for virtual scenario-based testing of autonomous vehicles,” in 2021 IEEE International Conference on Artificial Intelligence Testing (AITest), 2021, pp. 143–150.
  11. C. Gómez-Huélamo, J. Del Egido, L. M. Bergasa, R. Barea, E. López-Guillén, F. Arango, J. Araluce, and J. López, “Train here, drive there: Ros based end-to-end autonomous-driving pipeline validation in carla simulator using the nhtsa typology,” Multimedia Tools and Applications, vol. 81, no. 3, pp. 4213–4240, 2022.
  12. J. Guo, U. Kurup, and M. Shah, “Is it safe to drive? an overview of factors, metrics, and datasets for driveability assessment in autonomous driving,” IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 8, pp. 3135–3151, 2019.
  13. IPG, “IPG CarMaker, ipg-automotive.com/en/products-solutions/software/carmaker/,” 2022.
  14. MSC, “Vires VTD, vires.mscsoftware.com,” 2022.
  15. Siemens, “Siemens Simcenter Prescan, plm.automation.siemens.com/global/en/products/simcenter/prescan.html,” 2022.
  16. NVIDIA, “NVIDIA DRIVE Sim and Omniverse, nvidia.com/en-sg/self-driving-cars/simulation/,” 2022.
  17. A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning, 2017, pp. 1–16.
  18. G. Rong, B. H. Shin, H. Tabatabaee, Q. Lu, S. Lemke, M. Možeiko, E. Boise, G. Uhm, M. Gerow, S. Mehta et al., “Lgsvl simulator: A high fidelity simulator for autonomous driving,” arXiv:2005.03778, 2020.
  19. A. Cortés, C. Rodríguez, G. Vélez, J. Barandiarán, and M. Nieto, “Analysis of classifier training on synthetic data for cross-domain datasets,” IEEE Transactions on Intelligent Transportation Systems, 2020.
  20. D. Talwar, S. Guruswamy, N. Ravipati, and M. Eirinaki, “Evaluating validity of synthetic data in perception tasks for autonomous vehicles,” in 2020 IEEE International Conference On Artificial Intelligence Testing (AITest).   IEEE, 2020, pp. 73–80.
  21. M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International journal of computer vision, vol. 88, no. 2, pp. 303–338, 2010.
  22. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  23. K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: the clear mot metrics,” EURASIP Journal on Image and Video Processing, vol. 2008, pp. 1–10, 2008.
  24. C.-H. Cheng, C.-H. Huang, H. Ruess, H. Yasuoka et al., “Towards dependability metrics for neural networks,” in ACM/IEEE International Conference on Formal Methods and Models for System Design, 2018, pp. 1–4.
  25. M. Lyssenko, C. Gladisch, C. Heinzemann, M. Woehrle, and R. Triebel, “From evaluation to verification: Towards task-oriented relevance metrics for pedestrian detection in safety-critical domains,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (Workshop contribution), 2021, pp. 38–45.
  26. G. Volk, J. Gamerdinger, A. von Bernuth, and O. Bringmann, “A comprehensive safety metric to evaluate perception in autonomous systems,” in IEEE International Conference on Intelligent Transportation Systems, 2020, pp. 1–8.
  27. C.-H. Cheng, A. Knoll, and H.-C. Liao, “Safety metrics for semantic segmentation in autonomous driving,” in IEEE International Conference on Artificial Intelligence Testing, 2021, pp. 57–64.
  28. H. Arnelid, E. L. Zec, and N. Mohammadiha, “Recurrent conditional generative adversarial networks for autonomous driving sensor modelling,” in 2019 IEEE Intelligent Transportation Systems Conference (ITSC).   IEEE, 2019, pp. 1613–1618.
  29. V. Berkhahn, M. Kleiber, J. Langner, C. Timmermann, and S. Weber, “Traffic dynamics at intersections subject to random misperception,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 5, pp. 4501–4511, 2021.
  30. N. Hirsenkorn, T. Hanke, A. Rauch, B. Dehlink, R. Rasshofer, and E. Biebl, “Virtual sensor models for real-time applications,” Advances in Radio Science, vol. 14, no. B., pp. 31–37, 2016.
  31. R. Krajewski, M. Hoss, A. Meister, F. Thomsen, J. Bock, and L. Eckstein, “Using drones as reference sensors for neural-networks-based modeling of automotive perception errors,” in 2020 IEEE Intelligent Vehicles Symposium (IV), 2020, pp. 708–715.
  32. P. Mitra, A. Choudhury, V. R. Aparow, G. Kulandaivelu, and J. Dauwels, “Towards modeling of perception errors in autonomous vehicles,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2018, pp. 3024–3029.
  33. A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in 2012 IEEE conference on computer vision and pattern recognition.   IEEE, 2012, pp. 3354–3361.
  34. C. Innes and S. Ramamoorthy, “Testing rare downstream safety violations via upstream adaptive sampling of perception error models,” arXiv preprint arXiv:2209.09674, 2022.
  35. J. Sadeghi, B. Rogers, J. Gunn, T. Saunders, S. Samangooei, P. K. Dokania, and J. Redford, “A step towards efficient evaluation of complex perception tasks in simulation,” arXiv preprint arXiv:2110.02739, 2021.
  36. E. L. Zec, N. Mohammadiha, and A. Schliep, “Statistical sensor modelling for autonomous driving using autoregressive input-output hmms,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2018, pp. 1331–1336.
  37. J. Zhao, Y. Li, B. Zhu, W. Deng, and B. Sun, “Method and applications of lidar modeling for virtual testing of intelligent vehicles,” IEEE Transactions on Intelligent Transportation Systems, 2020.
  38. M. Hoss, M. Scholtes, and L. Eckstein, “A review of testing object-based environment perception for safe automated driving,” arXiv:2102.08460, 2021.
  39. G. Ros, V. Koltun, F. Codevilla, and A. Lopez, “The carla autonomous driving challenge https://carlachallenge.org/,” 2019.
  40. T. Hanke, N. Hirsenkorn, B. Dehlink, A. Rauch, R. Rasshofer, and E. Biebl, “Classification of sensor errors for the statistical simulation of environmental perception in automated driving systems,” in 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2016, pp. 643–648.
  41. D. Hoiem, Y. Chodpathumwan, and Q. Dai, “Diagnosing error in object detectors,” in European conference on computer vision.   Springer, 2012, pp. 340–353.
  42. H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong, “Baidu Apollo EM Motion Planner,” arXiv: 807.08048, 2018. [Online]. Available: www.apollo.auto
  43. H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proc. IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 621–11 631.
  44. X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, and R. Yang, “The apolloscape dataset for autonomous driving,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition Workshops, 2018, pp. 954–960.
  45. R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and V. Shet, “Level 5 perception dataset 2020,” https://level-5.global/level5/data/, 2019.
  46. “Waymo open dataset: An autonomous driving dataset,” https://www.waymo.com/open, 2019.
  47. “CyberRT,” https://cyber-rt.readthedocs.io/, 2019.
  48. J. Florbäck, L. Tornberg, and N. Mohammadiha, “Offline object matching and evaluation process for verification of autonomous driving,” in 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2016, pp. 107–112.
  49. J. Salvatier, T. V. Wiecki, and C. Fonnesbeck, “Probabilistic programming in python using pymc3,” PeerJ Computer Science, vol. 2, p. e55, 2016.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now