Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DrPlanner: Diagnosis and Repair of Motion Planners for Automated Vehicles Using Large Language Models (2403.07470v2)

Published 12 Mar 2024 in cs.RO and cs.PL

Abstract: Motion planners are essential for the safe operation of automated vehicles across various scenarios. However, no motion planning algorithm has achieved perfection in the literature, and improving its performance is often time-consuming and labor-intensive. To tackle the aforementioned issues, we present DrPlanner, the first framework designed to automatically diagnose and repair motion planners using LLMs. Initially, we generate a structured description of the planner and its planned trajectories from both natural and programming languages. Leveraging the profound capabilities of LLMs, our framework returns repaired planners with detailed diagnostic descriptions. Furthermore, our framework advances iteratively with continuous feedback from the evaluation of the repaired outcomes. Our approach is validated using both search- and sampling-based motion planners for automated vehicles; experimental results highlight the need for demonstrations in the prompt and show the ability of our framework to effectively identify and rectify elusive issues.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. B. Paden, M. Čáp, S. Z. Yong, D. Yershov, and E. Frazzoli, “A survey of motion planning and control techniques for self-driving urban vehicles,” vol. 1, no. 1, pp. 33–55, 2016.
  2. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in Neural Info. Processing Syst., vol. 33, pp. 1877–1901, 2020.
  3. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., “Training language models to follow instructions with human feedback,” Advances in Neural Info. Processing Syst., vol. 35, pp. 27 730–27 744, 2022.
  4. OpenAI, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
  5. T. Gu, J. M. Dolan, and J.-W. Lee, “Runtime-bounded tunable motion planning for autonomous driving,” in Proc. of the IEEE Intell. Veh. Symp., 2016, pp. 1301–1306.
  6. S. Liu and P. Liu, “Benchmarking and optimization of robot motion planning with motion planning pipeline,” Int. J. of Advanced Manufacturing Technology, vol. 118, pp. 949–961, 2022.
  7. M. Zucker, N. Ratliff, A. D. Dragan, M. Pivtoraiko, M. Klingensmith, C. M. Dellin, J. A. Bagnell, and S. S. Srinivasa, “CHOMP: Covariant Hamiltonian optimization for motion planning,” Int. J. of Robot. Research, vol. 32, no. 9-10, pp. 1164–1193, 2013.
  8. S. M. LaValle, J. J. Kuffner, B. Donald, et al., “Rapidly-exploring random trees: Progress and prospects,” Algorithmic and Computational Robot.: New Directions, vol. 5, pp. 293–308, 2001.
  9. S. Karaman and E. Frazzoli, “Incremental sampling-based algorithms for optimal motion planning,” Robot. Science and Systems, vol. 104, no. 2, pp. 267–274, 2010.
  10. S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” J. of Field Robot., vol. 37, pp. 362–386, 2019.
  11. S. Aradi, “Survey of deep reinforcement learning for motion planning of autonomous vehicles,” IEEE Trans. on Intell. Transp. Syst., vol. 23, no. 2, pp. 740–759, 2022.
  12. B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. A. Sallab, S. Yogamani, and P. Pérez, “Deep reinforcement learning for autonomous driving: A survey,” IEEE Trans. on Intell. Transp. Syst., vol. 23, no. 6, pp. 4909–4926, 2022.
  13. H. Krasowski, X. Wang, and M. Althoff, “Safe reinforcement learning for autonomous lane changing using set-based prediction,” in Proc. of the IEEE Int. Conf. on Intell. Transp. Syst., 2020, pp. 1–7.
  14. L. Wang, L. Sun, M. Tomizuka, and W. Zhan, “Socially-compatible behavior design of autonomous vehicles with verification on real human data,” IEEE Robot. and Automation Letters, vol. 6, no. 2, pp. 3421–3428, 2021.
  15. N. Mehdipour, M. Althoff, R. D. Tebbens, and C. Belta, “Formal methods to comply with rules of the road in autonomous driving: State of the art and grand challenges,” Automatica, vol. 152, no. 110692, 2023.
  16. L. Gazzola, D. Micucci, and L. Mariani, “Automatic software repair: A survey,” IEEE Trans. on Software Engineering, vol. 45, no. 1, pp. 34–67, 2019.
  17. W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest, “Automatically finding patches using genetic programming,” in Proc. of the IEEE Int. Conf. on Software Engineering, 2009, pp. 364–374.
  18. D. Kim, J. Nam, J. Song, and S. Kim, “Automatic patch generation learned from human-written patches,” in Proc. of the IEEE Int. Conf. on Software Engineering, 2013, pp. 802–811.
  19. X. Liu and H. Zhong, “Mining StackOverflow for program repair,” in Proc. of the IEEE Int. Conf. on Software analysis, Evolution and Reengineering, 2018, pp. 118–129.
  20. K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “TBar: Revisiting template-based automated program repair,” in Proc. of the ACM Int. Symp. on Software Testing and Analysis, 2019, pp. 31–42.
  21. A. Koyuncu, K. Liu, T. F. Bissyandé, D. Kim, J. Klein, M. Monperrus, and Y. Le Traon, “Fixminer: Mining relevant fix patterns for automated program repair,” Empirical Software Engineering, vol. 25, pp. 1980–2024, 2020.
  22. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in Int. Conf. on Learning Representations, 2015.
  23. T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, “CoCoNut: combining context-aware neural translation models using ensemble for program repair,” in Proc. of the ACM Int. Symp. on Software Testing and Analysis, 2020, pp. 101–114.
  24. Q. Zhu, Z. Sun, Y.-a. Xiao, W. Zhang, K. Yuan, Y. Xiong, and L. Zhang, “A syntax-guided edit decoder for neural program repair,” in Proc. of the ACM Joint Meeting on European Software Engineering Conf. and Symp. on the Foundations of Software Engineering, 2021, pp. 341–353.
  25. N. Jiang, T. Lutellier, and L. Tan, “CURE: Code-aware neural machine translation for automatic program repair,” in Proc. of the IEEE/ACM Int. Conf. on Software Engineering, 2021, pp. 1161–1173.
  26. H. Ye, M. Martinez, and M. Monperrus, “Neural program repair with execution-based backpropagation,” in Proc. of the IEEE/ACM Int. Cof. on Software Engineering, 2022, pp. 1506–1518.
  27. C. S. Xia and L. Zhang, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,” in Proc. of the ACM Joint European Software Engineering Conf. and Symp. on the Foundations of Software Engineering, 2022, pp. 959–971.
  28. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al., “CodeBERT: A pre-trained model for programming and natural languages,” in Proc. of the Conf. on Emperical Methods in Natural Language Processing, 2020, pp. 1536–1547.
  29. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.
  30. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021.
  31. D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “InCoder: A generative model for code infilling and synthesis,” in Proc. of the Int. Conf. on Learning Representations, 2023.
  32. S. D. Kolak, R. Martins, C. Le Goues, and V. J. Hellendoorn, “Patch generation with language models: Feasibility and scaling behavior,” in Proc. of the Int. Conf. on Learning Representations: Deep Learning for Code Workshop, 2022.
  33. J. A. Prenner, H. Babii, and R. Robbes, “Can OpenAI’s Codex fix bugs? an evaluation on QuixBugs,” in Proc. of the Int. Workshop on Automated Program Repair, 2022, pp. 69–75.
  34. C. S. Xia, Y. Wei, and L. Zhang, “Practical program repair in the era of large pre-trained language models,” in Proc. of the Int. Conf. on Software Engineering, 2023, pp. 1482–1494.
  35. N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” Advances in Neural Info. Processing Syst., vol. 36, 2024.
  36. X. Chen, M. Lin, N. Schärli, and D. Zhou, “Teaching large language models to self-debug,” in Proc. of the Int. Conf. on Learning Representations, 2023.
  37. N. Jain, T. Zhang, W.-L. Chiang, J. E. Gonzalez, K. Sen, and I. Stoica, “LLM-assisted code cleaning for training accurate code generators,” arXiv preprint arXiv:2311.14904, 2023.
  38. A. Madaan, A. Shypula, U. Alon, M. Hashemi, P. Ranganathan, Y. Yang, G. Neubig, and A. Yazdanbakhsh, “Learning performance-improving code edits,” in Proc. of the Int. Conf. on Learning Representations, 2024.
  39. Y. Lin, S. Maierhofer, and M. Althoff, “Sampling-based trajectory repairing for autonomous vehicles,” in Proc. of the IEEE Int. Conf. on Intell. Transp. Syst., 2021, pp. 572–579.
  40. Y. Lin and M. Althoff, “Rule-compliant trajectory repairing using satisfiability modulo theories,” in Proc. of the IEEE Intell. Veh. Symp., 2022, pp. 449–456.
  41. S. Maierhofer, Y. Ballnath, and M. Althoff, “Map verification and repairing using formalized map specifications,” in Proc. of the IEEE Int. Conf. on Int. Transp. Syst., 2023, pp. 1277–1284.
  42. A. Pacheck and H. Kress-Gazit, “Physically feasible repair of reactive, linear temporal logic-based, high-level tasks,” IEEE Trans. on Robot., vol. 39, no. 6, pp. 4653–4670, 2023.
  43. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” Advances in Neural Info. Processing Syst., vol. 35, pp. 22 199–22 213, 2022.
  44. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in Proc. of the Int. Conf. on Learning Representations, 2023.
  45. E. Kıcıman, R. Ness, A. Sharma, and C. Tan, “Causal reasoning and large language models: Opening a new frontier for causality,” arXiv preprint arXiv:2305.00050, 2023.
  46. H. Sha, Y. Mu, Y. Jiang, L. Chen, C. Xu, P. Luo, S. E. Li, M. Tomizuka, W. Zhan, and M. Ding, “LanguageMPC: Large language models as decision makers for autonomous driving,” arXiv preprint arXiv:2310.03026, 2023.
  47. L. Wen, D. Fu, X. Li, X. Cai, T. Ma, P. Cai, M. Dou, B. Shi, L. He, and Y. Qiao, “DiLu: A knowledge-driven approach to autonomous driving with large language models,” in Proc. of the Int. Conf. on Learning Representations, 2024.
  48. W. Wang, J. Xie, C. Hu, H. Zou, J. Fan, W. Tong, Y. Wen, S. Wu, H. Deng, Z. Li, et al., “DriveMLM: Aligning multi-modal large language models with behavioral planning states for autonomous driving,” arXiv preprint arXiv:2312.09245, 2023.
  49. C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, P. Luo, A. Geiger, and H. Li, “DriveLM: Driving with graph visual question answering,” arXiv preprint arXiv:2312.14150, 2023.
  50. C. Cui, Y. Ma, X. Cao, W. Ye, and Z. Wang, “Drive as you speak: Enabling human-like interaction with large language models in autonomous vehicles,” in Proc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision, 2024, pp. 902–909.
  51. J. Mao, Y. Qian, H. Zhao, and Y. Wang, “GPT-driver: Learning to drive with GPT,” arXiv preprint arXiv:2310.01415, 2023.
  52. J. Mao, J. Ye, Y. Qian, M. Pavone, and Y. Wang, “A language agent for autonomous driving,” arXiv preprint arXiv:2311.10813, 2023.
  53. Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K. K. Wong, Z. Li, and H. Zhao, “DriveGPT4: Interpretable end-to-end autonomous driving via large language model,” arXiv preprint arXiv:2310.01412, 2023.
  54. L. Chen, O. Sinavski, J. Hünermann, A. Karnsund, A. J. Willmott, D. Birch, D. Maund, and J. Shotton, “Driving with LLMs: Fusing object-level vector modality for explainable autonomous driving,” arXiv preprint arXiv:2310.01957, 2023.
  55. H. Shao, Y. Hu, L. Wang, S. L. Waslander, Y. Liu, and H. Li, “LMDrive: Closed-loop end-to-end driving with large language models,” arXiv preprint arXiv:2312.07488, 2023.
  56. M. Althoff, M. Koschi, and S. Manzinger, “CommonRoad: Composable benchmarks for motion planning on roads,” in Proc. of the IEEE Intell. Veh. Symp., 2017, pp. 719–726.
  57. C. Pek, V. Rusinov, S. Manzinger, M. C. Üste, and M. Althoff, “CommonRoad Drivability Checker: Simplifying the development and validation of motion planning algorithms,” in Proc. of the IEEE Intell. Veh. Symp., 2020, pp. 1013–1020.
  58. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023.
  59. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Info. Processing Syst., vol. 35, pp. 24 824–24 837, 2022.
  60. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Info. Processing Syst., vol. 30, pp. 5998–6008, 2017.
  61. J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, and X. Hu, “Harnessing the power of LLMs in practice: A survey on ChatGPT and beyond,” arXiv preprint arXiv:2304.13712, 2023.
  62. Y. Lin and M. Althoff, “CommonRoad-CriMe: A toolbox for criticality measures of autonomous vehicles,” in Proc. of the IEEE Intell. Veh. Symp., 2023, pp. 1–8.
  63. W. Schwarting, A. Pierson, J. Alonso-Mora, S. Karaman, and D. Rus, “Social behavior for autonomous vehicles,” Proc. of the National Academy of Sciences, vol. 116, no. 50, pp. 24 972–24 978, 2019.
  64. Z. Liu, A. Bahety, and S. Song, “REFLECT: Summarizing robot experiences for failure explanation and correction,” in Proc. of the Conferene on Robot Learning, 2023.
  65. M. Skreta, N. Yoshikawa, S. Arellano-Rubach, Z. Ji, L. B. Kristensen, K. Darvish, A. Aspuru-Guzik, F. Shkurti, and A. Garg, “Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting,” arXiv preprint arXiv:2303.14100, 2023.
  66. M. Pivtoraiko and A. Kelly, “Kinodynamic motion planning with state lattice motion primitives,” in Proc. of the IEEE/RSJ Int. Conf. on Intell. Robots and Syst., 2011, pp. 2172–2179.
  67. M. Klischat, E. I. Liu, F. Holtke, and M. Althoff, “Scenario factory: Creating safety-critical traffic scenarios for automated vehicles,” in Proc. of the IEEE Int. Conf. on Intell. Transp. Syst., 2020, pp. 1–7.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yuanfei Lin (4 papers)
  2. Chenran Li (18 papers)
  3. Mingyu Ding (82 papers)
  4. Masayoshi Tomizuka (261 papers)
  5. Wei Zhan (130 papers)
  6. Matthias Althoff (66 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube