TARGET: Automated Scenario Generation from Traffic Rules for Testing Autonomous Vehicles (2305.06018v2)

Abstract: Ensuring the safety and robustness of autonomous driving systems (ADSs) is imperative. One of the crucial methods towards this assurance is the meticulous construction and execution of test scenarios, a task often regarded as tedious and laborious. In response to this challenge, this paper introduces TARGET, an end-to-end framework designed for the automatic generation of test scenarios grounded in established traffic rules. Specifically, we design a domain-specific language (DSL) with concise and expressive syntax for scenario descriptions. To handle the natural language complexity and ambiguity in traffic rule descriptions, we leverage a LLM to automatically extract knowledge from traffic rules and convert the traffic rule descriptions to DSL representations. Based on these representations, TARGET synthesizes executable test scenario scripts to render the testing scenarios in a simulator. Comprehensive evaluations of the framework were conducted on four distinct ADSs, yielding a total of 217 test scenarios spread across eight diverse maps. These scenarios identify approximately 700 rule violations, collisions, and other significant issues, including navigation failures. Moreover, for each detected anomaly, TARGET provides detailed scenario recordings and log reports, significantly easing the process of troubleshooting and root cause analysis. Two of these causes have been confirmed by the ADS developers; one is corroborated by an existing bug report from the ADS, and the other one is attributed to the limited functionality of the ADS.