RiskBench: A Scenario-based Benchmark for Risk Identification (2312.01659v2)

Abstract: Intelligent driving systems aim to achieve a zero-collision mobility experience, requiring interdisciplinary efforts to enhance safety performance. This work focuses on risk identification, the process of identifying and analyzing risks stemming from dynamic traffic participants and unexpected events. While significant advances have been made in the community, the current evaluation of different risk identification algorithms uses independent datasets, leading to difficulty in direct comparison and hindering collective progress toward safety performance enhancement. To address this limitation, we introduce \textbf{RiskBench}, a large-scale scenario-based benchmark for risk identification. We design a scenario taxonomy and augmentation pipeline to enable a systematic collection of ground truth risks under diverse scenarios. We assess the ability of ten algorithms to (1) detect and locate risks, (2) anticipate risks, and (3) facilitate decision-making. We conduct extensive experiments and summarize future research on risk identification. Our aim is to encourage collaborative endeavors in achieving a society with zero collisions. We have made our dataset and benchmark toolkit publicly on the project page: https://hcis-lab.github.io/RiskBench/

• The paper’s main contribution is RiskBench, a framework enabling systematic risk identification through a detailed scenario taxonomy and augmentation pipeline.
• It benchmarks ten algorithms using metrics like Risk Localization, Progressive Increasing Cost (PIC), and Influenced Ratio (IR) to assess planning awareness.
• The study highlights challenges in temporal consistency and advocates for advanced spatial-temporal models to enhance risk prediction in autonomous driving.

RiskBench: A Scenario-based Benchmark for Risk Identification

The paper "RiskBench: A Scenario-based Benchmark for Risk Identification" introduces a comprehensive benchmarking framework designed to address the challenge of risk identification in intelligent driving systems (IDS). This effort is in response to the fragmented landscape of risk evaluation, where diverse algorithms have been evaluated on disparate datasets, inhibiting direct comparisons and the collective progress towards improving IDS safety performance.

The primary contribution of the paper is RiskBench—a large-scale scenario-based benchmark tailored for risk identification. The authors constructed a detailed scenario taxonomy and augmentation pipeline, enabling the systematic collection of ground truth risks across a range of scenarios. This taxonomy comprises four interaction types—Interactive, Collision, Obstacle, and Non-interactive—aiming to cover diverse risk situations encountered in real-world traffic environments. The dataset comprises 6916 scenarios generated using the CARLA simulator, encompassing variations in actor behaviors, road structures, traffic densities, and weather conditions.

Evaluation Metrics and Benchmarking

The paper emphasizes three critical evaluation metrics: (1) Risk Localization, (2) Risk Anticipation, and (3) Planning Awareness. Risk Localization measures the precision and recall of risk detection, while Risk Anticipation assesses the ability to predict risks before they materialize, using a novel metric called Progressive Increasing Cost (PIC). Planning Awareness is evaluated through a planning-aware metric, Influenced Ratio (IR), which measures the impact of risk identification on decision-making processes.

The authors benchmarked ten risk identification algorithms across four categories:

1. Rule-based Methods: These include simplistic approaches where risks are detected based on predefined distance thresholds.
2. Trajectory Prediction and Collision Checking: Methods like Kalman Filter and Social-GAN predict future trajectories and check for overlaps to infer risks.
3. Collision Anticipation: Vision-based methods such as DSA and RRL predict imminent collisions, encouraging early anticipation.
4. Behavior Prediction-based Methods: BP and BCP identify risks based on predicted influences on the ego vehicle’s behavior, leveraging graph attention networks and causal inference.

Key Findings

The benchmarking reveals several critical insights:

• Trajectory Prediction Methods demonstrate reasonable performance in Collision scenarios but suffer from high false alarm rates in Non-interactive scenarios due to ineffective trajectory overlap mechanisms.
• Collision Anticipation methods, particularly RRL, outperform in risk localization due to direct supervision from ground truth risks. However, their temporal consistency is lacking.
• Behavior Prediction-based Methods exhibit suboptimal recall due to weak supervision signals, although causal inference in BCP shows promise.

The paper highlights that high-performing trajectory predictors do not guarantee better risk identification, underscoring the need for holistic system perspectives. Furthermore, the temporal consistency of risk predictions remains a significant challenge, particularly for vision-based methods.

Practical and Theoretical Implications

Practically, RiskBench provides a standardized framework for evaluating and comparing risk identification algorithms, fostering collaborative efforts aimed at achieving zero-collision mobility. The introduction of the planning-aware metric, IR, aligns the risk identification process with the ultimate goal of safe path planning and decision-making in IDS.

Theoretically, the findings call for advancements in spatial-temporal modeling and object representation learning to enhance the robustness and reliability of risk identification processes. The integration of visual information, although imperative, necessitates improvements in temporal consistency to ensure stable and reliable risk predictions over time.

Future Directions

Future research directions as suggested by the paper include:

• Temporal Consistency: Developing robust spatial-temporal models, such as transformers, to improve the temporal consistency of risk predictions.
• Planning-aware Risk Identification: Jointly optimizing risk identification and planning processes to create algorithms that are both predictive and actionable in real-world driving scenarios.
• Advanced Risk-aware Planners: Utilizing advanced planners that can better interact with risk predictions, enhancing the overall safety performance of IDS.

In summary, RiskBench represents a significant step towards standardized risk identification evaluation in intelligent driving systems. The insights garnered from this benchmark provide valuable guidance for future research efforts, aiming to bridge the gap between risk prediction and practical decision-making, ultimately contributing to the realization of safer and more reliable autonomous driving technologies.