Neural Network Compression of ACAS Xu Early Prototype is Unsafe: Closed-Loop Verification through Quantized State Backreachability (2201.06626v3)

Abstract: ACAS Xu is an air-to-air collision avoidance system designed for unmanned aircraft that issues horizontal turn advisories to avoid an intruder aircraft. Due the use of a large lookup table in the design, a neural network compression of the policy was proposed. Analysis of this system has spurred a significant body of research in the formal methods community on neural network verification. While many powerful methods have been developed, most work focuses on open-loop properties of the networks, rather than the main point of the system -- collision avoidance -- which requires closed-loop analysis. In this work, we develop a technique to verify a closed-loop approximation of the system using state quantization and backreachability. We use favorable assumptions for the analysis -- perfect sensor information, instant following of advisories, ideal aircraft maneuvers and an intruder that only flies straight. When the method fails to prove the system is safe, we refine the quantization parameters until generating counterexamples where the original (non-quantized) system also has collisions.

• The paper introduces a closed-loop verification technique using state quantization and backreachability to identify unsafe behaviors in NN-compressed ACAS Xu controllers.
• The methodology efficiently partitions and tests millions of state conditions, uncovering scenarios that lead to near mid-air collision outcomes.
• The findings reveal critical limitations in traditional open-loop verification, underscoring the need for robust safety protocols in neural network-based control systems.

Overview of Neural Network Compression in ACAS Xu and Safety Concerns

The paper "Neural Network Compression of ACAS Xu Early Prototype is Unsafe: Closed-Loop Verification through Quantized State Backreachability" by Stanley Bak and Hoang-Dung Tran addresses the safety verification of neural network controllers within the ACAS Xu system, an air-to-air collision avoidance system designed for unmanned aircraft. The paper pivots around a novel methodology that employs state quantization combined with backreachability analysis to evaluate the safety of a compressed neural network approximation of ACAS Xu.

Key Contributions

The authors introduce a technique for safety verification of closed-loop neural network control systems (NNCS) by focusing on state quantization and backreachability instead of direct neural network verification. The research primarily deals with addressing the shortfall of open-loop verification methods which fail to ensure the safety-critical collision avoidance property in practice. Here are the essential components of the approach:

1. Closed-Loop Verification through State Quantization: The paper departs from the conventional approach of input quantization to perform state quantization for closed-loop verification. The reasoning focuses on mathematically reducing the predecessor states of unsafe partitions as a refinement strategy.
2. Quantitative Counterexample Discovery: The method generates counterexamples where the original system exhibits unsafe behavior, emphasizing the inadequacy of the main controllers in specific scenarios.
3. Performance of the Proposed Technique: The authors showcase that by continuously refining quantization parameters, a guaranteed identification of either safe conditions or demonstrative unsafe scenarios is achievable.

Key Findings

In their experimental evaluation, the authors set out to verify the safety of ACAS Xu's neural network policies across the entire possible set of initial states. The testing initially involved larger quantized partitions and strategically scaled down the quantizations to probe system safety up to numeric precision limits.

• Unsafe Behavior Discovery: The validation procedure revealed several unsafe conditions, identifying scenarios where ACAS Xu's advised maneuvers led to near mid-air collision (NMAC) situations. Notably, findings highlighted that under certain initial conditions—even with optimized advisory actions—collisions were inevitable.
• Efficiency and Fidelity: The novel application of state quantization and backreachability allowed testing across millions of partitions swiftly, contrasting significantly with simulation-only or strict open-loop verification, which are often computationally intensive with less informative guarantees.

Implications and Future Directions

The implications of this research extend to the domains of autonomous systems where safety-critical operations rely on neural network controllers. The proposed approach counters some of the limitations faced in neural network verification—such as handling large networks, accommodating complex architectures, and requiring exact floating-point semantics—by focusing on input-output behavior under quantized conditions instead.

• Practical Relevance: For industry practitioners and researchers focusing on the certification of safety-critical systems, this demonstrates a viable path to ensuring safety that is computationally feasible and practically relevant.
• Potential Extensions: Future work could explore refining this method to accommodate real-world nondeterministic behavior like sensor noise or inaccurate execution of maneuvers, and the possibility of providing safety guarantees in these contexts.

This method stands as a significant contribution to the evolving conversation on neural network verification in autonomous systems, emphasizing the necessity of closed-loop thinking in safety-critical tasks such as collision avoidance. By balancing computational feasibility with safety assurance, it presents an attractive approach for high-assurance systems where neural networks play a central role.