Modular Control Architecture for Safe Marine Navigation: Reinforcement Learning and Predictive Safety Filters
(2312.01855)Abstract
Many autonomous systems face safety challenges, requiring robust closed-loop control to handle physical limitations and safety constraints. Real-world systems, like autonomous ships, encounter nonlinear dynamics and environmental disturbances. Reinforcement learning is increasingly used to adapt to complex scenarios, but standard frameworks ensuring safety and stability are lacking. Predictive Safety Filters (PSF) offer a promising solution, ensuring constraint satisfaction in learning-based control without explicit constraint handling. This modular approach allows using arbitrary control policies, with the safety filter optimizing proposed actions to meet physical and safety constraints. We apply this approach to marine navigation, combining RL with PSF on a simulated Cybership II model. The RL agent is trained on path following and collision avpodance, while the PSF monitors and modifies control actions for safety. Results demonstrate the PSF's effectiveness in maintaining safety without hindering the RL agent's learning rate and performance, evaluated against a standard RL agent without PSF.
Overview
-
The paper discusses the integration of Reinforcement Learning (RL) and Predictive Safety Filters (PSF) for safe marine navigation.
-
PSFs assess and allow only safe control actions from the RL agent, which is trained using Cybership II simulations.
-
The PSF is designed as an Optimal Control Problem minimally altering the RL agent's actions while ensuring safety.
-
The study incorporates moving obstacles into the PSF, enabling risk-aware navigation by predicting motions of other vessels.
-
Simulation results show PSF's effectiveness in avoiding collisions, suggesting its potential in advancing safe ASV operations.
Modular Control Architecture for Safe Marine Navigation
Introduction
The rapid advancement of AI has led to significant developments in autonomous systems, particularly in marine navigation. Autonomous Surface Vessels (ASVs), which operate in complex marine environments, must tackle non-linear and uncertain dynamics influenced by time-varying disturbances such as waves, currents, and wind. Ensuring the safety and stability of such systems, especially when using Reinforcement Learning (RL) for control, is a crucial challenge. Recently, Predictive Safety Filters (PSF) have surfaced as a method to ensure safety without explicitly integrating constraints within the RL algorithm.
Predictive Safety Filtering Combined with Reinforcement Learning
Innovatively, the research unites Reinforcement Learning (RL) with a Predictive Safety Filter to address path following and collision avoidance in marine navigation. The PSF assesses the control actions proposed by the RL agent, permitting only those that maintain the system's safety. The RL agent is trained using a simulated model of a supply ship, known as Cybership II, across a broad spectrum of environmental conditions. The effectiveness of the combined PSF/RL system is shown through simulations. It was found that integrating a PSF successfully retains safety without deterring the learning performance or progress of the RL agent.
Safe Marine Navigation Strategy
The study formulates the PSF as an Optimal Control Problem, which ensures minimal alterations to the agent's proposed actions while satisfying both physical and safety constraints. Specifically for ASVs, the research innovates by also accounting for moving obstacles like other vessels. This is achieved by incorporating their projected motions within the PSF to compute safe trajectories. Additionally, the approach includes a collision risk metric, which utilizes a blend of ship motion predictions and environmental perceptions to achieve risk-aware autonomous navigation.
Results and Implications for Future Research
The safety performance of the combined PSF/RL scheme is assessed against traditional RL-based control strategies. Results from numerous simulated scenarios reveal that the PSF is highly effective in preventing collisions. This work exemplifies the potential of integrating PSFs with RL algorithms to create robust and safe autonomous navigation systems for ASVs. It opens new avenues for future research, such as improving the realism of training scenarios and exploring multi-agent collaborative behaviors for collective safety and efficiency in marine traffic management.
Create an account to read this summary for free: