Guided Policy Improvement for Satisfying STL Tasks using Funnel Adaptation

We introduce a sampling-based learning method for solving optimal control problems involving task satisfaction constraints for systems with partially known dynamics. The control problems are defined by a cost to be minimized and a task to be satisfied, given in the language of signal temporal logic (STL). The complex nature of possible tasks generally makes them difficult to satisfy through random exploration, which limits the practical feasibility of the learning algorithm. Recent work has shown, however, that using a controller to guide the learning process by leveraging available knowledge of system dynamics to aid task satisfaction is greatly beneficial for improving the sample efficiency of the method. Motivated by these findings, this work introduces a controller derivation framework which naturally leads to computationally efficient controllers capable of offering such guidance during the learning process. The derived controllers aim to satisfy a set of so-called robustness specifications or funnels that are imposed on the temporal evolutions of the atomic propositions composing the STL task. Ideally, these specifications are prescribed in a way such that their satisfaction would lead to satisfaction of the STL task. In practice, however, such ideal funnels are not necessarily known a priori, and the guidance the controller offers depends on their estimates. This issue is hereby addressed by introducing an adaptation scheme for automatically updating the funnels during the learning procedure, thus diminishing the role of their initial, user-specified values. The effectiveness of the resulting learning algorithm is demonstrated by two simulation case studies.