Data-driven Estimation of the Algebraic Riccati Equation for the Discrete-Time Inverse Linear Quadratic Regulator Problem (2402.06180v1)
Abstract: In this paper, we propose a method for estimating the algebraic Riccati equation (ARE) with respect to an unknown discrete-time system from the system state and input observation. The inverse optimal control (IOC) problem asks, ``What objective function is optimized by a given control system?'' The inverse linear quadratic regulator (ILQR) problem is an IOC problem that assumes a linear system and quadratic objective function. The ILQR problem can be solved by solving a linear matrix inequality that contains the ARE. However, the system model is required to obtain the ARE, and it is often unknown in fields in which the IOC problem occurs, for example, biological system analysis. Our method directly estimates the ARE from the observation data without identifying the system. This feature enables us to economize the observation data using prior information about the objective function. We provide a data condition that is sufficient for our method to estimate the ARE. We conducted a numerical experiment to demonstrate that our method can estimate the ARE with less data than system identification if the prior information is sufficient.
- K. Mombaur, A. Truong, and J.-P. Laumond, “From human to humanoid locomotion–an inverse optimal control approach”, Autonomous Robots, vol. 28, pp. 369–383, 2010.
- W. Li, E. Todorov, and D. Liu, “Inverse Optimality Design for Biological Movement Systems”, IFAC Proceedings Volumes, vol. 44, no. 1, pp. 9662–9667, 2011.
- H. El-Hussieny, A.A. Abouelsoud, S.F.M. Assal, and S.M. Megahed, “Adaptive learning of human motor behaviors: An evolving inverse optimal control approach”, Engineering Applications of Artificial Intelligence, vol. 50, pp. 115–124, 2016
- R.M. Alexander, “The gaits of bipedal and quadrupedal animals”, International Journal of Robotics Research, 3(2), pp. 49–59, 1984.
- R.E. Kalman, “When Is a Linear Control System Optimal?”, Journal of Basic Engineering, 86(1), pp. 51–60, 1964.
- B. Molinari, “The stable regulator problem and its inverse”, IEEE Transactions on Automatic Control, vol. 18, no. 5, pp. 454–459, 1973.
- P. Moylan and B. Anderson, “Nonlinear regulator theory and an inverse optimal control problem”, IEEE Transactions on Automatic Control, vol. 18, no. 5, pp. 460–465, 1973.
- A.Y. Ng and S. Russell, “Algorithms for Inverse Reinforcement Learning”, Proceedings of the Seventeenth International Conference on Machine Learning, pp. 663–670, 2000.
- N. Ab Azar, A. Shahmansoorian, and M. Davoudi, “From inverse optimal control to inverse reinforcement learning: A historical review”, Annual Reviews in Control, vol. 50, pp. 119–138, 2020.
- M.C. Priess, R. Conway, J. Choi, J.M. Popovich, and C. Radcliffe, “Solutions to the Inverse LQR Problem With Application to Biological Systems Analysis”, IEEE Transaction on Control Systems Technology, vol. 23, no. 2, pp. 770–777, 2015.
- M. Herman, T. Gindele, J. Wagner, F. Schmitt, and W. Burgard, “Inverse reinforcement learning with simultaneous estimation of rewards and dynamics”. Artificial intelligence and statistics, PMLR, pp. 102–110, 2016.
- N. Aghasadeghi and T. Bretl, “Maximum entropy inverse reinforcement learning in continuous state spaces with path integrals”. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1561–1566, 2011.