Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Stability-Certified On-Policy Data-Driven LQR via Recursive Learning and Policy Gradient (2403.05367v1)

Published 8 Mar 2024 in eess.SY and cs.SY

Abstract: In this paper, we investigate a data-driven framework to solve Linear Quadratic Regulator (LQR) problems when the dynamics is unknown, with the additional challenge of providing stability certificates for the overall learning and control scheme. Specifically, in the proposed on-policy learning framework, the control input is applied to the actual (unknown) linear system while iteratively optimized. We propose a learning and control procedure, termed RELEARN LQR, that combines a recursive least squares method with a direct policy search based on the gradient method. The resulting scheme is analyzed by modeling it as a feedback-interconnected nonlinear dynamical system. A Lyapunov-based approach, exploiting averaging and singular perturbations theory for nonlinear systems, allows us to provide formal stability guarantees for the whole interconnected scheme. The effectiveness of the proposed strategy is corroborated by numerical simulations, where RELEARN LQR is deployed on an aircraft control problem, with both static and drifting parameters.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. S. J. Bradtke, B. E. Ydstie, and A. G. Barto, “Adaptive linear quadratic control using policy iteration,” in Proceedings of 1994 American Control Conference-ACC’94, vol. 3, pp. 3475–3479, IEEE, 1994.
  2. B. Recht, “A tour of reinforcement learning: The view from continuous control,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 2, pp. 253–279, 2019.
  3. D. Kleinman, “On an iterative technique for Riccati equation computations,” IEEE Transactions on Automatic Control, vol. 13, no. 1, pp. 114–115, 1968.
  4. B. Pang, T. Bian, and Z.-P. Jiang, “Robust policy iteration for continuous-time linear quadratic regulation,” IEEE Transactions on Automatic Control, vol. 67, no. 1, pp. 504–511, 2021.
  5. V. G. Lopez, M. Alsalti, and M. A. Müller, “Efficient off-policy Q-learning for data-based discrete-time LQR problems,” IEEE Transactions on Automatic Control, 2023.
  6. C. Qin, H. Zhang, and Y. Luo, “Online optimal tracking control of continuous-time linear systems with unknown dynamics by using adaptive dynamic programming,” International Journal of Control, vol. 87, no. 5, pp. 1000–1009, 2014.
  7. K. Krauth, S. Tu, and B. Recht, “Finite-time analysis of approximate policy iteration for the linear quadratic regulator,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  8. H. Modares, F. L. Lewis, and Z.-P. Jiang, “Optimal output-feedback control of unknown continuous-time linear systems using off-policy reinforcement learning,” IEEE Transactions on Cybernetics, vol. 46, no. 11, pp. 2401–2410, 2016.
  9. B. Pang, T. Bian, and Z.-P. Jiang, “Data-driven finite-horizon optimal control for linear time-varying discrete-time systems,” in 2018 IEEE Conference on Decision and Control (CDC), pp. 861–866, IEEE, 2018.
  10. C. Possieri and M. Sassano, “Q-learning for continuous-time linear systems: A data-driven implementation of the Kleinman algorithm,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 10, pp. 6487–6497, 2022.
  11. T. Bian and Z.-P. Jiang, “Value iteration and adaptive dynamic programming for data-driven adaptive optimal control design,” Automatica, vol. 71, pp. 348–360, 2016.
  12. I. Ziemann, A. TSIAMis, H. Sandberg, and N. Matni, “How are policy gradient methods affected by the limits of control?,” in 2022 IEEE 61st Conference on Decision and Control (CDC), pp. 5992–5999, IEEE, 2022.
  13. B. Kiumarsi, F. L. Lewis, and Z.-P. Jiang, “H∞subscript𝐻H_{\infty}italic_H start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT control of linear discrete-time systems: Off-policy reinforcement learning,” Automatica, vol. 78, pp. 144–152, 2017.
  14. C. De Persis and P. Tesi, “Formulas for data-driven control: Stabilization, optimality, and robustness,” IEEE Transactions on Automatic Control, vol. 65, no. 3, pp. 909–924, 2019.
  15. H. J. Van Waarde, J. Eising, H. L. Trentelman, and M. K. Camlibel, “Data informativity: a new perspective on data-driven analysis and control,” IEEE Transactions on Automatic Control, vol. 65, no. 11, pp. 4753–4768, 2020.
  16. M. Rotulo, C. De Persis, and P. Tesi, “Data-driven linear quadratic regulation via semidefinite programming,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 3995–4000, 2020.
  17. M. Rotulo, C. De Persis, and P. Tesi, “Online learning of data-driven controllers for unknown switched linear systems,” Automatica, vol. 145, p. 110519, 2022.
  18. C. De Persis and P. Tesi, “Low-complexity learning of linear quadratic regulators from noisy data,” Automatica, vol. 128, p. 109548, 2021.
  19. F. Dörfler, P. Tesi, and C. De Persis, “On the certainty-equivalence approach to direct data-driven LQR design,” IEEE Transactions on Automatic Control, 2023.
  20. J. Berberich, A. Koch, C. W. Scherer, and F. Allgöwer, “Robust data-driven state-feedback design,” in 2020 American Control Conference (ACC), pp. 1532–1538, IEEE, 2020.
  21. H. J. van Waarde, M. K. Camlibel, and M. Mesbahi, “From noisy data to feedback controllers: Nonconservative design via a matrix s-lemma,” IEEE Transactions on Automatic Control, vol. 67, no. 1, pp. 162–175, 2020.
  22. C. De Persis and P. Tesi, “Learning controllers for nonlinear systems from data,” Annual Reviews in Control, p. 100915, 2023.
  23. S. Dean, S. Tu, N. Matni, and B. Recht, “Safely learning to control the constrained linear quadratic regulator,” in IEEE American Control Conference (ACC), pp. 5582–5588, 2019.
  24. H. Mania, S. Tu, and B. Recht, “Certainty equivalence is efficient for linear quadratic control,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  25. M. Ferizbegovic, J. Umenberger, H. Hjalmarsson, and T. B. Schön, “Learning robust lq-controllers using application oriented exploration,” IEEE Control Systems Letters, vol. 4, no. 1, pp. 19–24, 2019.
  26. A. Iannelli, M. Khosravi, and R. S. Smith, “Structured exploration in the finite horizon linear quadratic dual control problem,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 959–964, 2020.
  27. S. Formentin and A. Chiuso, “Core: Control-oriented regularization for system identification,” in 2018 IEEE Conference on Decision and Control (CDC), pp. 2253–2258, IEEE, 2018.
  28. F. Dörfler, J. Coulson, and I. Markovsky, “Bridging direct and indirect data-driven control formulations via regularizations and relaxations,” IEEE Transactions on Automatic Control, vol. 68, no. 2, pp. 883–897, 2022.
  29. B. Hu, K. Zhang, N. Li, M. Mesbahi, M. Fazel, and T. Başar, “Toward a theoretical foundation of policy optimization for learning control policies,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 6, pp. 123–158, 2023.
  30. J. Bu, A. Mesbahi, M. Fazel, and M. Mesbahi, “LQR through the lens of first order methods: Discrete-time case,” arXiv preprint arXiv:1907.08921, 2019.
  31. M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, “Global convergence of policy gradient methods for the linear quadratic regulator,” in International conference on machine learning, pp. 1467–1476, PMLR, 2018.
  32. K. Zhang, B. Hu, and T. Basar, “Policy optimization for H2subscript𝐻2H_{2}italic_H start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT linear control with H∞subscript𝐻H_{\infty}italic_H start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT robustness guarantee: Implicit regularization and global convergence,” in Learning for Dynamics and Control, pp. 179–190, PMLR, 2020.
  33. H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanović, “Convergence and sample complexity of gradient methods for the model-free linear–quadratic regulator problem,” IEEE Transactions on Automatic Control, vol. 67, no. 5, pp. 2435–2450, 2021.
  34. H. Mohammadi, M. Soltanolkotabi, and M. R. Jovanović, “On the linear convergence of random search for discrete-time LQR,” IEEE Control Systems Letters, vol. 5, no. 3, pp. 989–994, 2020.
  35. Y. Abbasi-Yadkori and C. Szepesvári, “Regret bounds for the adaptive control of linear quadratic systems,” in Proceedings of the 24th Annual Conference on Learning Theory, pp. 1–26, JMLR Workshop and Conference Proceedings, 2011.
  36. A. Cohen, T. Koren, and Y. Mansour, “Learning linear-quadratic regulators efficiently with only T𝑇\sqrt{T}square-root start_ARG italic_T end_ARG regret,” in International Conference on Machine Learning, pp. 1300–1309, PMLR, 2019.
  37. A. Cassel, A. Cohen, and T. Koren, “Logarithmic regret for learning linear quadratic regulators efficiently,” in International Conference on Machine Learning, pp. 1328–1337, PMLR, 2020.
  38. M. Akbari, B. Gharesifard, and T. Linder, “Achieving logarithmic regret via hints in online learning of noisy LQR systems,” in IEEE 61st Conference on Decision and Control (CDC), pp. 4700–4705, 2022.
  39. S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu, “On the sample complexity of the linear quadratic regulator,” Foundations of Computational Mathematics, vol. 20, no. 4, pp. 633–679, 2020.
  40. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, “Adaptive optimal control for continuous-time linear systems based on policy iteration,” Automatica, vol. 45, no. 2, pp. 477–484, 2009.
  41. Y. Jiang and Z.-P. Jiang, “Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics,” Automatica, vol. 48, no. 10, pp. 2699–2704, 2012.
  42. C. Possieri and M. Sassano, “Value iteration for continuous-time linear time-invariant systems,” IEEE Transactions on Automatic Control, 2022.
  43. B. Kiumarsi, F. L. Lewis, M.-B. Naghibi-Sistani, and A. Karimpour, “Optimal tracking control of unknown discrete-time linear systems using input-output measured data,” IEEE transactions on cybernetics, vol. 45, no. 12, pp. 2770–2779, 2015.
  44. M. Simchowitz and D. Foster, “Naive exploration is optimal for online LQR,” in Proceedings of the 37th International Conference on Machine Learning (H. D. III and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research, pp. 8937–8948, PMLR, 13–18 Jul 2020.
  45. B. D. Anderson and J. B. Moore, Optimal control: linear quadratic methods. Courier Corporation, 2007.
  46. J. Bu, A. Mesbahi, and M. Mesbahi, “On topological properties of the set of stabilizing feedback gains,” IEEE Transactions on Automatic Control, vol. 66, no. 2, pp. 730–744, 2020.
  47. C. S. Turner, “Recursive discrete-time sinusoidal oscillators,” IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 103–111, 2003.
  48. J. C. Willems, P. Rapisarda, I. Markovsky, and B. L. De Moor, “A note on persistency of excitation,” Systems & Control Letters, vol. 54, no. 4, pp. 325–329, 2005.
  49. E.-W. Bai and S. S. Sastry, “Persistency of excitation, sufficient richness and parameter convergence in discrete time adaptive control,” Systems & control letters, vol. 6, no. 3, pp. 153–163, 1985.
  50. A. Isidori, Lectures in feedback design for multivariable systems. Springer, 2017.
  51. L. Grüne, E. D. Sontag, and F. R. Wirth, “Asymptotic stability equals exponential stability, and iss equals finite energy gain—if you twist your eyes,” Systems & Control Letters, vol. 38, no. 2, pp. 127–134, 1999.
  52. P. Kapasouris, M. Athans, and G. Stein, “Design of feedback control systems for unstable plants with saturating actuators,” in Proc. IFAC Symp. on Nonlinear Control System Design, pp. 302–307, Pergamon Press, 1990.
  53. A. Padoan, G. Scarciotti, and A. Astolfi, “A geometric characterization of the persistence of excitation condition for the solutions of autonomous systems,” IEEE Transactions on Automatic Control, vol. 62, no. 11, pp. 5666–5677, 2017.
  54. E.-W. Bai, L.-C. Fu, and S. S. Sastry, “Averaging analysis for discrete time and sampled data adaptive systems,” IEEE Transactions on Circuits and Systems, vol. 35, no. 2, pp. 137–148, 1988.
  55. R. M. Johnstone, C. R. Johnson Jr, R. R. Bitmead, and B. D. Anderson, “Exponential convergence of recursive least squares with exponential forgetting factor,” Systems & Control Letters, vol. 2, no. 2, pp. 77–82, 1982.
  56. W. M. Haddad and V. Chellaboina, “Nonlinear dynamical systems and control,” in Nonlinear Dynamical Systems and Control, Princeton university press, 2011.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now