Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Maximum Entropy Density Control of Discrete-Time Linear Systems with Quadratic Cost (2309.10662v1)

Published 19 Sep 2023 in math.OC, cs.SY, and eess.SY

Abstract: This paper addresses the problem of steering the distribution of the state of a discrete-time linear system to a given target distribution while minimizing an entropy-regularized cost functional. This problem is called a maximum entropy (MaxEnt) density control problem. Specifically, the running cost is given by quadratic forms of the state and the control input, and the initial and final distributions are Gaussian. We first reveal that our problem boils down to solving two Riccati difference equations coupled through their boundary values. Based on them, we give the closed-form expression of the unique optimal policy. Next, we show that the optimal policy for the density control of the time-reversed system can be obtained simultaneously with the forward-time optimal policy. Finally, by considering the limit where the entropy regularization vanishes, we derive the optimal policy for the unregularized density control problem.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. Y. Chen, T. T. Georgiou, and M. Pavon, “Optimal transport in systems and control,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 4, pp. 89–113, 2021.
  2. J. Ridderhof and P. Tsiotras, “Uncertainty quantfication and control during Mars powered descent and landing using covariance steering,” in 2018 AIAA Guidance, Navigation, and Control Conference, 2018, p. 0611.
  3. K. Okamoto and P. Tsiotras, “Optimal stochastic vehicle path planning using covariance steering,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2276–2281, 2019.
  4. J. Knaup, K. Okamoto, and P. Tsiotras, “Safe high-performance autonomous off-road driving using covariance steering stochastic model predictive control,” IEEE Transactions on Control Systems Technology, 2023, Early Access.
  5. Y. Shirai, D. K. Jha, and A. U. Raghunathan, “Covariance steering for uncertain contact-rich systems,” arXiv preprint arXiv:2303.13382, 2023.
  6. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” in International Conference on Learning Representations, 2021.
  7. V. De Bortoli, J. Thornton, J. Heng, and A. Doucet, “Diffusion Schrödinger bridge with applications to score-based generative modeling,” Advances in Neural Information Processing Systems, vol. 34, pp. 17 695–17 709, 2021.
  8. V. Krishnan and S. Martínez, “Distributed optimal transport for the deployment of swarms,” in 2018 IEEE Conference on Decision and Control (CDC).   IEEE, 2018, pp. 4583–4588.
  9. Y. Chen, T. T. Georgiou, and M. Pavon, “Steering the distribution of agents in mean-field games system,” Journal of Optimization Theory and Applications, vol. 179, pp. 332–357, 2018.
  10. R. Brockett, “Notes on the control of the Liouville equation,” Control of Partial Differential Equations, pp. 101–129, 2012.
  11. K. M. Grigoriadis and R. E. Skelton, “Minimum-energy covariance controllers,” Automatica, vol. 33, no. 4, pp. 569–578, 1997.
  12. E. Collins and R. Skelton, “A theory of state covariance assignment for discrete systems,” IEEE Transactions on Automatic Control, vol. 32, no. 1, pp. 35–41, 1987.
  13. E. Bakolas, “Optimal covariance control for discrete-time stochastic linear systems subject to constraints,” in 2016 IEEE 55th Conference on Decision and Control (CDC).   IEEE, 2016, pp. 1153–1158.
  14. M. Goldshtein and P. Tsiotras, “Finite-horizon covariance control of linear time-varying systems,” in 2017 IEEE 56th Annual Conference on Decision and Control (CDC).   IEEE, 2017, pp. 3606–3611.
  15. F. Liu, G. Rapakoulias, and P. Tsiotras, “Optimal covariance steering for discrete-time linear stochastic systems,” arXiv preprint arXiv:2211.00618, 2022.
  16. Y. Chen, T. T. Georgiou, and M. Pavon, “Optimal steering of a linear stochastic system to a final probability distribution, Part I,” IEEE Transactions on Automatic Control, vol. 61, no. 5, pp. 1158–1169, 2016.
  17. ——, “Optimal steering of a linear stochastic system to a final probability distribution―Part III,” IEEE Transactions on Automatic Control, vol. 63, no. 9, pp. 3112–3118, 2018.
  18. K. Ito and K. Kashima, “Maximum entropy optimal density control of discrete-time linear systems and Schrödinger bridges,” IEEE Transactions on Automatic Control, Early Access.
  19. S. Levine, “Reinforcement learning and control as probabilistic inference: Tutorial and review,” arXiv preprint arXiv:1805.00909, 2018.
  20. J. Kim and I. Yang, “Maximum entropy optimal control of continuous-time dynamical systems,” IEEE Transactions on Automatic Control, vol. 68, no. 4, pp. 2018–2033, 2023.
  21. T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in International Conference on Machine Learning.   PMLR, 2017, pp. 1352–1361.
  22. B. Eysenbach and S. Levine, “Maximum entropy RL (provably) solves some robust RL problems,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=PtSAD3caaA2
  23. M. H. De Badyn, E. Miehling, D. Janak, B. Açıkmeşe, M. Mesbahi, T. Başar, J. Lygeros, and R. S. Smith, “Discrete-time linear-quadratic regulation via optimal transport,” in 2021 60th IEEE Conference on Decision and Control (CDC).   IEEE, 2021, pp. 3060–3065.
  24. A. Terpin, N. Lanzetti, and F. Dörfler, “Dynamic programming in probability spaces via optimal transport,” arXiv preprint arXiv:2302.13550, 2023.
  25. A. Halder and E. D. Wendel, “Finite horizon linear quadratic Gaussian density regulator with Wasserstein terminal cost,” in 2016 American Control Conference (ACC).   IEEE, 2016, pp. 7249–7254.
  26. K. Okamoto, M. Goldshtein, and P. Tsiotras, “Optimal covariance control for stochastic systems under chance constraints,” IEEE Control Systems Letters, vol. 2, no. 2, pp. 266–271, 2018.
  27. Z. Yi, Z. Cao, E. Theodorou, and Y. Chen, “Nonlinear covariance control via differential dynamic programming,” in 2020 American Control Conference (ACC).   IEEE, 2020, pp. 3571–3576.
  28. J. Ridderhof, K. Okamoto, and P. Tsiotras, “Nonlinear uncertainty control with iterative covariance steering,” in 2019 IEEE 58th Conference on Decision and Control (CDC).   IEEE, 2019, pp. 3484–3490.
  29. T. Mikami, “Monge’s problem with a quadratic cost by the zero-noise limit of h-path processes,” Probability Theory and Related Fields, vol. 129, no. 2, pp. 245–260, 2004.
  30. T. Mikami and M. Thieullen, “Optimal transportation problem by stochastic optimal control,” SIAM Journal on Control and Optimization, vol. 47, no. 3, pp. 1127–1139, 2008.
  31. S. Chan, G. Goodwin, and K. Sin, “Convergence properties of the Riccati difference equation in optimal filtering of nonstabilizable systems,” IEEE Transactions on Automatic Control, vol. 29, no. 2, pp. 110–118, 1984.
  32. H. K. Wimmer and M. Pavon, “A comparison theorem for matrix Riccati difference equations,” Systems & Control Letters, vol. 19, no. 3, pp. 233–239, 1992.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now