Variational excess risk bound for general state space models (2312.09607v1)
Abstract: In this paper, we consider variational autoencoders (VAE) for general state space models. We consider a backward factorization of the variational distributions to analyze the excess risk associated with VAE. Such backward factorizations were recently proposed to perform online variational learning and to obtain upper bounds on the variational estimation error. When independent trajectories of sequences are observed and under strong mixing assumptions on the state space model and on the variational distribution, we provide an oracle inequality explicit in the number of samples and in the length of the observation sequences. We then derive consequences of this theoretical result. In particular, when the data distribution is given by a state space model, we provide an upper bound for the Kullback-Leibler divergence between the data distribution and its estimator and between the variational posterior and the estimated state space posterior distributions.Under classical assumptions, we prove that our results can be applied to Gaussian backward kernels built with dense and recurrent neural networks.
- Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349.
- Online variational filtering and parameter learning. Advances in Neural Information Processing Systems, 34.
- Inference in Hidden Markov Models (Springer Series in Statistics). Springer-Verlag, Berlin, Heidelberg.
- Additive smoothing error in backward variational inference for general state-space models. arXiv preprint arXiv:2206.00319.
- Consistency of variational Bayes inference for estimation and model selection in mixtures. Electronic Journal of Statistics, 12(2):2995 – 3035.
- Diffusion bridges vector quantized variational autoencoders. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4141–4156. PMLR.
- Consistent estimation of the filtering and marginal smoothing distributions in nonparametric hidden markov models. IEEE Transactions on Information Theory, 63(8):4758–4777.
- Nonlinear time series: Theory, methods and applications with R examples. CRC press.
- A pseudo-marginal sequential Monte Carlo online smoothing algorithm. Bernoulli, 28(4):2606 – 2633.
- Validated variational inference via practical posterior error bounds. In International Conference on Artificial Intelligence and Statistics, pages 1792–1802. PMLR.
- Statistical guarantees for variational autoencoders using pac-bayesian theory. Advances in Neural Information Processing Systems.
- Sequential Monte Carlo smoothing with application to parameter estimation in nonlinear state space models. Bernoulli, 14(1):155 – 179.
- On empirical bayes variational autoencoder: An excess risk bound. In Proceedings of Machine Learning Research, Conference on Learning Theory COLT 2021.
- NVAE: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:19667–19679.
- van de Geer, S. (2000). Empirical processes in M-estimation. Cambridge University Press.
- Wainwright, M. J. (2019). High-dimensional statistics, volume 48 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge. A non-asymptotic viewpoint.