Spatio-Temporal Variational Gaussian Processes (2111.01732v1)

Abstract: We introduce a scalable approach to Gaussian process inference that combines spatio-temporal filtering with natural gradient variational inference, resulting in a non-conjugate GP method for multivariate data that scales linearly with respect to time. Our natural gradient approach enables application of parallel filtering and smoothing, further reducing the temporal span complexity to be logarithmic in the number of time steps. We derive a sparse approximation that constructs a state-space model over a reduced set of spatial inducing points, and show that for separable Markov kernels the full and sparse cases exactly recover the standard variational GP, whilst exhibiting favourable computational properties. To further improve the spatial scaling we propose a mean-field assumption of independence between spatial locations which, when coupled with sparsity and parallelisation, leads to an efficient and accurate method for large spatio-temporal problems.

• The paper introduces linear temporal scaling through ST-VGP using natural gradients for efficient parallel filtering and smoothing.
• It employs a sparse approximation with a state-space model over spatial inducing points to maintain accuracy with large datasets.
• The framework supports non-Gaussian likelihoods and demonstrates improved RMSE and NLPD in applications like air quality and crime prediction.

Spatio-Temporal Variational Gaussian Processes: A Structured Approach to Scalable Inference

The paper "Spatio-Temporal Variational Gaussian Processes" presents a novel inference framework that enhances the scalability of Gaussian Processes (GPs) applied to spatio-temporal data. Through the integration of spatio-temporal filtering and natural gradient variational inference, the authors offer a method that maintains the robustness of traditional variational GPs while significantly improving computational efficiency.

Key Contributions

1. Linear Temporal Scaling: The approach introduces the Spatio-Temporal Variational Gaussian Process (ST-VGP), which scales linearly with the number of time steps. This improvement is facilitated by applying natural gradients, enabling efficient parallel filtering and smoothing which altogether reduces temporal complexity to logarithmic terms concerning the number of time steps.
2. Sparse Approximation: The paper extends the ST-VGP to a sparse variant (ST-SVGP) by employing a state-space model over spatial inducing points. This variant retains the accuracy of classical variational GPs while supporting a greater number of inducing points and maintaining favorable computational properties.
3. Mean-Field Approximation: To enhance spatial scalability, the method incorporates a mean-field assumption that treats spatial locations as independent. This assumption, combined with the sparse model's inherent parallelization, further optimizes performance for large-scale spatio-temporal problems.
4. Applicability to Non-Conjugate Likelihoods: The authors demonstrate how these techniques apply beyond Gaussian likelihoods, extending the method's application breadth to settings where data distributions may diverge from Gaussian assumptions.

Numerical Results and Implications

The paper provides compelling numerical results that underscore the method's efficiency and performance. For example, ST-VGP and ST-SVGP show superior computational scalability over baseline methods such as the Spectral Mixture Kernel and Sparse Variational GPs (SVGP), especially as temporal resolution increases. This scalability is crucial for real-world applications, such as air quality modeling and urban crime prediction, as illustrated in the experiments.

ST-SVGP outperforms SVGP in both RMSE and negative log predictive density (NLPD), especially when the temporal horizon is extensive. Furthermore, these models deliver improved predictive performance by effectively capturing rapid temporal dynamics, which are typically challenging for standard GP models due to computational burdens.

Future Development and Impact

The resemblance of this framework to existing variational GP methodologies opens pathways for several extensions. Applying this approach to multi-task scenarios or deep Gaussian processes could benefit from the scalability improvements demonstrated here. Furthermore, the ability to perform efficiently on both CPU and GPU platforms broadens the accessibility of GPs for large datasets in diverse environments, potentially leading to more widespread adoption in industry and academia.

The paper also contributes to the ongoing dialogue concerning the appropriate application of machine learning models to societal issues such as crime forecasting. The authors acknowledge potential societal impacts and suggest responsible application, with considerations of biases and approximation limitations.

Overall, the ST-VGP and ST-SVGP frameworks offer a substantial step forward in scalable Gaussian process inference, particularly for spatio-temporal domains. This contribution is valuable both in theory, by addressing the inherent computational challenges of GPs, and in practice, by facilitating the deployment of GPs in high-dimensional, real-world data applications.