Low-rank attention augmented Gaussian processes for multivariate data analysis

We have developed an efficient low-rank attention-augmented Gaussian processes (LAAGP) model that effectively combines accuracy with a reduction in the computational costs associated with transformer attention and Gaussian processes (GP). This model addresses the limitations of standard GP models, such as poor covariance function expressiveness for long-range multivariate forecasting and inadequate data representation capacity. LAAGP is a powerful forecasting technique that integrates the transformer self-attention mechanism with GP. The framework features a transformer encoder that processes the input embeddings to extract essential information, using positional and variable encoding along with relative embeddings to enhance attention scores. The GP decoder, known for its flexibility and reliable uncertainty estimates, has been adapted to predict the system's evolution over time. This enhancement allows the model to achieve a balance between computational efficiency, predictive accuracy, and uncertainty quantification, thereby improving performance on intricate tasks like long-range time-series forecasting. Our model has been evaluated on several benchmark regression and classification datasets.