Abstract
Variational autoencoders are a versatile class of deep latent variable models. They learn expressive latent representations of high dimensional data. However, the latent variance is not a reliable estimate of how uncertain the model is about a given input point. We address this issue by introducing a sparse Gaussian process encoder. The Gaussian process leads to more reliable uncertainty estimates in the latent space. We investigate the implications of replacing the neural network encoder with a Gaussian process in light of recent research. We then demonstrate how the Gaussian Process encoder generates reliable uncertainty estimates while maintaining good likelihood estimates on a range of anomaly detection problems. Finally, we investigate the sensitivity to noise in the training data and show how an appropriate choice of Gaussian process kernel can lead to automatic relevance determination.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Braithwaite, D.T., Kleijn, W.B.: Bounded information rate variational autoencoders. In: KDD Deep Learning Day (2018)
Casale, F.P., Dalca, A., Saglietti, L., Listgarten, J., Fusi, N.: Gaussian process prior variational autoencoders. In: Advances in Neural Information Processing Systems, pp. 10369–10380 (2018)
Dai, B., Wipf, D.: Diagnosing and enhancing VAE models. In: International Conference on Learning Representations (2018)
Dai, Z., Damianou, A.C., González, J., Lawrence, N.D.: Variational auto-encoded deep gaussian processes. In: ICLR (Poster) (2016)
Duvenaud, D.: Automatic model construction with Gaussian processes. Ph.D. thesis, University of Cambridge (2014)
Fortuin, V., Rätsch, G., Mandt, S.: Multivariate time series imputation with variational autoencoders. In: 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) (2020)
Ha, D., Schmidhuber, J.: Recurrent world models facilitate policy evolution. In: Advances in Neural Information Processing Systems, pp. 2450–2462 (2018)
He, J., Spokoyny, D., Neubig, G., Berg-Kirkpatrick, T.: Lagging inference networks and posterior collapse in variational autoencoders. In: International Conference on Learning Representations (2018)
Hendrycks, D., Mazeika, M., Dietterich, T.G.: Deep anomaly detection with outlier exposure. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019)
Jankowiak, M., Pleiss, G., Gardner, J.: Parametric gaussian process regressors. In: International Conference on Machine Learning, pp. 4702–4712. PMLR (2020)
Jazbec, M., Ashman, M., Fortuin, V., Pearce, M., Mandt, S., Rätsch, G.: Scalable gaussian process variational autoencoders. In: International Conference on Artificial Intelligence and Statistics, vol. 130, pp. 3511–3519. PMLR (2021)
Johnson, M.J., Duvenaud, D.K., Wiltschko, A., Adams, R.P., Datta, S.R.: Composing graphical models with neural networks for structured representations and fast inference. In: Advances in Neural Information Processing Systems, pp. 2946–2954 (2016)
Kingma, D.P., Welling, M.: An introduction to variational autoencoders. Found. Trends® Mach. Learn. 12(4), 307–392 (2019)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: 2nd International Conference on Learning Representations, ICLR 2014 (2014)
scikit learn: two moons dataset. In: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html, scikit-learn dataset make\_moons (2021). Accessed 2021
LeCun, Y., Cortes, C., Burges, C.: MNIST handwritten digit database. ATT Labs [Online]. http://yann.lecun.com/exdb/mnist 2 (2010)
Nalisnick, E.T., Matsukawa, A., Teh, Y.W., Görür, D., Lakshminarayanan, B.: Do deep generative models know what they don’t know? In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019)
Nash, C., Williams, C.K.: The shape variational autoencoder: a deep generative model of part-segmented 3D objects. In: Computer Graphics Forum. vol. 36, pp. 1–12. Wiley Online Library (2017)
Pang, G., Shen, C., van den Hengel, A.: Deep anomaly detection with deviation networks. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 353–362 (2019)
Rayana, S.: Odds library. In: Stony Brook University, Department of Computer Sciences (2016)
Ren, J., et al.: Likelihood ratios for out-of-distribution detection. In: Advances in Neural Information Processing Systems, pp. 14680–14691 (2019)
Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: International Conference on Machine Learning (2014)
Rubenstein, P., Schölkopf, B., Tolstikhin, I.: Learning disentangled representations with Wasserstein auto-encoders. In: International Conference on Learning Representations (ICLR 2018) Workshops (2018)
Skafte, N., Jørgensen, M., Hauberg, S.: Reliable training and estimation of variance networks. In: Advances in Neural Information Processing Systems, pp. 6326–6336 (2019)
Snelson, E., Ghahramani, Z.: Local and global sparse gaussian process approximations. In: Artificial Intelligence and Statistics, pp. 524–531 (2007)
Titsias, M.: Variational learning of inducing variables in sparse Gaussian processes. In: Artificial Intelligence and Statistics, pp. 567–574 (2009)
Tomczak, J., Welling, M.: VAE with a VampPrior. In: International Conference on Artificial Intelligence and Statistics, pp. 1214–1223 (2018)
Tran, D., Ranganath, R., Blei, D.M.: The variational Gaussian process. In: 4th International Conference on Learning Representations, ICLR 2016 (2016)
Van der Wilk, M., Rasmussen, C.E., Hensman, J.: Convolutional gaussian processes. In: Advances in Neural Information Processing Systems, pp. 2849–2858 (2017)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017)
Zhang, C., Bütepage, J., Kjellström, H., Mandt, S.: Advances in variational inference. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 2008–2026 (2018)
Zhao, S., Song, J., Ermon, S.: Infovae: balancing learning and inference in variational autoencoders. Proc. AAAI Conf. Artif. Intell. 33, 5885–5892 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Bütepage, J., Maystre, L., Lalmas, M. (2021). Gaussian Process Encoders: VAEs with Reliable Latent-Space Uncertainty. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2021. Lecture Notes in Computer Science(), vol 12976. Springer, Cham. https://doi.org/10.1007/978-3-030-86520-7_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-86520-7_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86519-1
Online ISBN: 978-3-030-86520-7
eBook Packages: Computer ScienceComputer Science (R0)
