Abstract
Variational methods are becoming increasingly popular for inference and learning in probabilistic models. By providing bounds on quantities of interest, they offer a more controlled approximation framework than techniques such as Laplace’s method, while avoiding the mixing and convergence issues of Markov chain Monte Carlo methods, or the possible computational intractability of exact algorithms. In this paper we review the underlying framework of variational methods and discuss example applications involving sigmoid belief networks, Boltzmann machines and feed-forward neural networks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
M. I. Jordan, Z. Gharamani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. In M. I. Jordan, editor, Learning in Graphical Models. Kluwer, 1998.
R. M. Neal and G. E. Hinton. A new view of the EM algorithm that justifies incremental and other variants. In M. I. Jordan, editor, Learning in Graphical Models. Kluwer, 1998.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, B, 39(1):1–38, 1977.
L. K. Saul, T. Jaakkola, and M. I. Jordan. Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research, 4:61–76, 1996.
T. Jaakkola and M. I. Jordan. Approximating posteriors via mixture models. In M. I. Jordan, editor, Learning in Graphical Models. Kluwer, 1998.
C. M. Bishop, N. Lawrence, T. Jaakkola, and M. I. Jordan. Approximating posterior distributions in belief networks using mixtures. In Advances in Neural Information Processing Systems, volume 10, 1998.
B. Frey, N. Lawrence, and C. M. Bishop. Markovian inference in belief networks, 1998. Draft technical report.
D. Ackley, G. Hinton, and T. Sejnowski. A learning algorithm for Boltzmann machines. Cognitive Science, 9:147–169, 1985.
C. Peterson and J. R. Anderson. A mean field learning algorithm for neural networks. Complex Systems, 1:995–1019, 1987.
N. Lawrence, C. M. Bishop, and M. Jordan. Mixture representations for inference and learning in Boltzmann machines. In Uncertainty in Artificial Intelligence. Morgan Kaufmann, 1998.
C. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
D. J. C. MacKay. A practical Bayesian framework for back-propagation networks. Neural Computation, 4(3):448–472, 1992.
G. E. Hinton and D. van Camp. Keeping neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, pages 5–13, 1993.
D. Barber and C. M. Bishop. Variational learning in Bayesian neural networks. In C. M. Bishop, editor, Generalization in Neural Networks and Machine Learning. Springer Verlag, 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1998 Springer-Verlag London
About this paper
Cite this paper
Bishop, C.M. (1998). Variational Learning in Graphical Models and Neural Networks. In: Niklasson, L., Bodén, M., Ziemke, T. (eds) ICANN 98. ICANN 1998. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-1599-1_2
Download citation
DOI: https://doi.org/10.1007/978-1-4471-1599-1_2
Published:
Publisher Name: Springer, London
Print ISBN: 978-3-540-76263-8
Online ISBN: 978-1-4471-1599-1
eBook Packages: Springer Book Archive