Abstract
In this article we investigate the relationship between the EM algorithm and the Gibbs sampler. We show that the approximate rate of convergence of the Gibbs sampler by Gaussian approximation is equal to that of the corresponding EM-type algorithm. This helps in implementing either of the algorithms as improvement strategies for one algorithm can be directly transported to the other. In particular, by running the EM algorithm we know approximately how many iterations are needed for convergence of the Gibbs sampler. We also obtain a result that under certain conditions, the EM algorithm used for finding the maximum likelihood estimates can be slower to converge than the corresponding Gibbs sampler for Bayesian inference. We illustrate our results in a number of realistic examples all based on the generalized linear mixed models.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Breslow, N. E. and Clayton, D. G. (1993) Approximate inference in generalized linear mixed models. J. Amer. Statist. Assoc., 88, 9-25.
Box, G. E. P. and Tiao, G. C. (1992) Bayesian Inference in Sta-tistical Analysis. London: Addison-Wesley.
Crowder, M. J. (1978) Beta-binomial ANOVA for proportions. Appl. Statist., 27, 34-37.
Dellaportas, P. and Smith, A. F. M. (1993) Bayesian inference for generalised linear and proportional hazards models via Gibbs sampling. Appl. Statist., 42, 443-59.
Dempster, A. P., Laird, N. M. and Rubin, D. B. (1977) Maximum likelihood from incomplete data via the EM algorithm (with discussion). J.R. Statist. Soc. B, 39, 1-38.
Gamerman, D. (1997) Markov Chain Monte Carlo. London: Chapman and Hall.
Gelfand, A. E., Sahu, S. K. and Carlin, B. P. (1995) Efficient parametrization for normal linear mixed models. Biometrika, 82, 479-88.
Gelman, A. (1997) Discussion of the paper by Meng and van Dyk. J. R. Statist. Soc. B, 59, 554.
Gilks, W. R. (1997) Discussion of the paper by Meng and van Dyk. J. R. Statist. Soc. B, 59, 543-45.
Gilks, W. R., Best, N. G. and Tan, K. K. C. (1995) Adaptive Rejection Metropolis Sampling within Gibbs Sampling. Appl. Statist., 455-72.
Gilks, W. R., Richardson, S. and Spiegelhalter, D. G. (1996) Markov Chain Monte Carlo In Practice, London: Chapman and Hall.
Jacobs, D. (Eds.) (1977) The State of the Art in Numerical Anal-ysis. London: Academic Press.
Liu, C., Rubin, D. B. and Wu, Y. (1998) Parameter expansion for EM acceleration: the PX-EM algorithm. Biometrika, 85, 973-979.
Liu, J. S. (1996) Fraction of Missing Information and Conver-gence Rate of Data Augmentation. Preprint.
McCullagh, P. and Nelder, J. A. (1989) Generalized Linear Models. London: Chapman and Hall.
McCulloch, C. E. (1997) Maximum Likelihood Algorithms for Generalized Linear Mixed Models. J. Amer. Statist. Assoc., 92, 162-70.
Meng, X.-L. (1994) On the Rate of Convergence of the ECM Algorithm. Ann. Statist., 22, 326-339.
Meng, X.-L. and Rubin, D. B. (1993) Maximum likelihood esti-mation via the ECM algorithm: a general framework. Bio-metrika, 80, 267-78.
Meng, X-L. and van Dyk, D. (1997) The EM algorithm-an old folk-song sung to a fast new tune (with discussion). J. R. Statist. Soc. B, 59, 511-67.
Roberts, G. O. and Sahu, S. K. (1996) Rate of Convergence of the Gibbs Sampler by Gaussian Approximation. Technical Re-port, University of Cambridge.
Roberts, G. O. and Sahu, S. K. (1997) Updating Schemes, Cor-relation Structure, Blocking and Parameterisation for the Gibbs Sampler. J. R. Statist. Soc. B, 59, 291-317.
Roberts, G. O. and Sahu, S. K. (1997b) Discussion of the paper by Meng and van Dyk. J. R. Statist. Soc. B, 59, 558-59.
Sahu, S. K. (1997) Bayesian Estimation and Model Choice in Item Response Models. Available from http://www.cf.ac.uk=/ uwcc/maths/sahu/.
Schafer, D. W. (1987) Covariate measurement error in generalized linear models. Biometrika, 74, 385-91.
Steele, B. M. (1996) A Modified EM Algorithm for Estimation in Generalized Mixed Models. Biometrics, 52, 1295-310.
Spiegelhalter, D. J., Thomas, A. and Best, N. G. (1996) Com-putation on Bayesian graphical models. In Bayesian Statis-tics 5, (Eds. J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith). Oxford: Oxford University Press, pp. 407-26.
Tanner, M. A. (1996) Tools for Statistical Inference. Springer-Verlag: Heidelberg.
van Dyk, D. and Meng, X.-L. (1997) On the Orderings and Groupings of Conditional Maximizations within ECM-type Algorithms. J. Comput. Graph. Statist., 6, 202-23.
Wedderburn, R. W. M. (1976) On the existence and uniqueness of the maximum likelihood estimates for certain generalized linear models. Biometrika, 63, 27-32.
Whitmore, A. S. and Keller J. B. (1989) Approximations for Regression With Covariate Measurement Error. J. Amer. Statist. Assoc., 83, 1057-66.
Rights and permissions
About this article
Cite this article
Sahu, S.K., Roberts, G.O. On convergence of the EM algorithmand the Gibbs sampler. Statistics and Computing 9, 55–64 (1999). https://doi.org/10.1023/A:1008814227332
Issue Date:
DOI: https://doi.org/10.1023/A:1008814227332