Abstract
This paper is a contribution to the methodology of fully Bayesian inference in a multivariate Gaussian mixture model using the reversible jump Markov chain Monte Carlo algorithm. To follow the constraints of preserving the first two moments before and after the split or combine moves, we concentrate on a simplified multivariate Gaussian mixture model, in which the covariance matrices of all components share a common eigenvector matrix. We then propose an approach to the construction of the reversible jump Markov chain Monte Carlo algorithm for this model. Experimental results on several data sets demonstrate the efficacy of our algorithm.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Andrieu C., de Freitas N., and Doucet A. 2001. Robust full Bayesian learning for radial basis networks. Neural Computation 13: 2359–2407.
Andrieu C. and Doucet A. 1999. Joint Bayesian model selection and estimation of noisy sinusoids via reversible jump MCMC. IEEE Transactions Signal Processing 47: 2667–2676.
Attial H. 2000. A Variational Bayesian framework for graphical models. Advances in Neural Information Processing Systems 12: 21–30.
Banfield J.D. and Raftery A.E. 1993. Model-based Gaussian and non-Gaussian clustering. Biometrics 49: 803–821.
Bensmail H. and Celeux G. 1996. Regularized Gaussian discriminant analysis through eigenvalue decomposition. Journal of the American Statistical Association 91(436): 1743–1748.
Bensmail H., Celeux G., Raftery A.E., and Robert C.P. 1997. Inference in model-based cluster analysis. Statistics and Computing 7: 1–10.
Cappé O., Robert C.P., and Rydén T. 2003. Reversible jump, birth-anddeath and more general continuous time Markov Chain Monte Carlo samplers. Journal of the Royal Statistical Society, B 65: 679–700.
Celeux G., Hurn M., and Robert C.P. 2000. Computational and inferential difficulties with mixture posterior distributions. Journal of the American Statistical Association 95(451): 957–970.
Dempster A.P., Laird N.M., and Rubin D.B. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society Series B 39(1): 1–38.
Diebolt J. and Robert C.P., 1994, Estimation of finite mixture distribution through Bayesian sampling. Journal of the Royal Statistical Society Series B 56(2): 363–375.
Gilks W.R., Richardson S., and Spiegelhalter D.J. 1996. Markov Chain Monte Carlo in practice. Chapman and Hall, London.
Golub G.H. and Van Loan, C.F. 1996. Matrix Computations, 3rd edition. The Johns Hopkins University Press, Baltimore.
Green P.J. 1994. Discussion on representations of knowledge in complex systems (by U. Grenander). Journal of the Royal Statistial Society Series B 56: 589–590.
Green P.J. 1995. Reversible jump Markov Chain Monte Carlo computation and Bayesian model determination. Biometrika 82: 711–732.
Grenander U. and Miller M.I. 1994. Representations of knowledge in complex systems (with discussion). Journal of the Royal Statistical Society Series B 56: 549–603.
Hastie T. and Tibshirani R. 1996. Discriminant analysis by Gaussian mixtures. Journal of the Royal Statistical Society Series B 58: 155–176.
Holmes C.C. and Mallick B.K. 1998. Bayesian radial basis functions of variable dimension. Neural Computation 10: 1217–1233.
Larocque J.R. and Reilly J.P. 2002. Reversible jump MCMC for joint detection and estimation of sources in colored noise. IEEE Transactions on Signal Processing 50(2): 231–240.
McLachlan G. and Krishnan T. 1997. The EM Algorithm and Extensions. John Wiley and Sons, New York.
McLachlan G. and Peel D. 2000. Finite Mixture Models. John Wiley and Sons, New York.
Redner R.A. and Walker H.F. 1984. Mixture densities, maximum likelihood and the EM algorithm. SIAM Review 26(2): 195–239.
Richardson S. and Green P.J. 1997. On Bayesian analysis of mixtures with an unknown number of components (with discussion). Journal of the Royal Statistical Society Series B 59: 731–792.
Richardson S. and Green P.J. 1998. Corrigendum: On Bayesian analysis of mixtures with an unknown number of components (with discussion). Journal of the Royal Statistical Society Series B 60: 661.
Robert C.P. 1996. Mixtures of distributions: Inference and estimation. In Gilks W.R., Richardson S., and Spiegelhalter D.J. (eds.), Markov Chain Monte Carlo in Practice. Chapman and Hall, London. Chapt. 24, pp. 441–464.
Robert C.P., Rydén T., and Titterington D.M. 2000. Bayesian inference in hidden Markov models through the reversible jump Markov Chain Monte Carlo method. Journal of the Royal Statistical Society Series B 62: 57–75.
Sato M. 2001. On-line model selection based on the variational Bayes. Neural Computation 13(7): 1649–1681.
Stephens M. 2000a. Bayesian analysis of mixtures with an unknown number of components-An llternative to reversible jump methods. Annals of Statistics 28: 40–74.
Stephens M. 2000b. Dealing with label switching in mixture models. Journal of the Royal Statistical Society Series B 62: 795–809.
Titterington D.M., Smith A.F.M., and Makov U.E. 1985. Statistical Analysis of Finite Mixture Distributions, Wiley Series in Probability and Mathematical Statistics. Wiley, New York.
Zhang Z., Chen C., Sun J., and Chan K.L. 2003. EM algorithms for Gaussian mixtures with split-and-merge operation. Pattern Recognition 36: 1973–1983.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Zhang, Z., Chan, K.L., Wu, Y. et al. Learning a multivariate Gaussian mixture model with the reversible jump MCMC algorithm. Statistics and Computing 14, 343–355 (2004). https://doi.org/10.1023/B:STCO.0000039484.36470.41
Issue Date:
DOI: https://doi.org/10.1023/B:STCO.0000039484.36470.41