Mixture Models (in Statistics)

Last update: 21 Apr 2025 21:17
First version: 10 May 2012

---

Yet Another Inadequate Placeholder, because I don't feel like re-writing a textbook chapter.

One thing I will say here, which I don't emphasize in the textbook (but maybe should?) is that I think mixtures of models are probably under-used in applications. This is because there's usually little theoretical reason to think that there is a true regression line (or whatever) for the whole population; there will often be multiple sub-populations, each with its own appropriate model...

See also:
• Clustering
• Density Estimation
• Ensemble Methods in Machine Learning
• Factor Models
• Model Selection
• Statistical Inference in Markov and Hidden Markov Models
• Statistics
• Topic Models

Recommended, big picture:
• David J. Bartholomew, Latent Variable Models and Factor Analysis
• Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning

Recommended, close ups:
• Animashree Anandkumar, Daniel Hsu, Sham M. Kakade, "A Method of Moments for Mixture Models and Hidden Markov Models", arxiv:1203.0683
• David M. Blei, Andrew Y. Ng and Michael I. Jordan, "Latent Dirichlet Allocation", Journal of Machine Learning Research 3 (2003): 993--1022 [See under Topic Models]
• Jiahua Chen and Xianming Tan, "Inference for Multivariate Normal Mixtures", arxiv:0805.3906 [Little of the machinery looks like it depends on using mixtures of normals]
• Jeffrey W. Miller and Matthew T. Harrison
  • "A simple example of Dirichlet process mixture inconsistency for the number of components", pp. 199--206 of Burges et al. (eds), NIPS 2013, arxiv:1301.2708
  • "Inconsistency of Pitman-Yor Process Mixtures for the Number of Components", Journal of Machine Learning Research 15 (2014): 3333--3370
• Babak Shahbaba, "Discovering Hidden Structure Using Mixture Models: Application to Nonlinear Time Series Processes", Studies in Nonlinear Dynamics and Econometrics 13:2 (2009): 5 [Not as groundbreaking as it makes itself sound, but still interesting]
• Christopher Tosh, Sanjoy Dasgupta, "Maximum Likelihood Estimation for Mixtures of Spherical Gaussians is NP-hard", Journal of Machine Learning Research 18 (2018): 175

Modesty forbids me to recommend:
• The chapter on mixture models in Advanced Data Analysis from an Elementary Point of View

To read:
• Elizabeth S. Allman, Catherine Matias, John A. Rhodes, "Identifiability of parameters in latent structure models with many observed variables", Annals of Statistics 37 (2009): 3099--3132, arxiv:0809.5032
• A. Anandkumar, D. Hsu, S. M. Kakade, "Learning High-Dimensional Mixtures of Graphical Models", arxiv:1203.0697
• Karim Anaya-Izquierdo, Paul Marriott, "Local mixture models of exponential families", Bernoulli 13 (2007): 623--640, arxiv:0709.0447
• Florentina Bunea, Alexandre B. Tsybakov, Marten H. Wegkamp, Adrian Barbu, "Spades and Mixture Models", Annals of Statistics 38 (2010): 2525--2558, arxiv:0901.2044
• Olivier Cappé, Eric Moulines, "On-line expectation-maximization algorithm for latent data models", Journal of the Royal Statistical Society B 71 (2009): 593--613, arxiv:0712.4273 p
• Gilles Celeux, Stéphane Chrétien, Florence Forbes, "A Component-wise EM Algorithm for Mixtures", Journal of Computational and Graphical Statistics 10 (2001): 697--712, arxiv:1201.5913
• Natalie Doss, Yihong Wu, Pengkun Yang, Harrison H. Zhou, "Optimal estimation of high-dimensional location Gaussian mixtures", arxiv:2002.05818
• Elisabeth Gassiat, Ramon Van Handel
  • "The local geometry of finite mixtures", arxiv:1202.3482
  • "Consistent order estimation and minimal penalties", arxiv:1002.1280
• Lancelot F. James, David J. Marchette and Carey Priebe, "Consistent estimation of mixture complexity", Annals of Statistics 29 (2001): 1281--1296
• Hiroyuki Kasahara and Katsumi Shimotsu, "Non-parametric identification and estimation of the number of components in multivariate mixtures", Journal of the Royal Statistical Society: Series B forthcoming
• Mikhail Kovtun, Igor Akushevich, Kenneth G. Manton and H. Dennis Tolley
  • "Linear Latent Structure Analysis: Mixture Distribution Models with Linear Constraints", math.PR/0507025
  • "A New Efficient Algorithm for Construction of LLS Models", math.PR/0507021
• Yanyuan Ma, Jeffrey D. Hart and Raymond J. Carroll, "Density Estimation in Several Populations With Uncertain Population Membership", Journal of the American Statistical Association 106 (2011): 1180--1192
• Geoffrey J. McLachlan and David Peel, Finite Mixture Models
• Volodymyr Melnykov and Ranjan Maitra, "Finite mixture models and model-based clustering", Statistics Surveys 4 (2010): 80--116
• Frank Nielsen, "$k$-MLE: A fast algorithm for learning statistical mixture models", arxiv:1203.5181
• M. Pavlic and M. J. van der Laan, "Fitting of mixtures with unspecified number of components using cross validation distance estimate", Computational Statistics and Data Analysis 41 (2003): 413--428
• Titterington
• Guenther Walther, "The Average Likelihood Ratio for Large-scale Multiple Testing and Detecting Sparse Mixtures", arxiv:1111.0328