Computational Statistics
Last update: 22 Jul 2026 21:56First version: 28 March 2009
--- I am drawing a somewhat arbitrary terminological divide between "statistical computing", meaning computing environments for statistical data analysis, and "computational statistics", meaning computational methods of special relevance to statistical problems, or tricky or interesting computational problems arising from statistical problems. (One might even call it "numerical methods for statistics", except that some of the most relevant algorithms aren't very numerical.) When I teach statistical computing, only part of what I teach is actually computational statistics...
(Non-computational statistics embraces theory, methodology, and applications or calculations where the computations are too trivial to be an object of study in their own right. The last of these is, I admit, a moving target. Today, taking the mean of a million people's incomes is computationally trivial. [Or, trivial once we have all the incomes recorded, and once we believe the records; measurement is another can of worms.] But it would not be trivial with only pencil and paper, and then one would need to think carefully about how to arrange the work of the human beings doing the addition. [Recall that "computer" was originally a line of work, like "baker" or "sweeper".] How tricky, exactly, a problem needs to be before it graduates to "computational statistics", I decline to state.)
Right now, I am stashing a lot of stuff about estimating models with latent variables here, but it may end up deserving its own notebook.
- See also:
- Bootstrap
- Computation / Automata Theory / Formal Languages
- Cross-Validation
- Data Mining
- Experimental Design
- Indirect Inference
- Monte Carlo
- Optimization
- Programming
- Simulation-Based Inference
- Simulations
- Statistical Computing
- Statistics
- Stochastic Approximation
- Statistical Emulators for Simulation Models
- Recommended, bigger picture:
- Michael I. Jordan, "On statistics, computation and scalability", Bernoulli 19 (2013): 1378--1390
- John F. Monahan, Numerical Methods of Statistics
- Recommended, very miscellaneous close-ups:
- Venkat Chandrasekaran and Michael I. Jordan, "Computational and Statistical Tradeoffs via Convex Relaxation", Proceedings of the National Academy of Sciences (USA) 110 (2013): E1181--E1190, arxiv:1211.1073
- Michael Kane, John W. Emerson, Stephen Weston, "Scalable Strategies for Computing with Massive Data", Journal of Statistical Software 55 (2013): 14
- Norman Matloff, "Software Alchemy: Turning Complex Statistical Computations into Embarrassingly-Parallel Ones", arxiv:1409.5827 [I think this "chunk-and-average" approach can be extended. In the first place, I don't think the restriction to Gaussian asymptotics is essential. Say that \( \Var{\hat{\theta}_n} \rightarrow \Sigma/n \), or really that \( n \Var{\hat{\theta}_n} \rightarrow \Sigma \). If we divide the data up into \( r \) chunks, each of size \( k=n/r \), in Matloff's regime, where \( r \) stays fixed as \( n \) grows, the estimate from each chunk will have a variance matrix of approximately \( r\Sigma/n \). Assuming IID chunks, when we average the chunk-wise estimates we will kill the factor of \( r \). So we'll get the same asymptotic variance from the chunked-and-averaged estimator as from the full-data estimator, even if the limiting distribution is non-Gaussian. Second, under suitable mixing assumptions, one should be able to extend this straightforwardly to dependent data, as in Lunde (2019)'s discussion of sample splitting. Each chunk is entering the large-\( k \) asymptotic regime, and the dependence between chunks is small (limited to the few data points on either side of a chunk boundary). An interesting question would be whether the number of chunks can grow with \( n \).]
- Guosheng Yin, Yanyuan Ma, Faming Liang, and Ying Yuan, "Stochastic Generalized Method of Moments", Journal of Computational and Graphical Statistics 20 (2011): 714--727
- To read:
- Stéphanie Allassonniere, Estelle Kuhn, "Convergent Stochastic Expectation Maximization algorithm with efficient sampling in high dimension. Application to deformable template model estimation", arxiv:1207.5938
- Johannes Blömer, Kathrin Bujna, Daniel Kuntze, "A Theoretical and Experimental Comparison of the EM and SEM Algorithm", arxiv:1310.5034
- Luc Devroye, Non-Uniform Random Variate Generation
- James E. Gentle, Elements of Computational Statistics
- James E. Gentle, Wolfgang Härdle, Yuichi Mori (eds.), Handbook of Computational Statistics
- Mark Girolami, Anne-Marie Lyne, Heiko Strathmann, Daniel Simpson, Yves Atchade, "Playing Russian Roulette with Intractable Likelihoods", arxiv:1306.4032
- Matt Hoffman, David M. Blei, Chong Wang and John Paisley, "Stochastic Variational Inference", arxiv:1206.7051
- National Research Council
- Jeffrey S. Racine, "A Primer on Regression Splines" [PDF preprint]
- Rajesh Ranganath, Sean Gerrish, David Blei, "Black Box Variational Inference", AI Stats 2014
- Thomas J. Santner, Brian J. Williams and William J. Note, Design and Analysis of Computer Experiments
- Sonnenberg et al., "The SHOGUN Machine Learning Toolbox", Journal of Machine Learning Research 11 (2010): 1799--1802
- James C. Spall, Introduction to Stochastic Search and Optimization [Book website]
- Ronald A. Thisted, Elements of Statistical Computing
- Mitchell Watnik, "Early Computational Statistics", Journal of Computational and Graphical Statistics 20 (2011): 811--817