Experimental Design in Statistics

Last update: 06 May 2026 14:05
First version: 24 April 2026

---

Yet Another Inadequate Placeholder

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of. --- R. A. Fisher, 1938 (p. 17)

This is especially inadequate in light of the fact that this is one of the areas where my chosen profession has had the biggest impact on the world...

Two topics of particular interest:

Model discrimination experiments

That is, designing experiments so as to discriminate between competing classes of model, experiments for model selection. Adaptation to data issues here.

Computer experiments

To what extent are computer experiments really experiments, so that experimental-design principles apply?

See also:
• Causal Inference
• Experiments on Social Networks
• Scientific Method
• Statistics

Recommended, big picture:
• A. C. Atkinson and A. N. Donev, Optimum Experimental Design

Recommended, close-ups:
• Susan A. Murphy, "An experimental design for the development of adaptive treatment strategies", Statistics in Medicine 24 (2005): 1455--1481
• Liam Paninski, "Asymptotic theory of information-theoretic experimental design", Neural Computation 17 (2005): 1480--1507 [Preprint]

Recommended, historical interest [Actually, these books are probably still fine for a huge range of experimenters, though we find computation vastly easier these days]:
• D. J. Finney, Experimental Design and Its Statistical Basis
• Oscar Kempthorne, The Design and Analysis of Experiments

Modesty forbids me to recommend:
• Sabina J. Sloman, Daniel M. Oppenheimer, Stephen B. Broomell, and CRS, "Characterizing the robustness of Bayesian adaptive experimental designs to active learning bias", arxiv:2205.13698

To read:
• R. A. Bailey, Design of Comparative Experiments
• D. R. Cox, Planning of Experiments
• Josep Ginebra, "On the Measure of the Information in a Statistical Experiment", Bayesian Analysis 2 (2007): 167--212
• Mead et al., Statistical Principles for the Design of Experiments: Applications to Real Experiments