The Bactra Review: Occasional and eclectic book reviews by Cosma Shalizi   103

Optimum Experimental Designs

by A. C. Atkinson and A. N. Donev

Oxford Statistical Science Series, vol. 8
Oxford: Clarendon Press, 1992
Statistics being the continuation of methodology by other means, it's natural to look to it for advice on how to do experiments. We do not hope in vain: there is a large literature on this subject, going back to the great R. A. Fisher if not before, embodied in books with titles like Design of Experiments and Experimental Design. For the most part these are cookbooks: compilations of standard procedures and the associated statistical machinery (estimators, tests, etc.). These designs are often better than what unaided scientists-at-the-bench come up with, but the books rarely have any discussion of what, in general, makes for a good or bad design, or criteria by which designs may be compared. Still, the fact that the traditional designs are better than intuitive or haphazard ones suggests that, if only we could pin down such criteria, we could study the merits of experimental designs. In fact, we can, and can even devise optimal experiments for a given purpose. Thus optimal experimental design theory, to which this book is both an introduction and a fairly complete reference.

The general problem situation in this book is as follows. We have a bunch of independent variables, and a single scalar dependent variable, the response. We assume that the dependence takes a specified functional form, plus noise, and we want to learn about the parameters of the function, either estimating them by least squares, or testing hypothesis about the parameter values. An experimental design is, for these purposes, a distribution (discrete or continuous) over the independent variables, at which we measure the response. In the case of linear models (linear in the parameters, that is, so including polynomial regressions), F is the ``extended design matrix'', the ith row of which consists of the independent variables at the ith data-point. The ``information matrix'' is F^TF, where F^T is F transpose. The variances of the least-squares estimates of the model's parameters are proportional to the elements of the inverse of the information matrix --- which is why, when Fisher invented it, he called it the information matrix. For nonlinear models, the ijth element of the information matrix consists of the partial derivative of the response function with respect to the ith parameter, multiplied by the partial with respect to the jth parameter, and then integrated over the design density. (Those interested in the meaning and uses of the information matrix in general should see Kullback's Information Theory and Statistics.)

Information matrix, however obtained, in hand, we now need to calculate the optimal design density. To do this we need to pick an optimality criterion. The most basic one is to maximize the determinant of the information matrix, since this will minimize the elements of its inverse and so the variances of our parameter estimates. There is a remarkable result, known as the General Equivalence Theorem, which states that designs which maximize the determinant also minimize the maximum of the (appropriately re-scaled) variance of the estimated response function. The General Equivalence Theorem and its use in finding optimal designs are the subjects of ch. 9, but you need to have read the first eight chapters to follow its notation and results. Most of those chapters will constitute a review of basic topics (e.g. linear least squares regression) for the book's audience, and they lose little from skimming.

There are a very large number of other optimality criteria, depending on the particular task at hand; these form the subject of ch. 10. By a vexing tradition to which our authors adhere, they are named by letters of dubious mnemonic value (indeed the subject is sometimes called ``alphabetical optimality theory''): maximizing the determinant is D-optimality, minimaxing the variance of predictions is G-optimality. There are A-, c-, C-, D_A-, D_\beta-, D_s-, E-, L-, T- and V- optimalities as well, and there are equivalence theorems connecting many of these. Doubtless there are more lurking in the literature, but here for once it would have been a blessing to add to the jargon, even coining new terms if necessary.

Ch. 11 describes generic properties of D-optimal designs, showing how a published experimental paper might've been improved by following their methods. Ch. 15 provides algorithms for their exact construction. (Fortran 77 (!) code is provided as an appendix.) Beyond that, chs. 12--22 treat separate special topics, and can be read in almost any order. The most useful ones for me were ch. 17, on ``design augmentation,'' or how to recover from having chosen a bad design initially, ch. 18, on non-linear models, and ch. 20, on model discrimination. This last turns on two linked questions: given that you have several different kinds of models (polynomial vs. exponential, say), how should you choose experimental points so as to distinguish between them? Given that you already have some data, where should you put the next experimental point so as to learn the most about which is the right set of models? Despite being a rather obvious and important topic, and the one which sent me to the experimental design literature in the first place, it's unnaturally hard to find references to it. Of five recent texts on the subject I consulted, this is the only one which did more than name it (some didn't even do that), and in fact solved my problem for me.

Readers will need a thorough grounding in statistics, and acquaintance with the cookbook experimental design literature will help. Scientists and engineers doing large experiments or simulations will find the time they spend learning the methods and calculating their designs well-spent, as will computer scientists interested in machine learning. It's well-suited to self-study, and would make a good text with the addition of problems. As with most monographs, the price is outrageous for individuals (failing the xerox test by more than a factor of three), but not so high that technical libraries should have any hesitation over buying it.

xvi + 328 pp., numerous graphs and tables, bibliography (plus ``further reading'' at the end of each chapter), author index, subject index (very thin, e.g. no entry for ``information matrix'')
Probability and Statistics
Currently in print as a hardback (reprinted with unspecified corrections, 1996), ISBN 0-19-852254-1, US$75. LoC QA279.A82
6 November 1999; typo about information matrix corrected next day
Thanks to Margaret Alexander for procuring a copy of this book