Oxford: Clarendon Press, 1992

Statistics being the continuation of methodology by other means, it's natural to look to it for advice on how to do experiments. We do not hope in vain: there is a large literature on this subject, going back to the great R. A. Fisher if not before, embodied in books with titles like Design of Experiments and Experimental Design. For the most part these are cookbooks: compilations of standard procedures and the associated statistical machinery (estimators, tests, etc.). These designs are often better than what unaided scientists-at-the-bench come up with, but the books rarely have any discussion of what, in general, makes for a good or bad design, or criteria by which designs may be compared. Still, the fact that the traditional designs

The general problem situation in this book is as follows. We have a bunch
of independent variables, and a single scalar dependent variable, the response.
We assume that the dependence takes a specified functional form, plus noise,
and we want to learn about the parameters of the function, either estimating
them by least squares, or testing hypothesis about the parameter values. An
experimental design is, for these purposes, a distribution (discrete or
continuous) over the independent variables, at which we measure the response.
In the case of linear models (linear in the parameters, that is, so including
polynomial regressions), *F* is the ``extended design matrix'', the
*i*th row of which consists of the independent variables at the
*i*th data-point. The ``information matrix'' is *F*^T*F,*
where *F*^T is *F* transpose. The variances of the
least-squares estimates of the model's parameters are proportional to the
elements of the inverse of the information matrix --- which is why, when Fisher
invented it, he called it the *information* matrix. For nonlinear
models, the *ij*th element of the information matrix consists of the
partial derivative of the response function with respect to the *i*th
parameter, multiplied by the partial with respect to the *j*th
parameter, and then integrated over the design density. (Those interested in
the meaning and uses of the information matrix in general should see Kullback's
Information Theory and
Statistics.)

Information matrix, however obtained, in hand, we now need to calculate the optimal design density. To do this we need to pick an optimality criterion. The most basic one is to maximize the determinant of the information matrix, since this will minimize the elements of its inverse and so the variances of our parameter estimates. There is a remarkable result, known as the General Equivalence Theorem, which states that designs which maximize the determinant also minimize the maximum of the (appropriately re-scaled) variance of the estimated response function. The General Equivalence Theorem and its use in finding optimal designs are the subjects of ch. 9, but you need to have read the first eight chapters to follow its notation and results. Most of those chapters will constitute a review of basic topics (e.g. linear least squares regression) for the book's audience, and they lose little from skimming.

There are a very large number of other optimality criteria, depending on the
particular task at hand; these form the subject of ch. 10. By a vexing
tradition to which our authors adhere, they are named by letters of dubious
mnemonic value (indeed the subject is sometimes called ``alphabetical
optimality theory''): maximizing the determinant is *D-*optimality,
minimaxing the variance of predictions is *G-*optimality. There are
*A-,* *c-,* *C-,* *D_A-,* *D_\beta-,*
*D_s-,* *E-,* *L-,* *T-* and *V-*
optimalities as well, and there are equivalence theorems connecting many of
these. Doubtless there are more lurking in the literature, but here for once
it would have been a blessing to add to the jargon, even coining new terms if
necessary.

Ch. 11 describes generic properties of *D-*optimal designs, showing
how a published experimental paper might've been improved by following their
methods. Ch. 15 provides algorithms for their exact construction. (Fortran 77
(!) code is provided as an appendix.) Beyond that, chs. 12--22 treat separate
special topics, and can be read in almost any order. The most useful ones for
me were ch. 17, on ``design augmentation,'' or how to recover from having
chosen a bad design initially, ch. 18, on non-linear models, and ch. 20, on
model discrimination. This last turns on two linked questions: given that you
have several different kinds of models (polynomial vs. exponential, say), how
should you choose experimental points so as to distinguish between them? Given
that you already have some data, where should you put the *next*
experimental point so as to learn the most about which is the right set of
models? Despite being a rather obvious and important topic, and the one which
sent me to the experimental design literature in the first place, it's
unnaturally hard to find references to it. Of five recent texts on the subject
I consulted, this is the only one which did more than name it (some didn't even
do that), and in fact solved my problem for
me.

Readers will need a thorough grounding in statistics, and acquaintance with the cookbook experimental design literature will help. Scientists and engineers doing large experiments or simulations will find the time they spend learning the methods and calculating their designs well-spent, as will computer scientists interested in machine learning. It's well-suited to self-study, and would make a good text with the addition of problems. As with most monographs, the price is outrageous for individuals (failing the xerox test by more than a factor of three), but not so high that technical libraries should have any hesitation over buying it.

xvi + 328 pp., numerous graphs and tables, bibliography (plus ``further reading'' at the end of each chapter), author index, subject index (very thin, e.g. no entry for ``information matrix'')

Probability and Statistics

Currently in print as a hardback (reprinted with unspecified corrections, 1996), ISBN 0-19-852254-1, US$75. LoC QA279.A82

6 November 1999; typo about information matrix corrected next day

Thanks to Margaret Alexander for procuring a copy of this book