The Bactra Review: Occasional and eclectic book reviews by Cosma Shalizi   147

The Myth of the Rational Market

A History of Risk, Reward, and Delusion on Wall Street

by Justin Fox

New York: Harper Business, 2009

Twilight of the Efficient Markets

[This review appeared in American Scientist 97:6 (November--December 2009: 504]

This is an account — popular but thorough — of the roots, rise, triumph and ongoing fall of the theory of efficient markets in finance. This school of thought is an exemplary specimen of a type of social science that flourished after World War II: It has mathematical models at its center, has supposedly been empirically validated by statistical analyses, is indifferent to history and to institutions, and takes as an axiom that people are intelligent, farsighted and greedy. Unlike many economic theories, the efficient-market school has been influential beyond academia. It helped reshape ideas about how companies should be run, how executives should be paid, and indeed how the economy should be regulated (or not) to promote the general welfare. (In comic-book form: A mild-mannered social science by day, at night efficient-market theory puts on a cloak of ideology and struggles for the Capitalist Way.) The theory contributed, arguably, to setting up the crisis that has gripped the world economy since 2007. Its story is of much more than just scholarly interest.

The founding principles of efficient-market theory are easily described. The assumption on which all else rests is that, unless one has private knowledge, there is no way to profit from financial markets without risk. Admittedly, some securities are as safe as humans can make them, and they do pay returns, notably bonds issued by the U.S. Treasury. (Whatever else he does, Uncle Sam pays his debts.) The rates of return on these next-to-riskless instruments are, however, extremely low. Other kinds of securities pay returns that are, on average, higher, but these returns are more variable — there is some nontrivial risk of getting back less than the amount one has invested, or getting back nothing at all. The basic idea of efficient-market theory is that anything that pays higher returns than the risk-free rate must also be more risky. There should be no opportunities for arbitrage (making money from riskless trades that exploit price discrepancies between markets). Moreover, the trade-off between risk and return must be the same across different assets: If stock A was as risky as stock B, but A paid lower returns than B, people would sell A, lowering its price and raising its rate of return, and buy B, with the opposite effect — arbitrage in which the arbitrageurs put themselves out of business.

Classically, there is a very specific idea — the "capital asset pricing model" — about how the risk-reward trade-off is supposed to go, at least for stocks. The return on a portfolio of several stocks is an average of those stocks' returns. More diversified portfolios are less exposed to the risks peculiar to individual companies, leaving only the risks common to the whole corporate sector. The returns for each stock, then, are supposed to combine a firm-specific term, alpha, and the firm's correlation with the economy as a whole, beta. Higher returns, in this scheme, compensate for higher betas — that is, for risk that cannot be mitigated by diversification.

One corollary is important enough to count as a principle itself. Legend relates that J. P. Morgan, when asked what the stock market would do the next day, replied, "It will fluctuate." Someone who could predict these fluctuations (or their absence) could increase their returns with no extra risk. Therefore, says efficient-market theory, securities prices are unpredictable. Current prices are supposed to be optimal forecasts, on the basis of currently available data, of the present value of future returns, because changes in optimal forecasts are, themselves, unpredictable. (If you know that tomorrow your forecast of next year's gasoline price will be higher than today's forecast by $1, you should raise your current forecast.) As Paul Samuelson put it, "properly anticipated prices fluctuate randomly." The efficient-market hypothesis, as a technical term, is the claim that market prices cannot be predicted, either from past prices alone or from past prices combined with other publicly available information. One of the early triumphs of the school was the demonstration that stock prices look very much indeed like random walks.

By now it is clear that the efficient-market school has been interestingly ambivalent about arbitrage and arbitrageurs. On the one hand, the assumption that there are no opportunities for arbitrage is the basis for all calculations. On the other hand, for all prices to be exactly right at all times is too much to ask, and arbitrageurs have been invoked as the restoring force pulling prices back to equilibrium. There is something almost Taoist about the assertion that arbitrage is so powerful and ubiquitous that it cannot be seen. Alas for paradoxes, this view is actually incoherent, as the effort that goes into figuring out what prices should be can only be compensated if prices are not fully efficient. But this leaves open the possibility that prices are close to efficient, without systematic deviations.

A vast superstructure was erected on these foundations, beginning in the 1950s and really taking off in the 1960s and 1970s. Particularly impressive wings of that edifice were devoted to the design of portfolios to balance risk against return and to the valuation of derivative securities ("contingent claims" or bets on the value of other securities), especially options to buy or sell stocks at given prices by given dates. As Fox notes, scholars of finance achieved acclaim, and were awarded substantial consulting fees, for solving pricing problems that by hypothesis were already being solved by the markets themselves! (Donald Mackenzie's An Engine, Not a Camera explores this paradox in depth.)

By the 1980s and 1990s, these ideas had led to changes in the way the investment industry worked, new concepts of corporate governance and new kinds of financial firms, which aimed to systematically identify arbitrage opportunities — deviations from what the theory said prices should be — and to earn a profit even as they eliminated those opportunities. More diffusely, the academic prestige of efficient-market theory provided, at the least, rhetorical support for deregulating markets, especially financial markets, and delegating more and more authority to them. This was aided by a conflation — subscribed to by many scholars — between those markets having informationally efficient prices (that is, unpredictable ones) and those markets allocating capital efficiently (directing savings to where the money can be used most profitably). The latter is the more usual economic notion of efficiency, but informationally efficient prices are neither necessary nor sufficient for efficient allocation.

The whole edifice, however, has turned out to be built, if not on sand, then at best on loose fill. More rigorous testing on larger data sets has shown that the capital asset pricing model does not fit the data; beta in particular does not predict returns at all. The response has been to identify variables that do predict returns and presume that they must be risk factors, although the extra risk has never been demonstrated. Prices are hard to predict, but not impossible, especially with high-frequency data (arriving minute-by-minute or faster). One reason markets are hard to predict is that they change much more than forecasts of future earnings should, and often they change on no detectable information at all. (Defenders claim that this just shows scholars aren't smart enough to grasp information known to everyone in the market.) Economists taking a behavioral approach have shown that actual investors don't act like the cool, farsighted calculators that efficient-market theory demands; worse, it turns out that having a handful of smart arbitrageurs around is actually not enough to swamp the "noise traders" — it really is the case that, as the saying goes, "markets can stay irrational longer than you can stay solvent."

This leaves us at an impasse. Efficient-market theory ought, with any methodological justice, to be relegated to the Museum of Nice Tries. But there is no unified replacement theory, and developing one will be arduous, involving empirical and theoretical work on all scales, from the experimental psychology of individual investors, through the institutional constraints under which money managers work, to solving for the aggregated effects of market participants' interactions. In the meantime, efficient-market theory provides a ready basis for precise calculations, and one that is moreover now built into the academic field of finance and into the practice and even infrastructure of the markets.

Fox does a superb job of recounting this history. Quite properly, he begins with the early 20th-century roots of the theory (in the writings of Irving Fisher and Louis Bachelier), tracing it from there through the Cowles Commission's incubation of modern econometrics in the 1930s and 1940s; the flowering of decision theory, operations research and mathematical economics in the 1940s and 1950s; and the rampant growth and metamorphosis of American academia after World War II. Fox's explanations of technical material are superbly accurate and readable, and he's equally good at describing the social and intellectual milieus in which these ideas propagated. He has also resisted the temptation to make some scholars into Heroes with Correct Ideas and others into Villains Clinging to Error (which is more than some of the economists he writes about have done). Clearly the result of many years of research and reading, the book is — its epilogue on the ongoing financial crisis notwithstanding — in no sense a rush job. Rather, it is a model of what the popularization of social science can be, but too rarely is, and it will continue to be read when the current crisis is many years behind us.

xviii + 382 pp., end-notes, bibliography, index

Economics; History of Science; Popular Science

Currently in print as a hardback, ISBN 978-0-06-059899-0, US$27.99

9 October 2009