Three-Toed Sloth

Books to Read While the Algae Grow in Your Fur, September 2025

Attention conservation notice: I have no taste; and while I do have some qualifications to opine on statistical mechanics and on machine learning, these books are ones I began in 1999 (Dotsenko) and 2000 (Husmeier), so the works themselves are unlikely to be of interest to you. Also, most of my reading this month was done at odd hours and/or while chasing after a pre-schooler, so I am probably less reliable and more grumpy than usual.

I will presume the reader understands the terms "spin glass" and "neural network".

(Left almost-finished in 2025, because I got interrupted, and posted in 2026, because I wanted to procrastinate about half-a-dozen research projects.)

Viktor Dotsenko, An Introduction to the Theory of Spin Glasses and Neural Networks (1994; doi:10.1142/2460)

This is a short (~150 pp), brisk first book on the topic for physicists. Most of the book --- the first ten chapters --- are good introduction to common models of spin glasses. On the mechanistic side, the emphasis is on frustration, and how it leads to slow relaxation through multiple local minima. On the theoretical side, the emphasis is overwhelmingly on the replica trick and replica symmetry breaking *. I appreciated the detailed discussion of experimental results, but I would have appreciated still more clarity about distinctions between quantities which are functionals of the ensemble / probability distribution, and those which are functions of the realized microscopic state **. Something which would have whizzed right past me in graduate school is the role of distributions over matrices which are symmetric under permutation of rows and columns, i.e., of exchangeable arrays; now I want to read about connections between spin glasses and graphons.

The final chapters, about neural networks, especially the Hopfield model (which is perhaps back in fashion). These are OK, but if you want to see the statistical mechanics of neural networks (of that vintage), you're really better off with either "The Statistical Mechanics of Learning a Rule" (Watkin, Rau and Biehl, 1993), or Engel and Van de Broeck (2001).

The first ten chapters are still a perfectly good introduction to spin glasses and replica symmetry breaking for those with a command of statistical mechanics at the level of a first graduate course (e.g., Landau and Lifshitz). §

*: Dotsenko has a whole later book about the replica method, but I haven't read it. ^

**: In a lot of statistical mechanics, we can be sloppy about this distinction, because considerations of ergodicity and concentration of measure imply there's not much difference (cf.). But spin glasses are different! ^

---Presumably (?) a different Viktor Dotsenko.

Dirk Husmeier, Neural Networks for Conditional Probability Estimation: Forecasting Beyond Point Predictions (1999, based on a 1998 Ph.D. thesis; doi:10.1007/978-1-4471-0847-4

The problem here is to make distributional forecasts of a (scalar) random variable \( Y \) given a (vector) random variable \( Z \), in the form of training a neural network to calculate the conditional cumulative distribution function \( \mathbb{P}\left( Y \leq y|Z = z \right) \), with \( y \) and \( z \) as inputs. The particular version of this of most concern here is when we have a dynamical system or stochastic process \( X_t \), and \( Z = (X_t, X_{t-1}, \ldots X_{t-k} \) and \( Y = X_{t+1} \), i.e., predictive distributions for time series, especially in a state-space reconstruction setting.

Back in the 1990s, when we were all very impressed by the universal approximation results for neural networks, lots of people did stuff with three-layer neural networks (i.e., one hidden layer), and one actually had to make a bit of argument if one wanted to do more than three layers. Husmeier uses four layers, because he's worried about the following problem: Suppose \( Y = f(Z) + \epsilon \), with independent noise \( \epsilon \), and we let the variance of \( \epsilon \) shrink towards zero. The nonlinearities in the neural network will let us represent a CDF, sure enough. but the nonlinearity gets applied to a linear transformation of the input variables. In that situation, it'd seem like the only way to approach a deterministic limit is if the function \( f \) is linear in the input \( Z \). I am not entirely sure this is right, but it's certainly true that the extra layer Husmeier allowed himself made it a lot easier to get good results.

Much of the book is about ringing changes on this basic arrangement, emphasizing, as I said, the prediction of dynamical systems plus noise, with good results. Several chapters (13--16) concern ensemble methods ("network committees"), with various quasi-Bayesian or cross-validated schemes for weighting the members of the ensemble.

Something which whizzed by me back in the day (if I read those parts at all) is that, starting in chapter 7, Husmeier assigns random parameters to the lowest layer(s) of the network and then leaves them fixed, only optimizing the upper layers. This "random vector functional link net" approach did not originate with him, but it does vastly speed up fitting, without any real degradation in predictive performance compared to full back-propagation (or other optimization). This is a precursor to, or even a form of, what we now call random feature methods, and I am going to mine his references (starting here) to figure out where it came from and what happened to it.

I should have paid a hell of a lot more attention to this book back when it came out. §

Books to Read While the Algae Grow in Your Fur; Enigmas of Chance; Physics