Attention conservation notice: An intemperate, unfair, 900-word rant about physicists' neglect of modern probability theory and statistics. Presents only anecdotal evidence, formulates no testable hypotheses, and does not decide on any course of action.
As part of a futile effort to beat my "Stuff to Blog" folder down to reasonable size, I came across Richard Gill's remarks on teaching physicists statistics. Since this is one of my hobby-horses, but everyone looks ridiculous riding a hobby-horse, I'll let Gill mount up instead.
I would like to make some comments on physics versus statistics (the latter includes for present purposes probability).As has been said by others we statisticians have a lot of difficulties trying to `sell our wares' to physicists. Physicists are renowned for their arrogance. They believe there is nothing much to know about statistics and that they can easily invent it for themselves if necessary. And one has to admit they are dammed clever and for instance do mathematical calculations with more ease and speed and originality than most mathematicians. This is certainly some cause for us to adopt some humility when dealing with them. The sheer amount of things they know is amazing and the finely tuned intuition about physical reality with which they home in on the right answer despite getting logico-mathematical arguments usually wrong is amazing too.
However I think they are wrong (about knowing already what they need to know about statistics) and that we do know something they do not know, and do need to know. And I believe that although we should adopt a certain amount of humbleness in our dealings with these uebermenschen, we should at the same time counter their arrogance with some of our own.
The trouble is that elementary probability is pretty boring and elementary statistics even more so. Clearly we should not be teaching these topics to such clever people by starting at the beginning, as we can do (?) with mathematicians. Rather we should assume a certain amount of sophistication and maturity and step in at the deep end with exciting topics which they will appreciate. For instance: Gibbs sampling, simulated annealing, Metropolis... : using ideas from physics to solve hard numerical and combinatorial optimisation problems certainly appeals to them. What about the beautiful results and challenging problems in random polymers, interacting particle systems, percolation,...?
Oh, heck, I can't resist taking a turn around the ring. Part of the reason theoretical physicists hallucinate power-law distributions and say bizarre things about Fisher information and flat-out wrong things about compression algorithms is that we learn nothing about statistical inference or information theory. Part of the reason for that is that we actually learn very little probability theory, even those of us in statistical mechanics. Gill again:
I attended a fascinating advanced physics course on thermal and statistical physics and was amazed how one time the lecturer wasted half an hour (of his precious 2 times 3/4) by not being able to use standard (and basic) ideas from measure theory or differential geometry in order to explain concisely what he means by a uniform distribution on a curved manifold...the students are bright and realise something is being hidden from them. The lecturer is certainly not daft either, so it is really strange...I've been one of those students. The problem isn't ignorance of differential geometry, which we do realize we need to know, but rather of measure theory, where that realization eludes us. I think our lack of interest in (exaggerating somewhat) everything that has been done in the theory of probability since 1905 is scandalous, just as if we ignored dynamics since Poincaré and Birkhoff, or geometry since Riemann and Poincaré. I am very happy to report that I share this opinion with R. F. Streater (whom my fellow physicists will remember for PCT, Spin and Statistics, and All That, among much else). Here is how he begins his paper "Classical and Quantum Probability" (math-ph/0002049):
There are few mathematical topics that are as badly taught to physicists as probability theory. Maxwell, Boltzmann and Gibbs were using probabilistic methods long before the subject was properly established as mathematics. Their language, of ensembles, complexions, fluctuations and most probable state, are still used. When quantum theory came along, the same notions were fitted into the new theory, sometimes leading to confusion.
As it happens, I had an excellent probability teacher, but I had to go to the math department to find him.
Of course there are theoretical physicists who do know and use probability theory, and (these sets intersect) others who are very good at statistical inference. (If you are a theoretical physicist and are actually reading this, I may well have you in mind.) I can even testify that there is at least one person who referees for the Journal of Statistical Physics who is very picky about the regularity of conditional probability functions. But by and large, this is an area where we're ignorant, which leads to bad work and/or wasted effort. (E.g., I myself have refereed multiple manuscripts for Physical Review which re-discovered special cases of the results Tom Kurtz published in 1970*.) Because we don't teach this stuff, coherent books on non-equilibrium proceses have to double as remedial math texts. To make things truly intolerable, this is math economists learn without trouble.
*: Thomas G. Kurtz, "Solutions of Ordinary Differential Equations as Limits of Pure Jump Markov Processes", Journal of Applied Probability 7 (1970): 49--58. In fairness, I did learn about that paper from this one.
Posted at November 22, 2004 11:45 | permanent link