Attention conservation notice: A 2000-word reaction to conferences in 2011 where too many people wanted impossible things of social science and data mining, and too many people seemed eager to offer those impossible things. Too long by more than half, too pleased with itself by much more than half, it lacks constructive suggestions and even a proper ending. To the extent there's any value to these ideas, you'd be better off getting them from the source I am merely parroting. Left to gather dust in my drafts folder for years, posted now for lack of new content.
Q: You have an old house with a slate roof, right?
A: You know that perfectly well.
Q: Does the roof ever need work?
A: You just said it's old and slate, and I live in Appalachia. (In the Paris of Appalachia; but still, in Appalachia.) Of course it does.
Q: Do you do the work yourself?
A: I have no idea how; I hire a roofer.
Q: How do you know the roofer knows what he's doing?
A: I am not sure what you mean. He fixes my roof.
Q: Well, does he accurately estimate where leaks will occur over the next year?
A: No.
Q: Does he accurately estimate how much the roof is going to leak each year?
A: No.
Q: Does he accurately estimate how many slate tiles will crack and "need replaced" each winter?
A: No.
Q: OK, so he's not much into point forecasts, I can get behind that. Does he give you probability forecasts of any of these? If so, are they properly calibrated?
A: No. I don't see where this is going.
Q: Everyone agrees that the ability to predict is a fundamental sign of scientific and technological knowledge. It sounds like your roofer can't predict much of anything, so what can they know? You should really hire someone else, preferably someone well-calibrated. Does Angie's list provide roofers' Brier scores? If not, why not?
A: I don't believe they do, I can't see why they should, and I really can't see how knowing that help me pick a better roofer.
Q: It's your business if you want to be profligate, but wouldn't it help people who do not enjoy wasting money to know whether supposed experts actually deserve to be taken seriously?
A: Well, yes, but if you will only listen to roofers who are also soothsayers, I foresee an endless succession of buckets under leaking ceilings.
Q: So you maintain a competent (never mind expert) roofer needn't be able to predict what will happen to your roof, not even probabilistically?
A: I do so maintain it.
Q: And how do you defend such an obscurantist opinion? Do you suppose that a good roofer is one who enters into a sympathetic human understanding with the top of the house, and can convey the meaning of a slate or a gutter?
A: Humanist-baiting is cheap even for you. No, when it comes to roofs, I am all about explanation, and to hell with understanding. (People are different.) But an expert roofer no more needs to predict what happens to the roof than an expert engineer needs to predict how a machine they have designed and built will behave. Indeed, it would be a bizarre miracle if they could make such predictions.
Q: And why would a plain, straightforward prediction be such a wonder?
A: However expert the roofer is about the roof, or the engineer about the machine, what happens to them depends not just on the object itself, but the big, uncontrolled environment in which it's embedded. You will allow, I hope, that what happens to my roof depends on how much rain we get, how much snow, etc.?
Q: Not being a roofer, I don't really know, but that sounds reasonable.
A: Might it not also matter how many sunny days with freezing nights we have, turning snow on the roof to ice?
Q: Sure.
A: And so on, through contingencies I'm too impatient, and ignorant, to run through. But then, to predict damage to the roof, doesn't the roofer need to not only know what condition it started in, but also all the insults it will be subjected to?
Q: That does seem reasonable. (But aren't you asking a lot of questions for "A"?)
A: (Shut up, I explain.) So your soothsaying roofer must be a weather-prophet, as well as knowing about roofs. And the same with the engineer and their machine: they would need to foresee not just the environment in which it will be put, but also the demands which its users will place upon it. It sounds very strange to say that such prophetic capacities are a necessary part of expertise in roofing, or even engineering.
Q: It might sound strange; many true things sound strange; indeed, doesn't every science become more and more un-intuitive and strange-sounding as it progresses? It sounds strange to lay-people to say that the economy consists of one immortal, lazy, greedy, infinitely calculating person, who does all the work, owns all the assets, and consumes all that the goods, but just think of the triumphs of successful prediction which the macroeconomists have achieved on this basis! If we abandon the criterion of prediction just because it sounds strange, how shall we ever distinguish an expert roofer from a mere pundit of the slates?
A: Perhaps by seeing if they can do the things roofers are supposed to do.
Q: Such as?
A: Well, when something goes wrong with the roof, they should be able to diagnose what caused the problem, and in favorable cases prescribe a course of action which will fix it, and even carry out the operation.
Q: Do you only call in the roofer when something has gone wrong? Your house must be in a sad state if so.
A: No, a good roofer should also be able to diagnose situations which, while they cause no immediate problem, are apt to lead to problems later.
Q: There can be few of those if you should have a winter without rain, which, you must admit, is possible. Are you not just sneaking back in my probabilistic forecasts, which you poured such scorn on before, with your "sooth-sayers" and your "weather prophets"?
A: Not at all; it's enough to recognize conditions which would cause problems under a broad range of circumstances which there is some reason to fear. Apprehension about the effects of a meter of snow sitting atop the roofer for three months on end is not reasonable in Pittsburgh; expecting even ten days in the winter without some precipitation is folly. At the very most, I am calling for some ability to say, conditional on typical weather, what consequences should be expected; the problem of giving a distribution for the weather is no part of the roofer's expertise.
Q: And next I suppose that you will pretend that prescription doesn't rest on prediction?
A: It certainly requires knowledge of the form If we do X, then condition Y will result; but if we do X', then Y'. Such conditional knowledge, about roofs, is actually immensely easier for a roofer to acquire, than for them to learn the whole huge multidimensional distribution of all the environmental factors which could influence the state of a roof, and their dependencies over time. But it is the latter which you are presuming, with your Brier scores and your notion of what sort of prediction is an adequate sign of knowledge. And what the roofer cannot foretell about the roof, neither can the engineer about the machine, nor the doctor about the patient, nor the natural scientist about their object of study.
Q: You insist that scientists do not make predictions?
A: In your unconditional, categorical, absolute sense, certainly not. Scientists certainly possess all sorts of conditional knowledge, about what would happen, if certain conditions were to be imposed, certain manipulations were to be made. Unconditional predictions, even unconditional probabilistic predictions, are for the most part beyond them.
Q: So a chemist cannot predict the course of a chemical reaction? Don't bother dodging with contaminants, or mis-labeled reagents.
A: Me, quibble? No. But even with pure, known reagents in a sealed reaction vessel at STP --- well, you were a student at Berkeley, where the chemistry labs are a block or two downhill of the Hayward Fault. I don't think you could have said what would have happened if there'd been a tremor in the middle of one of your experiments.
Q: You can't think it's fair to ask a chemist to predict earthquakes, can you?
A: My point exactly. To put it formally, in terms of Pearl, you could know \( \Pr\left(Y\middle |\mathrm{do}(X=x)\right) \) exactly, for all \( x \), but not know \( \Pr\left(Y\middle|X=x\right) \), still less \( \Pr(Y) \). (The Old Masters of Pittsburgh would say: you can know the distribution of \( Y \) after "surgery" to remove incoming edges to \( X \), without knowing the un-manipulated distribution of \( Y \).) But it is the last which you are insisting on, with your calibrated forecasts and prediction as the sign of expertise.
Q: Well, astronomers make such predictions, don't they?
A: I defy you to find a single other science which can also do so. Even then, our astronomy's successes merely testify to our engineering's weaknesses. When our descendants (or the cockroaches'; no matter) become able to move around comets, or move planets, or build Dyson spheres and Shkadov thrusters, even the predictions of celestial mechanics will become contingent on the interventions of sentient (I will not say "human") beings.
Q: Even in those science-fictional scenarios, the choices of human (or post-human or non-human) beings would be functions of their microscopic molecular state, and so physically predictable, so shouldn't —
A: I indulged in science fiction as a rhetorical flourish, but now you are seriously arguing on the basis of technical impossibilities, sheer metaphysical conjecture and even mythology.
Q: Never mind then. Suppose, for the sake of argument, that I accept that someone could be a knowledgeable expert without making predictions. Surely you would agree, though, that someone who makes a lot of predictions which turn out to be wrong definitely doesn't know what they're doing?
A: I can think of at least one way in which that fails, which is that when they also have effective control.
Q: Better control means un-predictability? This I have to hear.
A: Suppose I want to maintain a constant temperature in my house. I look at the sky, guess the day's weather, and turn on the air conditioner or the furnace as needed. If I express myself by saying "It's miserably humid and the sun isn't so much shining as pounding, the house will be intolerably hot", you will point out, at the end of a day during which the air conditioner labored heroically and the electric bill mounted shockingly, that the house was in fact entirely comfortable, and say not only that I can't predict anything worthwhile, but that, empirically, you see no particular relationship between the weather outside the house and the temperature inside it.
Q: Aren't you creating an air of paradox, simply by being sloppy in expressing the prediction? It's really "The house will be intolerably hot today, unless I run the AC".
A: Granted, but we can only ever check one branch of the condition. And I can be completely accurate in predictions about what would happen, absent imposed controls, even if none of those things actually happen, because of my forecast-based control.
--- I never did figure out how to end this.
Enigmas of Chance;
Commit a Social Science;
Constant Conjunction Necessary Connexion
Posted at March 30, 2020 09:50 | permanent link