March 30, 2020

Of the Evaluation of Expertise ("I am not so good for that as an old roofer")

Attention conservation notice: A 2000-word reaction to conferences in 2011 where too many people wanted impossible things of social science and data mining, and too many people seemed eager to offer those impossible things. To long by more than half, too pleased with itself by much more than half, it lacks constructive suggestions and even a proper ending. To the extent there's any value to these ideas, you'd be better off getting them from the source I am merely parroting. Left to gather dust in my drafts folder for years, posted now for lack of new content.

Q: You have an old house with a slate roof, right?

A: You know that perfectly well.

Q: Does the roof ever need work?

A: You just said it's old and slate, and I live in Appalachia. (In the Paris of Appalachia; but still, in Appalachia.) Of course it does.

Q: Do you do the work yourself?

A: I have no idea how; I hire a roofer.

Q: How do you know the roofer knows what he's doing?

A: I am not sure what you mean. He fixes my roof.

Q: Well, does he accurately estimate where leaks will occur over the next year?

A: No.

Q: Does he accurately estimate how much the roof is going to leak each year?

A: No.

Q: Does he accurately estimate how many slate tiles will crack and "need replaced" each winter?

A: No.

Q: OK, so he's not much into point forecasts, I can get behind that. Does he give you probability forecasts of any of these? If so, are they properly calibrated?

A: No. I don't see where this is going.

Q: Everyone agrees that the ability to predict is a fundamental sign of scientific and technological knowledge. It sounds like your roofer can't predict much of anything, so what can they know? You should really hire someone else, preferably someone well-calibrated. Does Angie's list provide roofers' Brier scores? If not, why not?

A: I don't believe they do, I can't see why they should, and I really can't see how knowing that help me pick a better roofer.

Q: It's your business if you want to be profligate, but wouldn't it help people who do not enjoy wasting money to know whether supposed experts actually deserve to be taken seriously?

A: Well, yes, but if you will only listen to roofers who are also soothsayers, I foresee an endless succession of buckets under leaking ceilings.

Q: So you maintain a competent (never mind expert) roofer needn't be able to predict what will happen to your roof, not even probabilistically?

A: I do so maintain it.

Q: And how do you defend such an obscurantist opinion? Do you suppose that a good roofer is one who enters into a sympathetic human understanding with the top of the house, and can convey the meaning of a slate or a gutter?

A: Humanist-baiting is cheap even for you. No, when it comes to roofs, I am all about explanation, and to hell with understanding. (People are different.) But an expert roofer no more needs to predict what happens to the roof than an expert engineer needs to predict how a machine they have designed and built will behave. Indeed, it would be a bizarre miracle if they could make such predictions.

Q: And why would a plain, straightforward prediction be such a wonder?

A: However expert the roofer is about the roof, or the engineer about the machine, what happens to them depends not just on the object itself, but the big, uncontrolled environment in which it's embedded. You will allow, I hope, that what happens to my roof depends on how much rain we get, how much snow, etc.?

Q: Not being a roofer, I don't really know, but that sounds reasonable.

A: Might it not also matter how many sunny days with freezing nights we have, turning snow on the roof to ice?

Q: Sure.

A: And so on, through contingencies I'm too impatient, and ignorant, to run through. But then, to predict damage to the roof, doesn't the roofer need to not only know what condition it started in, but also all the insults it will be subjected to?

Q: That does seem reasonable. (But aren't you asking a lot of questions for "A"?)

A: (Shut up, I explain.) So your soothsaying roofer must be a weather-prophet, as well as knowing about roofs. And the same with the engineer and their machine: they would need to foresee not just the environment in which it will be put, but also the demands which its users will place upon it. It sounds very strange to say that such prophetic capacities are a necessary part of expertise in roofing, or even engineering.

Q: It might sound strange; many true things sound strange; indeed, doesn't every science become more and more un-intuitive and strange-sounding as it progresses? It sounds strange to lay-people to say that the economy consists of one immortal, lazy, greedy, infinitely calculating person, who does all the work, owns all the assets, and consumes all that the goods, but just think of the triumphs of successful prediction which the macroeconomists have achieved on this basis! If we abandon the criterion of prediction just because it sounds strange, how shall we ever distinguish an expert roofer from a mere pundit of the slates?

A: Perhaps by seeing if they can do the things roofers are supposed to do.

Q: Such as?

A: Well, when something goes wrong with the roof, they should be able to diagnose what caused the problem, and in favorable cases prescribe a course of action which will fix it, and even carry out the operation.

Q: Do you only call in the roofer when something has gone wrong? Your house must be in a sad state if so.

A: No, a good roofer should also be able to diagnose situations which, while they cause no immediate problem, are apt to lead to problems later.

Q: There can be few of those if you should have a winter without rain, which, you must admit, is possible. Are you not just sneaking back in my probabilistic forecasts, which you poured such scorn on before, with your "sooth-sayers" and your "weather prophets"?

A: Not at all; it's enough to recognize conditions which would cause problems under a broad range of circumstances which there is some reason to fear. Apprehension about the effects of a meter of snow sitting atop the roofer for three months on end is not reasonable in Pittsburgh; expecting even ten days in the winter without some precipitation is folly. At the very most, I am calling for some ability to say, conditional on typical weather, what consequences should be expected; the problem of giving a distribution for the weather is no part of the roofer's expertise.

Q: And next I suppose that you will pretend that prescription doesn't rest on prediction?

A: It certainly requires knowledge of the form If we do X, then condition Y will result; but if we do X', then Y'. Such conditional knowledge, about roofs, is actually immensely easier for a roofer to acquire, than for them to learn the whole huge multidimensional distribution of all the environmental factors which could influence the state of a roof, and their dependencies over time. But it is the latter which you are presuming, with your Brier scores and your notion of what sort of prediction is an adequate sign of knowledge. And what the roofer cannot foretell about the roof, neither can the engineer about the machine, nor the doctor about the patient, nor the natural scientist about their object of study.

Q: You insist that scientists do not make predictions?

A: In your unconditional, categorical, absolute sense, certainly not. Scientists certainly possess all sorts of conditional knowledge, about what would happen, if certain conditions were to be imposed, certain manipulations were to be made. Unconditional predictions, even unconditional probabilistic predictions, are for the most part beyond them.

Q: So a chemist cannot predict the course of a chemical reaction? Don't bother dodging with contaminants, or mis-labeled reagents.

A: Me, quibble? No. But even with pure, known reagents in a sealed reaction vessel at STP --- well, you were a student at Berkeley, where the chemistry labs are a block or two downhill of the Hayward Fault. I don't think you could have said what would have happened if there'd been a tremor in the middle of one of your experiments.

Q: You can't think it's fair to ask a chemist to predict earthquakes, can you?

A: My point exactly. To put it formally, in terms of Pearl, you could know \( \Pr\left(Y\middle |\mathrm{do}(X=x)\right) \) exactly, for all \( x \), but not know \( \Pr\left(Y\middle|X=x\right) \), still less \( \Pr(Y) \). (The Old Masters of Pittsburgh would say: you can know the distribution of \( Y \) after "surgery" to remove incoming edges to \( X \), without knowing the un-manipulated distribution of \( Y \).) But it is the last which you are insisting on, with your calibrated forecasts and prediction as the sign of expertise.

Q: Well, astronomers make such predictions, don't they?

A: I defy you to find a single other science which can also do so. Even then, our astronomy's successes merely testify to our engineering's weaknesses. When our descendants (or the cockroaches'; no matter) become able to move around comets, or move planets, or build Dyson spheres and Shkadov thrusters, even the predictions of celestial mechanics will become contingent on the interventions of sentient (I will not say "human") beings.

Q: Even in those science-fictional scenarios, the choices of human (or post-human or non-human) beings would be functions of their microscopic molecular state, and so physically predictable, so shouldn't —

A: I indulged in science fiction as a rhetorical flourish, but now you are seriously arguing on the basis of technical impossibilities, sheer metaphysical conjecture and even mythology.

Q: Never mind then. Suppose, for the sake of argument, that I accept that someone could be a knowledgeable expert without making predictions. Surely you would agree, though, that someone who makes a lot of predictions which turn out to be wrong definitely doesn't know what they're doing?

A: I can think of at least one way in which that fails, which is that when they also have effective control.

Q: Better control means un-predictability? This I have to hear.

A: Suppose I want to maintain a constant temperature in my house. I look at the sky, guess the day's weather, and turn on the air conditioner or the furnace as needed. If I express myself by saying "It's miserably humid and the sun isn't so much shining as pounding, the house will be intolerably hot", you will point out, at the end of a day during which the air conditioner labored heroically and the electric bill mounted shockingly, that the house was in fact entirely comfortable, and say not only that I can't predict anything worthwhile, but that, empirically, you see no particular relationship between the weather outside the house and the temperature inside it.

Q: Aren't you creating an air of paradox, simply by being sloppy in expressing the prediction? It's really "The house will be intolerably hot today, unless I run the AC".

A: Granted, but we can only ever check one branch of the condition. And I can be completely accurate in predictions about what would happen, absent imposed controls, even if none of those things actually happen, because of my forecast-based control.

--- I never did figure out how to end this.

Of the Evaluation of Expertise ("I am not so good for that as an old roofer"); Commit a Social Science; Constant Conjunction Necessary Connexion

Posted at March 30, 2020 09:50 | permanent link

March 10, 2020

Ebola, and Mongol Modernity

Attention conservation notice: An old course slide deck, turned in to prose on the occasion of a vaguely-related news story from 2014. Not posted at the time because it felt over-dramatic. I have, of course, no authority to opine about either world history or epidemiology, and for that matter no formal training in networks.

Exhibit A

One of the books which most re-arranged my vision of the past was Janet Abu-Lughod's Before European Hegemony: The World System A.D. 1250--1350. It gave me the sense, as few other things have, of historical contingency, or more exactly of modernity as a belated phenomenon, and changed my teaching. She depicts an integrated (part-of-the-) world economy, an "archipelago of towns" linked by trade-routes stretching from Flanders to Hangzhou and centered in the Indian Ocean. This archipelago is where modernity should have begun. Beyond the market-oriented, urban-centered economy, China has the beginnings of an industrial revolution (a point explored by Mark Elvin in his Pattern of the Chinese Past, and his sources in Japanese scholars of Chinese economic history, and emphasized by William McNeill in his Pursuit of Power); the beginnings of a truly global perspective. All of this was politically supported by the unification of the most economically and technologically advanced regions (namely China and the Islamic world) under the Mongol Empire, admittedly at the cost of the occasional "shock and awe" campaign, destruction of Baghdad, etc.

Exhibit B

So what, according to Abu-Lughod, happened? What happened was Yersinia pestis, the bubonic plague, a bacterium transmitted by fleas that live on rodents. It long has been, and is, endemic to the rodents of Central Asia, such as the giant gerbil {\em Rhombomys opimus}, which seems to be perpetually perched at the edge of the epidemic threshold. The Mongol Empire didn't just unify the most advanced parts of Eurasiafrica; it brought them into intimate contact with Central Asia. And then, as usual, the plague followed the routes of trade and imperial travel:

It's impossible to know just how deadly it was, but estimates put it at around 25% of global population; up to 90% in some regions. It destroyed (according to Abu-Lughod) the world economy, that "archipelago of towns", leaving isolated and barely-functional fragments which could be dominated and re-purposed by western European pirates/traders poking into the post-pandemic landscape.

One aspect to this is how slowly, and how progressively, the plague spread. It took decades to spread from Central Asia to the peripheral region of western Europe, where it chewed steadily across the landscape:

As my old friend and sometime co-author Mark Newman and collaborators puts it, this is strong evidence that "the small-world effect is a modern phenomenon". The small-world effect, after all, is that the maximum distance between any two people in the social network of size $n$ grows like $O(\log{n})$. This implies that the number of people reachable in $d$ steps grows exponentially with $d$, which is hardly compatible with the steady geographic progress of the disease.

The argument here is so pretty that I can't resist sketching it. Suppose that every infected person passes it on to any one of their contacts with probability $t$, at least on average. We start the infection at a random person, say Irene, who selects a random one of their acquaintances, say Joey, for passing it on. The probability that Irene, or any other random person, has $k$ contacts is, by convention, $p(k)$. But Joey isn't a random person; Joey is someone reachable by following an edge in the social network. Joey's degree distribution is $\propto k p(k)$, since people with more contacts are more reachable. Specifically, Joey's degree distribution is $k p(k) / \langle K \rangle$, where $\langle K \rangle \equiv \sum_{k=0}^{\infty}{k p(k)}$, the average degree. If Joey gets infected, the number of additional infections he could create is up to $k-1$. So our initial random infection of Irene creates, on average, $t \sum_{k=1}^{\infty}{k(k-1)p(k)}/\langle K \rangle = t \langle K^2 - K\rangle / \langle K \rangle$ at one step away. At $d$ steps, we've reached $ \left(t \langle K^2 - K\rangle / \langle K \rangle \right)^d$ nodes. So long as $t > \langle K \rangle / \langle K^2 - K\rangle$, this is exponential growth:

If $t$ is above this critical level, the only way to avoid exponential growth is to have lots and lots of overlapping routes to the same nodes. [Some people might've had 1024 ancestors ten generations back, but most of us didn't.] Geographic clustering will do this, but the small-world effect makes it very hard to avoid. And the small-world effect is very much a part of modernity; whether the diameter of the social network is six or twelve or twenty is secondary to the fact that it's not a thousand.

Exhibit C

Zeynep Tufekci, "The Real Reason Everyone Should Panic: Our Global Institutions are Broken" (23 October 2014):
The conventional (smart) wisdom is that we should not panic about Ebola in the United States (or Europe). That is certainly true because, even with its huge warts, US and European health-care systems are well-equipped to handle the few cases of Ebola that might pop up.
However, we should panic. We should panic at the lack of care and concern we are showing about the epidemic where it is truly ravaging; we should panic at the lack of global foresight in not containing this epidemic, now, the only time it can be fully contained; and we should panic about what this reveals about how ineffective our global decision-making infrastructure has become. Containing Ebola is a no-brainer, and not that expensive. If we fail at this, when we know exactly what to do, how are we going to tackle the really complex problems we face?
Climate Change? Resource depletion? Other pandemics?
So, I have been panicking. ...
Globalization, in essence, means we really are one big family, in sickness and in health.
The more connected we are, the easier it is for a virus to spread wide and deep, before we get a chance to contain it.
And that is partially why Ebola is now ravaging through three countries in West Africa: it broke through in cities and large-enough settlements, and due to an accumulation of reasons, including recent civil wars, at a time when they were least equipped to handle it.
Containing an outbreak requires circumscribing the outbreak (isolating and treating the ill, tracing their contacts, isolating and treating them as well) so that it can no longer find new hosts, and healing those who are ill, or mourning those who die. Circumscribing an outbreak is easier when the cases are a few, or a few dozen, or a few hundred.
In fact, we know from previous Ebola outbreaks which parameter brings down the dreaded transmission rate: “the rapid institution of control measures.” It’s that simple.
After thousands of cases, this gets harder and harder.
After millions, it is practically impossible.

Of course, Ebola got under control (in 2014). It took far too much misery and fear and time, but it happened. But it left no sign that the powers that be had learned any lessons, about (to quote Tufekci again) either "basic math [or] basic humanity".

The Mongols, at least, had the excuse that they had no idea what they were doing. (It's not as though Nasir al-din al-Tusi had, between doing theology and pioneering Fourier analysis, worked out how network connectivity related to the likelihood of epidemic outbreaks.) Compared to them, our predecessors in globalization, we are as gods; we're just not very good at it.

Networks; Writing for Antiquity; The Continuing Crises; The Great Transformation; The Natural Science of the Human Species

Posted at March 10, 2020 23:46 | permanent link

Three-Toed Sloth