March 20, 2008

Chaos, Complexity, and Inference (36-462): Lecture Notes

This page will be updated as the semester goes on, if you want to use this RSS feed to track them. Alternately, lecture notes will be linked to on the course syllabus, which includes the readings.

Lecture 18 (March 20): Special problem-solving session for homework 2
partial solutions; R
Lecture 17 (March 18): Error Statistics and Severe Testing
slides
Lecture 16 (March 6): Heavy tails 4, testing and evaluation
slides and R
Lecture 15 (March 4): Heavy tails 3, estimation
slides
Lecture 14 (February 28): Heavy tails 2, origins
slides
Lecture 13 (February 26): Heavy tails 1, basics
General R files for the next several lectures
slides; R
Lecture 12 (February 21): Self-organization 2
slides
Lecture 11 (February 19): Cellular automata 2/Excitable media
slides
Lecture 10 (February 14): Cellular automata 1
slides
Lecture 9 (February 12): Self-organization 1
Philip and Phylis Morrison and the Office of Charles and Ray Eames, Powers of Ten
slides
see also: Pattern Formation in Cocktails
Lecture 8 (February 7): Determinism and randomness
Henri Poincaré, "Chance" (from Science and Method, 1908)
slides
Lecture 7 (February 5): Information theory
slides
M.C. Hawking, "Entropy", from Fear of a Black Hole [lyrics; mp3 (radio-safe Brief History of Rhyme version)]
Ray and Charles Eames, A Communications Primer
Lecture 6 (January 31): Inference for Markov chains and related processes
slides
Note: Maximum Likelihood Estimation for Markov Chains
Lecture 5 (January 29): Symbolic dynamics; stochastics from dynamics
slides
Note: More on the Topological Entropy Rate
Lecture 4 (January 24): Attractor reconstruction and nonlinear prediction
slides (see slides for R examples).
Note: Nonlinear prediction, nearest-neighbors, kernel methods
Lorenz time-series generator, written in Perl.
the Lorenz time series used in the lecture
Lecture 3 (January 22): Attractors
slides; R
Lecture 2 (January 17): More chaos
slides; R
The Arnold Cat Map Movie (starring Marlowe the Cat, directed by Evelyn Sander)
Lecture 1 (January 15): What is a dynamical system? What is chaos? What is a simulation?
slides; R

Corrupting the Young; Complexity; Enigmas of Chance; Networks

Posted by crshalizi at March 20, 2008 23:49 | permanent link

March 17, 2008

Career Advising Day

Attention Conservation Notice: Another thousand-odd-word rant about reactionary idiots pretending to be scientists.

Contemplating the writings of the now-deservedly-forgotten Pierre Teilhard de Chardin, the late, great Peter Medawar was driven to observe that "Just as compulsory primary education created a market catered for by cheap dailies and weeklies, so the spread of secondary and latterly tertiary education has created a large population of people, often with well-developed literary and scholarly tastes, who have been educated far beyond their capacity to undertake analytical thought." (Whether that is a trained incapacity is itself a nice question.) Some of those people, owing to those tastes, pursue careers in academic research; the problem for them is that they are not actually very good at what they are supposed to do, which is come up with novel, insightful, important, precise, and accurate findings. Suppose that you are such a person, and that you do not want to switch to some other line of work to which you might be better suited. What to do?

Perhaps the best thing which could happen to you would be to run across a new and controversial theory which speaks to you at a deep level, both intellectually and temperamentally. If you are what William James called "tender-minded", like Teilhard de Chardin, then Medawar has already mapped out your trajectory, though nowadays the Templeton Foundation would likely be involved. If instead you are what James called "tough-minded" — "materialistic, pessimistic, irreligious, fatalistic, sceptical" — then edification-through-obfuscation is not an option, but it wouldn't even occur to you. Instead, you take your theory and you write papers about it, where you make claims about lots of hot-button topics, especially sex and current political controversies. The papers seem to carry the signs of rigor, but are actually deeply fallacious — maybe you see this, but are so convinced the conclusions are right you don't care, or maybe you're so convinced of the conclusions you can't see the errors. (There is some peer-reviewed venue where you can publish almost arbitrarily sloppy papers, so getting into print won't be a problem.) Then — and this is the key — you start promoting your papers, and find that more salacious and provocative your spin on them, the bigger the response. Your possibly-unconscious shamelessness about publishing rubbish will not only give you an advantage in sheer publications over other mediocre scholars who happen to have an intellectual conscience, but will also get you media attention. The reason it will get you media attention, and credibility with the media, is that they will see your institutional affiliation and your peer-reviewed papers, and so you become not just another crank but a Serious Scholar Contributing to the Debate. The whole package — carelessness, provocation and publicity — is wonderfully self-reinforcing, so you write even more careless papers, with yet more provocative conclusions, which you push even harder. (As a wise woman once said, "No one ever forgets how to do something that's worked for them in the past.") With a bit of luck, book contracts, magazine columns, etc., will follow in their train. Your career becomes like two drunks supporting each other as they stagger down the street: neither crappy academic research nor media presence could stand up on their own, but together they can lurch and shamble in glorious, glorious inebriated freedom, bellowing about the fierce joys to be found in facing what's revealed by the harsh light of your pseudo-scientific prejudices.

Ladies, gentlemen, and distinguished others, I give you Dr. Satoshi Kanazawa of the London School of Economics, the Fenimore Cooper of sociobiology, a man who has leveraged an inability to do data analysis or understand psychometrics into an official blog at Psychology Today, where he gets to advocate genocidal nuclear war as revenge for 9/11. He seems to mean it, rather than be fukayaming.

His argument — to the extent that it is an argument and not just a wish-fulfillment fantasy — has to do with his earlier attempt to explain "why most suicide bombers are Muslims". Leave to one side whether his attempted explanation is coherent, two things strike one on reading that. The first is his near-total disconnection from the literature on, precisely, the causes and motivations of suicide bombing — no Sageman, no Pape, the only mention of Scott Atran (an actual evolutionary psychologist, and very aware of the problems with the kind of crude ad hominid argument Kanazawa pushes) basically misses Atran's point, etc. The second is that the fact he is trying to explain something which isn't true: the tactic was pioneered by the decidedly non-Muslim Liberation Tigers of Tamil Eelam. This is well-known to anyone even slightly interested in understanding this horrible practice; Kanazawa doesn't even try to explain it away. But facts like these don't matter when the real goal is some combination of, on the one hand, constructing and projecting a fantasy ideology, and, on the other, sheer self-promotion.

I don't know of any systematic data on whether James's distinction between tender-minded and tough-minded thinkers really holds up, but at the level of casual empiricism it's pretty persuasive, and I fall very much on the tough-minded end of the spectrum. I find that sort of position persuasive, but at the same time it takes only a minimal amount of self-knowledge (certainly that's all I've got) to realize that it exposes one to certain characteristic errors or temptations. One of them is self-congratulation at being, precisely, so tough-minded. And one prominent expression of that is a delight in one's superior ability to perceive things as, supposedly, they really are, stripped of sentimental ornament; more than that, a delight in imagining how the tender-minded will be shocked by having to confront these realities. It is especially a delight in reductionism, not as a productive if not inevitable explanatory strategy, but as a series of "nothing-but" claims. This is one of our characteristic forms of wishful thinking, just as much devising imaginary consolations for real sufferings is a characteristic of the tender-minded. With these thoughts in mind, I invite you to read the conclusion of Kanazawa's article on suicide bombers:

Maybe the Muslim suicide bombings are not "terrorist" acts, as the term is usually used. Maybe it has nothing to do with Israel or the American and British troops. Maybe it's all about sex, as everything else in life is. Men do everything they do in order to get laid (Kanazawa, 2003). Maybe young Muslim men are no exceptions.
Satoshi, mon semblable, mon frère: whoever they are, I really hope the sex is worth it.

Obligatory disclaimers:

  1. I am agnostic, and uninterested, as to whether Kanazawa, like one of Thorndike's cats, developed these habits by trial, error, and environmental reinforcement, or whether, like one of Köhler's chimpanzees, it came to him in a flash of insight that he could reach the seemingly-unobtainable prize, if only he stacked publicity atop bullshit.
  2. Of course, Kanazawa has a perfect right to express whatever opinions and beliefs he might have (within the usual limits of the laws of libel, etc.). Moreover he ought to have the right to pursue whatever lines of inquiry seem to him most promising. Reciprocally, the rest of us have every right to criticize his research as idiotic, ideological rubbish.
  3. I actually think there is a lot to be said in favor of evolutionary psychology, but if anything that only makes it more important to stomp on things like this.
  4. I cheerfully admit that "I hope we catch the bastards who did this and nuke them till they glow" was one of my first reactions to seeing the pictures from 9/11. (The very first was that I was being shown some kind of disgustingly tasteless viral marketing for a disaster movie.) This sort of reaction is not a sound basis for making policy, and I leave it as an exercise to explain why on evolutionary grounds. I would still like to see us catch the bastards who did this, however.

Credits: Kanazawa's blog found via the appropriately dumbfounded reaction of Michael Meadon. I owe the insights, and much of the phrasing, of my second paragraph to a correspondent who prefers to keep their name out of this.

Learned Folly; The Continuing Crisis; The Natural Science of the Human Species; The Running-Dogs of Reaction

Posted by crshalizi at March 17, 2008 15:05 | permanent link

Follow the Oil Money!

Via Skye Bender-deMoll, an old acquaintance from Santa Fe days, a lovely little example of network mapping and the visual display of quantitative information in the service of the public good: Follow the Oil Money. This website lets you track the network of campaign donations from the oil industry, in its various tentacles, to U.S. politicians, with nifty pictures and charts.

The root of the shrub

As Skye explains, while all this information is a matter of public record, available from the FEC, working with that data is surprisingly hard. (I didn't appreciate just how hard when I wrote this.) Skye and his collaborator Greg Michalec have done a really impressive job of making this accessible. The result is good for hours and hours of entertainment and enlightenment, even if you think that it's all about constitutionally-protected and democratically legitimate freedom of speech, in the form of dollars, on the part of the companies. (Incidentally, if you do think that, please contact me about exciting business opportunities in Lagos.)

It would be fascinating, if perhaps scary, to see a parallel website for military contractors.

Networks; The Beloved Republic

Posted by crshalizi at March 17, 2008 14:04 | permanent link

March 14, 2008

Profiled

I am this week's profilee at Norman Geras's blog. This is very pleasing to me, as my chance discovery of Norm's Solidarity in the Conversation of Human Kind (followed by his Marx and Human Nature) was an important part of my intellectual development in graduate school. There are some very important areas of politics where I believe that he is very wrong, but he's always sincerely benevolent and worth reading; and sometimes, simply right.

Self-Centered; The Progressive Forces; Linkage

Posted by crshalizi at March 14, 2008 16:10 | permanent link

Three-Toed Sloth:   Hosted, but not endorsed, by the Center for the Study of Complex Systems