February 25, 2003

Dumb Survey Tricks, or, Time to Restock the Adaptive Toolbox

A few weeks back, parts of the blogosphere were much aghast at reports of a poll where a huge fraction of Americans surveyed said some or most of the 9/11 hijackers were Iraqis, when of course none of them were. (Steven Johnson: "People! Can we please just try to pay the slightest bit of attention?") Examining the news reports of the survey (like this one in Salon), things really don't seem that bad. The phrasing of the survey question seems to have been something like "How many of the 9/11 terrorists were from Iraq? (a) All; (b) More than half; (c) Less than half; (d) None; (e) Don't know". (None of the news reports actually bothers to quote the question, so I'm forced to reconstruct.) Now imagine you get called up out of the blue and asked to answer this. You don't remember hearing anything about Iraqi hijackers, but, well, it was more than a year ago, and you know you don't pay as much attention to the news as you should. Presumably the pollster isn't trying to ask you a trick question, and what would be the point of the question if the answer was (d) and everyone knew that? (There were no survey questions about how many hijackers were Croats or Nigerians.) So you may say you don't know, which is actually what a third of those polled did. Or you assume the question has some kind of relevance, and guess, but you're unlikely to guess "all" or "none", since that's the kind of answer you'd think everyone would remember, and so no one would bother to ask. (People don't like giving extreme answers, generally speaking.) Result: the people surveyed end up looking like idiots, though they're just following what's normally a very adaptive strategy, namely assuming that the person talking to you isn't trying to mislead.

Minds, BRains, and Neurons

Posted at February 25, 2003 17:12 | permanent link

Three-Toed Sloth