Via Mason Porter, Danny Yee and others, I see a news story which my kith are glossing along the lines of a judge has ruled that Bayes's Theorem does not apply in Britain. Leave to one side my "tolerate/hate" relationship with Bayesianism; there are certainly cases, and ones of legal application at that, where Bayes's rule amounts to a simple arithmetic statement about population counts, so it would be very remarkable indeed if these were inadmissible in court. While I enjoy disparaging the innumeracy of the legal profession as much as the next mathematically-trained person, this seems like a distortion.
Let me quote from the Guardian story Mason linked to. (I can't find the actual opinion, at least not without more work than it's worth before lecture.) The story
begins with a convicted killer, "T", who took his case to the court of appeal in 2010. Among the evidence against him was a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. While appeals often unmask shaky evidence, this was different. This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed.But more importantly, as far as mathematicians are concerned, the judge also ruled against using similar statistical analysis in the courts in future. ...
In the shoeprint murder case, for example, [applying Bayes's rule] meant figuring out the chance that the print at the crime scene came from the same pair of Nike trainers as those found at the suspect's house, given how common those kinds of shoes are, the size of the shoe, how the sole had been worn down and any damage to it. Between 1996 and 2006, for example, Nike distributed 786,000 pairs of trainers. This might suggest a match doesn't mean very much. But if you take into account that there are 1,200 different sole patterns of Nike trainers and around 42 million pairs of sports shoes sold every year, a matching pair becomes more significant.
The data needed to run these kinds of calculations, though, isn't always available. And this is where the expert in this case came under fire. The judge complained that he couldn't say exactly how many of one particular type of Nike trainer there are in the country. National sales figures for sports shoes are just rough estimates.
And so he decided that Bayes' theorem shouldn't again be used unless the underlying statistics are "firm". The decision could affect drug traces and fibre-matching from clothes, as well as footwear evidence, although not DNA.
What I take from this is that the judge was asking for reasons to believe the numbers going in to Bayes's rule be accurate. This is, of course, altogether the right reaction. Unless the component numbers in the calculation --- the base rates and the likelihoods --- are right, the posterior probability has no value as evidence, because it has no connection whatsoever to the truth. Unless those components are validated, the differences between a witness who says "My posterior probability is 0.99" and one who says "I'm, like, really sure" are:
To reinforce just how wrong a simple-minded application of Bayes's rule can go, I invite you to consider the saga of the Phantom of Heilbronn. The combined police forces of Europe spent years searching for a criminal known from high-quality forensic evidence (DNA) left at more 40 crime scenes across a wide swathe of Europe. In the end, it turned out that the reason all these different crime scenes turned up the same DNA, is that the swabs used to collect the DNA from the scenes all came from the same factory, and had been contaminated by DNA from a worker there. (Presumably the contamination was accidental.) The case unraveled because while the common DNA was female, it was recovered from a male corpse. If it had been recovered from some unfortunate woman, it's very likely that this would now be regarded as a closed case. No doubt we would then be hearing Bayesian calculations about the odds against the suspect being anyone other than the Heilbronn serial killer --- who, recall, did not exist. (In fact, it's instructive to do a back-of-the-envelope version of the calculation, ignoring the contamination of the swabs.) If you say "Well, of course those calculations are off, the likelihood of the suspect matching a crime-scene in the test when the suspect wasn't really there is all wrong", I can only reply, "Exactly", and add that sensitivity analysis is no substitute for actually understanding where and how the data arise. This is related, of course, to the certainty of the Bayesian fortune-teller.
It is never pleasant to have claims to professional authority checked, so I certainly feel where my learned British colleagues are coming from*. But I have to conclude that, in so far as the judge said that Bayes's rule "shouldn't ... be used unless the underlying statistics are 'firm'," he was being entirely reasonable. He may, of course, have gone on to establish unreasonable standards for what counts as "firm" statistics; the news stories don't say. Unless that can be shown, however, the most damning verdict we statisticians can return is (what else?) "not proven".
Update, later that day: A reader has kindly supplied me with a copy of the ruling. On a first scan, phrases like "Maths! Nasty, wicked, tricksy maths! We hates them, Precious, hates them forever!" are absent, but I will try to read it and report back.
Update, 27 October: More than you would ever want to know.
*: Let me remind them that one trick which is proven to help people use Bayes's rule rightly is to eschew talk of probabilities, and employ frequency formats. Since Gigerenzer and Hoffrage were able to get doctors — a tribe notorious for their mis-understanding and mis-use of inverse probability — to use Bayes's rule correctly this way, it would be rather surprising if lawyers weren't helped too.
Posted at October 03, 2011 08:50 | permanent link