Bayesian trickery

[Chris Lloyd]

It is often claimed that regular, normal people naturally think like Bayesians. Leaving aside whether we should leave the foundations of our subject to the average punter, I suspect that this might be true. But it really depends on how you frame the question. Below is a description of a class discussion exercise used by Bill Jefferys, who is a Professor of Astronomy but also an adjunct professor of statistics, teaching a course in Bayesian statistics at the University of Texas.

See if you can spot the flaws.



I really support the use of both actual and thought experiments in class. In my undergraduate courses, I got told a great deal about what to do and how to do it, but almost nothing about why. Most mathematically savvy students are able to quickly see why something makes sense – this is probably why they are good at maths. Non-mathematical types often just cannot see the point. But epistemology is a complex and subtle field and should be given much more weight in standard statistics course.

Anyway, here is Bill Jeffrys’*nice class exercise, which I found in a comment on Andrew Gelman’s blog and which I will now summarise.

Step1. Bring a dollar coin to class. Ask the class “If I flip it what is the probability that it will come up heads?” Everyone agrees it is 0.5 or very close to 0.5, thought there might be some comment about biased flipping and sleight-of-hand.

Step2. Flip the coin onto the floor and immediately step on it. I do not know if it is heads or tails. Neither do the students. Ask again: “What is the probability that it is heads?” Most people will say 0.5, but those who have thought carefully about the foundations of frequentist inference might say that it is either heads or tails, but this cannot be quantified as a probability. Those that say 0.5 are thinking as Bayesians; the others are thinking as frequentists.

Step 3. Look at the coin without letting anyone else see it. Then say “I now know whether it is heads or tails. What is the probability that it’s heads?” Most will still say that it’s 0.5. They are still thinking as Bayesians. Their background information is personal and is not the same as mine. It is rational for them to be uncertain while I, having difference information, am certain.

Step 4. Announce that I have seen a head and ask them “What’s the probability that it’s heads?” This poses something of a conundrum, since many of the students will tumble to the fact that I might not be telling the truth; so many of them will offer a higher number, 0.8 or 0.9, but not 1.0! It may be argued that this is a Bayesian approach again as they are conditioning on “professor says it is heads” which is not the same as “It is heads.”

Step 5. Invite a student to look at the coin and announce what she saw. Usually the student will report the same thing you do!

Step 6. Let everyone take a look for themselves.

This is a great exercise. But Bill claims that

“This is an exercise in Bayesian thinking - it is legitimate to quantify your uncertainty about a state of nature by putting a probability distribution on it and conditioning on data.”

Stuff and nonsense. This is not even close to a justification of Bayesian statistics or a refutation of the frequentist paradigm - even if I trusted the thought processes of under-graduates as the basis for statistical inference. Here are the flaws.

Flaw 1. Statisticians are, or should be, aware of how careful wording of a survey question can produce almost any desired result. Bill is not a statistician by trade so this may not be front of mind for him. But imagine that throughout this exercise we ask an alternative question:

“Is this particular outcome a head or a tail? Yes or no?

We would get many more people saying “I cannot possibly say.” Bill’s question actually requires the students to give a probability! No wonder that so many students give him one. By forcing the students into a Bayesian straight-jacket, he is assuming what he is trying to prove.

Flaw 2. How about when he tells them that he saw a head? Would a frequentist update the inference to “condition on this data?” Well, we would not condition on the data. We would posit a model for the data. I might say to myself.

“There is a 90% chance that the professor will tell the truth. So the data “professor says it is a head” has probability 0.9 if the truth is heads and 0.1 if the truth is tails. The likelihood ratio is 9 to 1 in favour of heads.”

This is a pretty strong inference and it is not necessary to go the extra step, put a prior on head and tails, and convert this into a posterior probability*for heads of 10/11.

Flaw 3. A more subtle bias in the exercise is that the students see the spinning coin. So on this occasion nature has performed a random experiment to determine the parameter. In this case, it is clearly reasonable - though not necessary - to take the parameter to be a random variable. Would the students think it is so obvious that*the gravitational constant is a random vairable? I think not.

As I have said in other posts I am not anti-Bayesian. However, I am anti-Bayesian trickery. I am sick to death of hearing it claimed that (1) people not brain washed by frequentist theory are automatically Bayesian, (2) frequentistis cannot make use of prior information. I will have a(nother) post on the second issue next week.



Get More from the original blog...