Mr. Bayes, Tear Down This Strawman!

A long time ago, in a city far, far away, I worked in a unit of four folks dedicated to quants. We were named The Office of Analytic Methods, a small outpost in a small Federal agency on E st NW DC. Among the four was a Ab.D. in stats who talked much of Bayes. Now, this was at a time when stat packs ran only on mainframes, and had names such as SPSS, SAS, BMDP, PSTAT. I'd wager that no one has heard of the last two. I was of the frequentist mind (as such are called today), and viewed Bayesian methods as shown, on occasion, in the preamble to this endeavor (cite long lost, and from memory):

If you're so sure of your prior, then just state your posterior and be done with it.

The Bayesian approach asserts that one should start with an assertion of some parameter's value, then collect some data, and finally see whether this data is reason enough to change one's initial assertion. That is the prior. The result when one is done is the final answer, and is called the posterior.

Being somewhat open minded, I decided to have a new look at Bayes, and therefore got "Doing Bayesian Data Analysis: A Tutorial with R and BUGS". It's been highly reviewed, and has all those cute doggies on the cover (not explained, either). The first half of the book is built on Bernoulli and binomial distribution; a lot of coin flipping. Chapter 11 gets to the heart of the matter, "Null Hypothesis Significance Testing" (NHST, for short). Those who embrace Bayes (nearly?) universally object to usual statistical testing and confidence interval estimation, because they're based on testing whether values equal, as assumed. This is the Null Hypothesis: that two means are equal, for example. We assume that two samples (or one sample compared to a known control) have the same value for the mean, and set about to test whether the data support that equality. Depending on what the data say, we either accept or reject the null hypothesis. We don't get to say that the true mean (in this example) is the sample mean if we reject the null hypothesis. This lack of specificity bothers some. Bayesians say we can do better.

The NHST amounts to proof by contradiction, an age old maths proof method, but strongly objected to by Bayesians.

This chapter explains some of the gory details of NHST, to bring mathematical rigor to the preceding comments and to bring rigor mortis to NHST.

The crux of his argument is that a sample will accept or reject the null hypothesis, depending on the sampling plan. He puts a sinister spin on this fact:

We have just seen that the NHST confidence interval depends on the covert intentions of the experimenter.

That word, "covert" and similar, appears with some frequency in the discussion. In one sense, the NHST does accept in one case and reject in the other, but there's nothing covert going on. What happens is that one framing of the question is binomial, while the other is negative binomial. Since the distributions differ, the test, using the same data, can give different results. In the strawman presented, it does. Of course.

Later, he constructs a real strawman. He tells a tale of flipping flat-head nails.

We flip the nail 26 times, and find it comes up heads 8 times. We conclude, therefore, that we cannot reject the null hypothesis the nail can come up heads or tails 50/50. Huh? This is a nail we're talking about. How can you not reject the null hypothesis?

Think about that for a second. He's morphing a null hypothesis regarding a coin, asserted to be symmetrical in three dimensions and of homogeneous density, with a nail which has neither characteristic. This is flippin' silly.

So, I remain a frequentist.