Results Quote Due To Chance Unquote

Here we list differences between the claim “a p-value is the probability that the results are due to chance” and the actual definition of a p-value. Many of these differences apply only in some situations. But some people claim that a p-value is ALWAYS the probability that the results are due to chance, and finding differences which SOMETIMES occur is enough to defeat that claim.

These are in no particular order.

Bayes’ theorem is true, regardless of whether it’s always useful. (As Savage puts it, “whatever we may disagree about, we are surely agreed that Bayes’s Theorem is true when it applies.” —- Savage 1962, circa p.72).

So “the probability that the result is due to chance” = p(e|H) for some H = p(H|e) . p(H), but the p-value doesn’t mention p(H) for any H.

Similarly, the p-value actually depends on the test statistic, but “the probability that the result is due to chance” has nothing to do with any test statistic.

Sometimes “the probability that the result is due to chance” has a clear meaning (e.g. as given by Bayes’s Theorem). In these cases, it’s usually numerically completely different from a p-value. Example —- Alan, got any ideas? Clinical epi?

It’s ignoring alternative hypotheses, even though there may be some that are part of the SAME theory as H that are (in every sense) better supported

In general it needs a tail area (at least in the continuous case and often also in the discrete case).

The probability that the results are due to chance, if interpreted in a way that makes sense, depends on having a model of how chance operates in the situation in question, which is straightforward except that there might be more than one such model. In that case, there’s an interesting ambiguity. What we really want to know is the probability that the results are due to chance on any of these models. But it’s often difficult or impossible to work that out.

Cox’s paradox and related forms of incoherence (and relationsihp with likelihood principle).

Relatedly, if a p-value WAS the probability that the results were due to chance then it would be straightforward to merge results from separate studies (at least in the case in which the studies have the same design, similar errors etc.); this IS possible with likelihoods and with posteriors but not with p-values.