spacer spacer spacer meaningless logo

The pointlessness of the point null hypothesis test

Thesis: Point null hypothesis tests (PNHT) are probably not worth conducting. They do not seem to tell us much about whether there is a causal relationship between variables of interest and it is hard to see why else a researcher would use this test.

Subheadings

Intro

What PNHT is

  • Definition of a point null and why I’m focusing on it

  • Definition of p-values

  • Summary of similar procedures (e.g. Bayes’ factors, your group sequential testing procedure for controlled trials)

  • Why PNHT is suspect

What it is not (and what it shouldn’t be)

  • Tests of point nulls probably aren’t equivalent to tests of small-interval nulls (reply to Meehl 1997 and Rindskopf 1997)

  • The point null isn’t the ‘probability that the samples were drawn from different populations’ (reply to Hagen 1997)

  • It doesn’t matter that failing to reject point null means shortest confidence interval contains null value in common cases (reply to various authors)

The ‘superpopulation argument’ for PNHT

  • The point nil (not null) hypothesis might be interpreted as the prediction that there is no causal relationship between variables of interest

  • Summary of superpopulation modelling

  • PNHT may be a test of proposition that the sampling distribution of the superpopulation follows the point null distribution

  • Even Bayesians might like this solution (Berger and Sellke, Casella and Berger)

Why the argument probably doesn’t work

  • Issues with superpopulation modelling (large, unknown variance)
  • The most common class of PNHT (tests generating p-values) would overestimate the evidence against the superpopulation point null

  • The prior probability of the superpopulation point null isn’t in the right range enough of the time to justify the use of point p-values (returning to Berger and Sellke, Casella and Berger)

  • And why PNHT procedures other than those involving p-values might face similar problems

Conclusion

  • What (if anything) should replace PNHT?

  • What this means for results based on PNHT