Preliminary Thoughts On Superpopulation Modeling
These are some initial thoughts I have about the ‘superpopulation-modeling argument’ in defense of p-values. These aren’t based on any philosophical reading I’ve done on the subject. Part of the reason why I’m putting these thoughts down is because there is to my knowledge no philosophical literature on this argument specifically. I’m interested in whether you think these thoughts touch on any widely-discussed philosophical issues.
- Superpopulation models stipulate that the finite population is a random sample of the superpopulation. But this is not obviously true of most superpopulations of interest, at least not in political science. In most cases, the chance that any particular finite population gets sampled is almost certainly not equal to the chance that any other population gets sampled. It is not even clear that the chance of selecting a particular population is calculable. [TOTES. BUT WHEN THIS ASSUMPTION FAILS, ISN’T THAT THE SAME THING AS SAYING THAT THE FINITE POPULATION IS NOT A GOOD REPRESENTATIVE OF THE SUPERPOPULATION, AND SO ANY ANALYSIS IS GOING TO BE WRONG? IN WHICH CASE MAYBE THIS KIND OF ANALYSIS IS NO WORSE THAN ANY OTHER? ALTHOUGH EVEN IF THAT’S RIGHT, I STILL LIKE YOUR ANALYSIS. JASON 2017-08-24]
Suppose I wanted to measure the relationship between GDP/capita and unemployment among all ~190 countries at the end of 2016. What is the sampling procedure? The procedure would probably include the economic, sociological and psychological processes by which countries develop political institutions and economic backgrounds. It might even include the geological processes that generate the land masses on which countries are situated. It is hard to believe that these sampling procedures will not tend to produce any particular kind of finite population. It is for example unlikely that any given finite population would feature societies whose members contributed to the economy for purely altruistic reasons. [HEAVENS. I DON'T AGREE WITH THAT. I GUESS WE DON'T NEED TO AGREE ABOUT ECONOMICS. JASON 2017-08-24] Nor would one expect to sample a finite population in which the Earth was covered entirely by water except for one country situated on an island that is 1km-squared in area. And how would you even begin to calculate the likelihood of observing any particular finite population? The idea that a hypothetical (superpopulation) sampling procedure is in most cases anything like an actual (finite) population sampling procedure is, as Kendall and Stuart put it, 'baffling.' [DAVID LEWIS DOESN'T KNOW HOW TO REFUTE A BAFFLED STATISTICIAN. (ARCANE PHILOSOPHY IN-JOKE.) JASON 2017-08-24]
- It seems as though the likelihood that the null is true of the superpopulation with respect to two variables is analytically (or metaphysically?) equivalent to the likelihood that there is no causal relationship between the variables. [RIGHT, THAT WOULD BE THE WHOLE POINT OF IMAGINING A SUPERPOPULATION IN THE FIRST PLACE, WOULDN’T IT? IF I’VE UNDERSTOOD CORRECTLY. (AND ASSUMING THAT BY “NULL” YOU MEAN “HYPOTHESIS OF NO DIFFERENCE”, WHICH IS NOT ALWAYS WHAT “NULL” MEANS - WHEN FISHER INTRODUCED THE WORD, HE USED IT TO MEAN WHATEVER HYPOTHESIS YOU WANT TO TEST. BUT THAT’S A SIDE ISSUE.) JASON 2017-08-24] If there is a statistical relationship between the two variables taking into account random variation, then this either means or is decisive evidence for the claim that a change in one variable causes a change in the other.
So what prior probability should we assign to the null (with respect to the superpopulation)? It's not clear there is an general answer to this question (as you point out in your thoughts on [Populations Considered As Samples From Meta-populations]()). If you knew nothing about Event A and Event B, is is not clear how you would work out how the likelihood that A and B were causally linked. And in any case the answer does not matter. In practice, we already know a lot about the events we are investigating. [OOH, GOOD POINT. JASON 2017-08-24]
We should assign higher prior probabilities to causal hypotheses which, knowing what we do about the events in question, seem more plausible. It looks like the claim 'people who spend more time following politics on TV are more likely to sign petitions' should attract a higher prior probability than the claim that 'people who spend more time following politics on TV are more likely to develop arthritis.'
But recall from Casella and Berger that the posterior probability of the null is similar to the *p*-value only for a relatively small set of prior distributions. [SADLY, THAT PARTICULAR ANALYSIS IS ONLY FOR POINT NULL HYPOTHESIS. GOOD POINT ANYWAY, BUT WORTH NOTING THE RESTRICTION. JASON 2017-08-24] So are the kinds of hypotheses that tend to be proposed in political science (and other social sciences) plausible enough that they fall within the range of prior distributions that Casella and Berger specify? I'm not sure how to answer this question. [ARE ENOUGH OF THEM SO THAT THE USE OF P-VALUES IN GENERAL IS NOT RIDICULOUS? ALMOST CERTAINLY NOT. JASON 2017-08-24] Is the prior probability of the null for the petition example pi=1/2? pi=1/3? pi=1/20? Should we assign a similar prior probability to most political-scientific hypotheses?
- Won’t the superpopulation model have an extremely high variance? [RIGHT! EXTREMELY HIGH AND EXTREMELY UNKNOWN! JASON 2017-08-24] The sample size of finite populations is always 1. [Right! I’ve published something you can cite about that, although you don’t need to read it. Jason 2017-08-24 - citation Grossman J. `A Couple of the Nasties Lurking in Evidence-Based Medicine’ “Social Epistemology”, 2008; 22(4): 333–352] So the standard error of the superpopulation parameter will be high. [HIGH AND UNKNOWN. I THINK THIS MIGHT BE A GOOD POINT TO CONCENTRATE ON, BECAUSE IT MAKES CONCRETE YOUR WORRIES IN POINT 1 ABOVE, RIGHT? JASON 2017-08-24] Combine that with the variance of the population samples and the evidence for or against any given hypothesis about the superpopulation becomes quite weak indeed. So won’t the p-value overestimate the evidence against the causal theory under scrutiny? [YEP.]