The Problem With The Gallup Poll

From Steve Soto of The Left Coaster: Upon Closer Inspection, Another Gallup Poll Is Suspect

There are numerous ways in which polls can be misleading, as I suspect many of us are aware. I typically suggest to my research methods and stats students to ask themselves a few basic questions when interpreting poll data:

1. How were the questions worded?
2. What was the method of sampling?
3. If the pollsters claim to have used a random sampling technique, did they rely on simple random sampling or stratified random sampling?
4. If they relied on stratified random sampling, how did the pollsters define their strata?

Poorly worded questions can create problems either because the respondents can't understand the question. Biased or leading questions tend to result in data that are skewed in the direction of the pollster's biases. As far as I know that's not the problem with the Gallup Poll.

Instead, the problem with the Gallup Poll is one of sampling. Sampling methods can broadly be divided into probability and nonprobability sampling. Most cases of nonprobability sampling fall under the label of convenience samples in which the pollster simply questions anyone s/he can find. If you look at the nonscientific polls that CNN and MSNBC put on their websites, you'll see convenience sampling in action. Probability sampling is a method in which we can actually estimate the probability of someone being selected to participate in the survey. The two most common methods of probability sampling are simple random sampling and stratified random sampling. In simple random sampling, everyone in the population has an equal chance of being selected to participate. Sometimes a researcher wishes to divide the sample into subsets, and to make each subset proportional to what is found in the population. In such cases stratified random sampling is in order. A stratified random sample is one in which the pollster determines in advance the proportion of various subsets (strata) in the population and then randomly selects participants within each subset. This basically is what Gallup is doing.

Now that in and of itself is not a problem. What is problematic for Gallup is how they are defining their strata, their subsets. As Steve Soto notes, the proportions of Democrats, Republicans, and Independents specified by Gallup do not accurately reflect those proportions found in the population. In Gallup's case, the samples end up consisting of a disproportionately greater number of Republicans relative to what is likely to be the case in the population and a disproportionately smaller number of Democrats. Not too surprisingly, the effect is to inflate the numbers for Republican candidates and to deflate the numbers for Democrat candidates.

I'll refrain from reading anything sinister into Gallup's sampling methods. That said, I do think it is important to interpret the Gallup data skeptically and to question the validity of Gallup's results. In survey research, one's data are only valid to the extent that they generalize back to the target population (in methodologyspeak we call this external validity). At this point Gallup's results are of questionable generalizability, to say the least.