Fun with statistics: When survey results do not mean what you may think they mean

Fun with statistics: When survey results do not mean what you may think they mean

Spread the love

Gallup poll found that 52% of Americans say they were extremely proud to be American in 2016; 51% in 2017 and 47% in 2016.
Gallup says this is a record low.
Source: In U.S., Record-Low 47% Extremely Proud to Be Americans
At the very, very bottom of the press release, in slightly greyed out type, we read that the results had a sampling error of + or – 3 percentage points at the 95% confidence level.
Let’s rewrite the 2016-2018 results in terms of what they actually mean.

  • For 2016, the estimated mean was 52% but the actual population mean could be anywhere from 49% to 55%. They measured 52% by polling a sample of people, its possible that if we polled the entire population we could get a value anywhere from 49% to 55%. We are “95%” confident that our actual population mean lies within that range. Or, stated another way, if we ran this poll 20 times, in 19 out of the 20 surveys, we expect our estimated value to be between 49-55%; and 1 in 20 times, our estimated value would be outside that range.
  • For 2017, the estimated mean was 51% but the actual population mean could have been anywhere from 48% to 54%.
  • For 2018, the estimated mean was 47% but the actual population mean could be anywhere from 44% to 50%.

This is referred to as the margin of error or the confidence interval.
When the confidence interval of one sample overlaps with the confidence interval of another sample, we can not statistically distinguish between them. Statistically, they are equivalent.
Based on the survey data, the best we can say is that statistically, the percent of Americans who were extremely proud to be American has stayed the same (within our poll’s error margin) from 2016 to 2018.
Gallup can say this one poll found the lowest ever and that is pretty much what they are saying. But the public will interpret that result incorrectly. If they had done the same poll – again – on the same days, with a different sample of 1,520 people – they could have also gotten 50%!
During election season, news breathlessly reports that candidate A is ahead of candidate B this week, in a reversal of last week. But in fact, the two candidates are often in a statistical dead heat. We have no idea which one is ahead last week or this week!
The surveys cannot correctly identify the winner when the results are within the margin of error. This is why a poll showing candidate A ahead of B by 51% to 49% is then reversed on election night as candidate B wins with 51% of the vote. Nothing nefarious took place – because statistics!
In reality, we never really knew who was ahead  as the survey may have had a + or – 3 percentage point confidence interval (which is typical)!
Polls are also used extensively in propaganda messaging – opinion polls largely measure the effectiveness of earlier propaganda! Once they show support for a topic, the poll itself becomes a form of “Get on the bandwagon” endorsement. We covered this in a previous post.

Comments are closed.