Our theme for the month of October is “the elements.”
Last week, a meme gained steam on Facebook and Twitter that several polls had shown more than two-thirds of Democrats saying Bernie Sanders had won the first Democratic debate.
Still, pundits on almost every network—not to mention every live-interviewer poll since then—shows almost the opposite: Hillary Clinton had won the debate by almost the same margin.
Why? And how can you tell whether you’re looking at a good poll or a bad poll?
Our theme for this month is “the elements,” so here is a layperson’s guide to a few of the building blocks of a good public opinion poll – and how to quickly and easily tell what kind you have on your hands.
1. Does the poll pick the people? Or do the people pick the poll?
This is exactly what happened in the wake of the first Democratic debate. The fancy term for this is “self-selection.” The polls cited in that meme did not randomly pick people to take the poll—called “random sampling.” Instead, people selected themselves. This causes all sorts of problems, because certain people are more likely than others to choose to take the poll. For example, Sanders supporters are more likely to be higher-income and younger, while Clinton supporters are more likely to be lower-income and older. Which group do you think is on the Internet at 11:30 p.m. on a weeknight choosing to spend their time voting in an online poll? Bingo.
2. Does the poll use live interviewers over the telephone?
Okay, so let’s say that you’ve found a poll that picks its own people—or “sample.” The next thing to look for is whether the organization used live people to interview or used a robo-call poll. Robo-calls are nice because they are cheap, but they lead to lots of problems. It’s easier to lie to a machine than a person. For example, there’s no way to verify whether a person who presses two to say they are woman registered to vote isn’t actually a tweleve-year-old boy having fun. What if the person hears the robo-call question wrong? Or accidentally pushes the wrong button? The one other major point here is that laws say that you can’t robo-dial cell phones, so if you’re only doing a robo-dial poll, you’re not getting anyone who has only a cell phone—leaving a pretty big gap in who you’re choosing.
3. How does the poll pick its phone numbers?
So the fancy word for “who you’re choosing” is called “coverage.” The people who you don’t pick are called “noncoverage.” Some polls will choose their people using what’s called random digit dialing, which basically means that the company takes lots of chunks of 1,000 phone numbers that we know are residential lines and randomly dials phone numbers in those blocks. This results in the best “coverage.” Some polls use a voter registration list from the government, though. These can also work, but you have to ask how up-to-date the list is and how many phone numbers work. Believe it or not, it’s possible that up to half the list in some polls may have a non-working telephone number, which means that half of your sample (likely low-income voters or young people who have left home and have a new phone number) are excluded. There are some polls that are done with an online panel or something that doesn’t involve self-selection, but most have their own sampling problems with noncoverage: a topic for another day.
4. Watch Your Weight.
So what happens if, by random chance, you get 54 percent women and 46 percent men in your poll? But you know the country is actually 51 percent women and 49 percent men? Lots of polls use weighting to try to get the numbers to match the census demographics. Really good polls will use—hold onto your hats—an “iterative weighting model” to give each individual person a weight. For example, in the end, one sixty-five-year-old white male might be weighted to only 0.7 people in the final results because they are so easy to reach, while a twenty-six-year-old Hispanic woman making $50,000 per year might be worth 2.5 people in the poll because that group is difficult to reach. Weighting by age or by gender is usually really helpful, but weighting by political party can be dangerous, because it means that you’re assuming the turnout by party will be the same as a previous election (yikes!).
5. Last but not least, that pesky margin of error.
So what’s the deal with margin of error? It kind of seems like an insurance policy for pollsters in case the results are wrong. And that’s not too far off. The whole basis of polling rests on something called “inferential statistics” —like a blood sample. If a doctor takes a blood sample, she only needs a little bit to decide what all the blood looks like. Same thing with polling. You just need a few hundred people to get a pretty good picture of what the whole country looks like if you have good sampling—and that’s probably why you’ve never been called for a poll. But the more people you ask, the more exact your answer is going to be. So the margin of error says, hey, we know we are pretty close. In fact, if we did twenty more polls just like this one, nineteen of their results would fall within this margin of error.
Ryan Struyk (’14) graduated from Calvin with majors in political science and mathematics. He currently covers the 2016 elections for abc News in Washington. He’s also done political polling in New York City and reported on the Idaho state legislature for the Associated Press in Boise. In his free time, Ryan enjoys talking about inferential statistics, music theory, and his beloved Detroit Tigers.