Why to Discount Polls
Part One - Universe of those Polled
In the interest of full disclosure, I read the polls. And yes, I spend about 2 seconds happy or sad. If it’s a legitimate poll, I might read the crosstabs because that’s where the secret sauce is housed.
Mostly, though, I am well aware that polls are bleeding indicators, and not leading indicators. In addition, they are nowhere near as valuable as they used to be. And more often than not, they are wrong.
Polls are wrong for several reasons:
Universe of those Polled
Institutional Polling Bias
Adjustments and Weightings
But first, a bit of sympathy for pollsters. Polling can be expensive if you do it right: The pollster needs to acquire accurate lists of registered voters, and their valid phone numbers, make calls until they get a reasonable number of voters, and then adjust the results.
This is really hard, first, because few people use landlines anymore, and when people move, they keep their cell phone numbers. This means you might be polling New York voters, but one of the people called now lives in, say, Colorado, and you, as a pollster, have now paid for an unusable call. Why is that person still on the NY voter rolls? Because when people move, they generally don’t cancel their old voter registration, and if they move out of state, they might not be removed from the rolls of the old state for years. 1
Second, people don’t answer their phones from numbers they don’t recognize. This is true also of polls that are texted out. People don’t respond to texts that might be hinky.
And this is before we get to internet polls.
Universe of those Polled
It’s hard to generate a representative universe – who will respond? Will they be registered voters or likely voters? Are they who you think they are?
There is a huge difference between “registered” voters and “likely” voters. Everyone has access, if they want it, to voter information. Yes, you can too. 2 Part of that data indicates a voter score (which is called different things in different states). Sometimes it is a listing of which elections a voter has participated in, and whether in person or by mail (if that choice is available). Sometimes it’s a percentage, e.g., Voter 1 has participated in the last 80% of general elections and 50% of primaries over the preceding X years. A higher voter score correlates to the value of the polled person’s response. If someone is a “SuperVoter” (voter score of at least 95%) you can trust that they will be voting in the next election, while someone who registered 5 years ago and never voted, or voted only once is far less likely to vote in the upcoming election.
In May, the NYTimes/Siena poll made the error of including registered, but unlikely, voters in a poll they undertook. If you open the link, and scroll to the far right, you’ll see the proportion of people who didn’t vote, and they were even included in the “likely voter” screen. Link.
In that poll, Trump seems to be leading on the top line, but if unlikely voters are removed, President Biden holds the lead. And yet, NYTimes/Siena is the highest rated pollster. Nate Cohn, who runs that polling operation, is considered, along with Ann Selzer, link, to be a top pollster for both running polls and interpreting results. And yet, big error last month.
Some pollsters have adopted internet models for polling – sometimes off registered voter lists, and sometimes just open to anyone who responds. In the former method, there are pollsters, like Franklin and Marshall, who use voting lists and then send post cards to voters saying that they will receive an email asking them to participate in their poll, and the emails have an identifying number so that the pollster knows the demographics of the respondents. Other internet pollsters, like YouGov, have people register on their site providing name, address, phone, email and other info that enables YouGov to get a very clear sense of who comprises their universe.
In other cases, there are “pollsters” who just conduct internet polls that anyone can answer, and if they want, they can answer multiple times. Garbage in, garbage out.
WHO is polled matters greatly in the accuracy of the information put forth in the top line and crosstabs of any poll. And so, reason one for discounting polls relates to pollsters not presenting a legitimate cross-section of people who will actually be voting in the next election.
If a pollster gets the universe wrong, the results will potentially undercount a certain type of voter, or overstate an outcome, and we’ve seen this in many polls when the outcomes are compared to actual tallies on Election Day.
Tomorrow, we’ll look at the institutional bias of pollsters.
Thanks for reading. Please leave comments and questions, and if you appreciated this post, please tell your friends!
ERIC, the Electronic Registration Information Center helps states keep their registrations up to date when people move between states. Sadly, only 24 states and DC participate. Link
If you would like a voter list, they are available from Voter Services. This might be your city, or your county. While there are some differences between locations, normally, you’ll get the name, address, phone number, political party (if registration by party is required at the state level), age, and voter score. Voter registration information is public, so you can access it. Sometimes a nominal fee is charged. If you are involved with a party or a campaign, they will have a database and can provide you the information, potentially with additional information such as issues. If you want a voter list and don’t know how to get it, message me with your location, and I’ll help you.