Why to Discount Polls

Part Three: Adjustments and Weightings

We previously looked at the Universe of those Polled, as well as Institutional Polling Bias. Now let’s take a look at what happens to the polling data after it is collected, but before it is disseminated. We’ll wrap up this series with why we actually care about polls, even though they’re often wrong!

Polls are completed over a time window, normally several days, and at the end of that time, the pollster looks at what adjustments need to be made. The findings are then both adjusted and weighted in several areas. Normally, this includes:

• Sample size

• Population Sample Adjustment

• House Effect

• Geography

For a poll to be legitimate, there is a minimum sample size. If the poll hasn’t reached that number of respondents, they cannot just pretend that they did by adding in the missing responses. But if you are analyzing a set of polls, you can adjust the weighted average predicated on the sample size of the various polls.

The big thing here relates to the adjustment and weighting of the population sample. Legitimate pollsters want the polling universe to approximately match reality. For example, in both 2016 and 2020, college graduates were oversampled, and this led to a polling undercount of Republican voters compared to Election Day. In 2022, 48% of Democrats were college-educated, compared to 30% of Republicans. Source. After 2016, pollsters did a lot of post-mortems on why the polls were so far off, and one of the outcomes was to weight for the non-college educated.

There are other weightings that relate to the population sample. One of the most famous was in 2016 courtesy of the LA Times. They weight for small samples, in fact, their total number of black male respondents was one 19-year-old guy in Illinois who was all in for Trump. I’m not making this up. It skewed not only their topline data on the popular vote, but fed into aggregated polls and skewed those averages. Not to mention the conservative narratives.

Incumbents start campaigns with the advantage of incumbency. If you never saw it, take a look at the 1992 Eddie Murphy flick A Distinguished Gentleman, where he gets elected to Congress because he has the same name as the Congressman who recently passed away. More seriously, in general, more than 80% of members of the House and Senate are reelected. Charts here. And so, pollsters often add what is termed “House Effect” to their secret sauce when making weightings and adjustments, especially for Congressional and state races.

And then we have geography. The Franklin and Marshall polling organization was run for many years by Terry Madonna before he retired. He used to underweight southeastern Pennsylvania compared to the rest of the state. In the interest of full disclosure, Terry and I went back and forth about this many, many, times until he finally made some adjustments. You can check their rating and see that they still have a ways to go. Sigh.

And overall downside is that pollsters may well OVERWEIGHT for a previous error in judgement. This article looks at how pollsters approached their 2016 errors, missed again in 2020 and what they are trying this year.

Polls can be incredibly far off base. In the recent elections in India, for example, a prominent pollster cried during a TV interview because his poll results were so far off base. His polls indicated that Modi’s BJP party would sweep more than 400 seats, and they ended up not clearing the 272-seat bar for a majority.

Another far off polling story comes from South Carolina in 2010. A candidate named Vic Rawl was running in the Democratic primary to take on then-incumbent Senator Jim DeMint. While I was not involved in the campaign, I was friendly with Rawl’s campaign manager. Polling looked good: Rawl had a strong background, he advertised, he campaigned around the state, and he would have made a strong challenger in the fall. He ran against a guy named Alvin Greene. Alvin was unemployed, took care of his dad, and was facing federal obscenity charges. While he did pay the ten grand to file, his only other campaign expenditure was a bumper sticker on his truck. He didn’t campaign, he didn’t have any signs, didn’t advertise, wouldn’t debate. And he won, 59%-41% in a low turnout election. I am not making this up. And yeah, there is a good chance he was a Republican plant, due to Rawl’s potential strength against DeMint. By the way, Alvin is a current Convicted Felon Trump supporter.

Internal campaign polling is always more accurate than public polling. The campaigns are incredibly strict about their universes polled, they know what they are looking for, and use responses to, among other things, include persuadable voters in their outreach. Those internal polls might miss by a point or two, which is critical in a close race, but generally can be trusted. In general, these data are not released to the public, only available to those inside the campaign.

So why do we care about the polls? Face it, none of us ignores them – they give us both hope and sorrow. Most of us cannot look away when we see those headlines (no matter how wrong they may be).

Polls, especially when aggregated, can provide trendlines, which is important to the campaigns because it helps them in targeting, messaging, and deployment of resources. For the general public, looking at simulations drawn from multiple polls can give an idea of what may happen, although the only “poll” that really counts is on Election Day. If you want to look at a simulation, 538 has a good one, which will change over time. Their simulations are Monte Carlo simulations where they look at who has what probability of winning the general. It’s similar logic to financial Monte Carlo simulations generating the percentage probability that your savings will last in retirement.

Wrapping up: polls can be wrong for many reasons. And yes, sometimes they are accurate, but in close races the pollsters always include a margin of error which is often greater than the difference between the polled candidates’ numbers. So if a race is 51-49 with a 3-point margin of error, the topline numbers don’t matter.

For those of us who really, really, REALLY care about the outcome of any election, we always assume that our candidate is 10 points down no matter what the polls say.

There is one caveat to this - polling on issues is generally more accurate. For example, ballot initiatives often generate numbers that track with the outcome of the election. People will be voting “yes” or “no” and the vast majority know where they stand.

Thanks for reading!