The good, the bad and the ugly of election polling

Make Me Smart June 4, 2024 Transcript

Note: Marketplace podcasts are meant to be heard, with emphasis, tone and audio elements a transcript can’t capture. Transcripts are generated using a combination of automated software and human transcribers, and may contain errors. Please check the corresponding audio before quoting it. 

Kimberly Adams 

Hello everyone, I’m Kimberly Adams. Welcome back to Make Me Smart, where none of us is as smart as all of us.

Kai Ryssdal 

I’m Kai Ryssdal. It is Tuesday, the fourth day of June. We are, let’s see. July, August, September, October. Five months and a day, I think, from the presidential election this year. Polls are everywhere. But also, polls, hmm.

Kimberly Adams 

Polls, polls, polls. That’s right. We wanted to know more about the limits and challenges of polling, and what roles poles should have. That’s hard to say. Roles polls should have in shaping and understanding our elections. So, here to make us smart about this is David Dutwin. He’s the senior vice president of strategic initiatives at NORC, a research organization at the University of Chicago. You’ve probably heard it when people will say on TV, NORC AP poll or some other NORC associated poll. But anyhow, he is also the former president of the American Association for Public Opinion Research. Welcome to the show.

David Dutwin 

Thanks for having me.

Kimberly Adams 

So, first of all, what can polls tell us? And what can they not tell us?

David Dutwin 

Well, I would maybe start by saying that polls they’re designed to tell us what the electorate is feeling at the moment of the polling. And this is an often misperception that polls are designed somehow to project into the future and be predictive of an event that’s going to happen in the future. They’re designed to tell you what the electorate thinks today. And of course, when you’re five or even seven or eight months out from an election. Naturally, the electorate is not as well informed as hopefully they will be. And studies show that the accuracy, if you were to try to predict from those polling, the accuracy is not nearly as good as polls that are taken, of course, in the last week or two of an election. The other thing media outlets like to really focus naturally on the horse race question, you know, who are you going to vote for in the upcoming election? But polls tend to field with dozens of other questions about issues about sentiment, feelings about the candidates’ positions on issues. There’s a lot of rich data there, doesn’t always make the headlines. But there’s always a great deal of information that can be learned from polling if people dive in and look at the actual results from the full questionnaire.

Kai Ryssdal 

For those, though, David, who don’t do the crosstabs and don’t have a great wealth of knowledge of the ins and outs of polling. What should one look for to know that it is a good and quality poll and the snapshot it gives is reliable?

David Dutwin 

First of all, is it a reputable organization that has a long history of conducting polls? Secondarily, is the poll done through probabilistic methods? I know it’s a bit of a technical term. But probabilistic methods are polls that are done by sampling in a random fashion. So, I always say a good a good parallel to the word science is randomization. I think most listeners will understand. If someone has a new drug, and you randomly put half the people into a placebo like, they get a sugar pill and half get the actual drug, you can really isolate the effect of that drug. With surveys, the how the randomization occurs is that we have a pool of, say all addresses in the United States or all telephone numbers in the United States. And we will mix them all up and then randomly sample and think about a bowl of tomato soup. When you have a nice, you know, blended bowl of tomato soup, nobody questions that each spoonful is a perfect representation of that tomato soup, right? And that’s really the principle of probabilistic polling. The problem comes in what’s called nonprobability polling. This is polling done through convenience. So, you may have shopped somewhere and at the end of that shopping experience online, you get a hey, well, you know, can you take a survey? Or you belong to an airline’s mile program and all of a sudden you get a request for a survey through that miles program. So, these are not done through a random process, but rather than done through the convenience of, oh, you know, we know people shop here and we can sort of pick them off. But of course, you know, people who fly a lot on an airline are probably not representative of the full cross section of people in the United States. So, the last thing I’ll say about that is transparency. A lot of organizations, we push it the American Association for Public Opinion Research for our organizations to be as transparent as possible, disclose every little detail about how you conducted the poll and make that transparency readily available through what’s called like a methodology report or some sort of disclosure. These should be easily linkable. When you’re reading about a poll, say on a on a news media outlet website, there should be a link somewhere in there where you can really see that information. You don’t even have to necessarily know what that information is saying, down to all the technical details, but just knowing that that organization is willing to disclose every little detail about what they did, perhaps warts and all, is probably indicative that they’re well reputable organization compared to an organization that gives you almost no details whatsoever in terms of how they sampled, how they fielded it, and how they executed that poll.

Kimberly Adams 

But with the understanding that the vast majority of people are not going to do that and will just see the polls on TV or hear about them. What do you think we should be doing with the information that we’re receiving about polls?

David Dutwin 

Yeah. I think first of all, it’s important to note that. A couple of things, the accuracy of polls has always been and indeed, when people measure this through the history of polling has been quite flat. Polling error is not larger today than it was 20, 30 years ago when it comes to the accuracy of a horse race questions and election questions. That said, any single poll can have some error, and that error range might be very small, or it might be somewhat larger. And so, I would encourage listeners and folks who are interested in polling to understand that a single poll is not some sort of gospel truth necessarily that one should kind of consider it in the stream of polls. One of the, if you will, advantages of the modern political environment is that instead of having just a handful of polls that might come out in a given election, which was the case 20,30 years ago, there are literally polls coming out almost on a daily basis in that last month or two of the election. So put the poll you’re considering at the very moment you’re watching or listening or reading about its results in context with the other polls around that. There are actually some great websites that aggregate polls, and you can kind of see in a very quick graph, sort of like, okay, you know, how does this poll compare to other polls? And, you know, understand, of course, I think most people who have heard polling on TV or in the news somehow know the term margin of error, or have heard the term margin of error. It’s a very important point to make that margin of error is real. When people say that the accuracy in this poll is say plus or minus 3%. That is true. And you have to really fight as a listener to not say, okay, this poll says that candidate X has got 52% of the predicted vote. That it’s not 52%. It’s really 49, you know, to 55. And guess what? If it’s 49, that candidate is going to lose. And this is kind of a challenge in the world we live in now, where races have gotten incredibly tight, that unfortunately that margin of error oftentimes will fall into having a four at the beginning of the numbers compared to a five at the beginning of the number. And so that poll really just can’t say reliably who’s going to win the election.

Kai Ryssdal 

Alright, so let me ask you a question guaranteed to make your head explode, but it has literally ripped the pages of the New York Times today. The election in India, Narendra Modi, his party is going to win a plurality of seats, but they’re not going to get the landslide that had been predicted. So, when casual observers of the news on this election or any other see that, what do you want them to say to themselves about polling and validity?

David Dutwin 

Sure. I think there’s a couple of reasons why this sometimes happens, where polls are saying there’s going to be a certain outcome and then something different happens. And there are valid reasons and sort of good reasons and bad reasons if you will. Good reasons being it’s no fault of the pollsters. It’s reality, momentum effects.

Kai Ryssdal 

Wait, wait, wait, wait, wait. Sorry. Sorry. Sorry. Sorry. Sorry. Did you really just say it’s not my fault?

David Dutwin 

Well, I mean if a poll. It is very typical that pre-election polls happen, you know, three or four days before an election, and there are many, many examples of elections that have substantial election momentum.

Kai Ryssdal

Right.

David Dutwin

Where public sentiment is shifting in that last week, very sizably. This isn’t the best example. But I was in the field with a poll in the state of Virginia, the poll started three days before the Comey letter came out regarding Hillary Clinton. That poll was a six-day poll, the first three days before the letter went out, she was plus six. The last three days after the letter came out, she was negative for in that poll. That’s how fast public sentiment can sometimes change. And it’s hard for polls that do tend to field in three to six days, sometimes to capture that kind of momentum. And I’m not trying to absolve pollsters.

Kai Ryssdal

No, I get that.

David Dutwin

You know, there is a real concern. And I’ll tell you what the big concern is for that pollsters worry about right now. And that is that the people that agree to take a poll are in some way significantly different than the people who declined to participate when asked to take a poll. Now, generally speaking, it’s always been the case that there’s plenty of people that refuse to participate when asked to do a poll. And the good news is that historically, it’s generally been the case that the people who decline look so similar to the people who agree that it doesn’t hurt the accuracy of the poll. Right now, I think we’re worried and facing a moment where there’s a significant percent of the American electorate in particular, and I think this is true in other countries, too that really are distrusting of science, distrusting of media as an institution. And you can imagine if you distrust science and media, if you get a call to do a poll for a media outlet, you’re much more likely to say no. And it turns out that, you know, in the last 12 years or so, such people have voted for one particular candidate more than any other candidate. And so, there’s concern that if this phenomenon continues, polls will have more error than the past. And pollsters are working really hard to try and figure out ways to encourage, you know, the full spectrum of voters out there to equally want to participate in polls. And also, there are statistical adjustments we can do on the back end to try and sort of fix the bias that we know might be happening from these kinds of people. But it’s been a work in progress for a couple of election cycles now, and it’s still something that I think a lot of pollsters struggle with.

Kimberly Adams 

You mentioned that. You’re talking about Trump voters being not so likely to want to participate in polling. But are there also demographic groups that are a struggle to capture? I mean, we often hear about Gen Z voters being hard to capture because they don’t answer the phone from strangers. And they don’t necessarily want to opt in to polls. I mean, what other groups are hard to capture right now in a way that it may kind of skew what we learn about how people are feeling about this election?

David Dutwin 

Yeah, sure. So historically, polls have had to work harder than they normally do in order to get a really good sub sample, particularly of people at most a high school diploma. That’s been one of the biggest challenges for decades in polling. It hasn’t hurt election polling in the past because in the past such voters first of all, they don’t vote as much as say, college educated folks. But secondly, when they do vote, they kind of have been split equally across Republican and Democratic. In this day and age, those voters are largely Trump voters. And so, there is a challenge or at least more so than in the past. And so, capturing enough of them is definitely a challenge. Getting the young is a challenge, getting Spanish speakers is a challenge. And to be clear, you know, every poll does get a decent number of these three groups. But not in the numbers I think we would like to see. But you know, these things are fixable. We know how many people in the United States are, say under the age of 25. We know how many people in the United States have never spent any time in college. And these can be fixable. Again, the bigger challenge is things that are harder to fix because we don’t have census data on them. And you know, the best way to put it is when we look at polls and we look at what percent of the respondents are Republican versus Democrat. Polls are doing really, really well, getting a good cross section of people across party identification. So, it’s almost less worrying about do we have enough Republicans on our poll as to like within Republicanism? Do we have enough people who are say strongly pro-Trump versus weakly pro-Trump? And those are, that’s something that’s very hard to assess. But I know a lot of my colleagues have been working for a long time and developing, at least for the last decade of developing techniques to try and deal with this challenge. And I think there’s some really good science in place. And that’s something I really want to mention here. You know, polling has had challenges for 20, 30, 40 years, every decade, we seem to have something. You know, we used to poll all the time by landline telephone. Remember what those things were? That wired thing on your kitchen wall?

Kai Ryssdal 

Oh, Kimberly’s got one, Kimberly actually has one, so we are all familiar. Thank you very much.

Kimberly Adams 

And the only calls I get on it are solicitations for money or polls.

David Dutwin 

Yeah, exactly. Well, you know, I mean, we had to deal with the fact that all of a sudden, that nobody had or very few people have landline phones anymore. And for a while the field was like, wow, you know, what are we going to do? We got this new thing called cell phones, and how are we going to use them? You know, what the field met that challenge, overcame it and, and moved on. It seems like every 10 years, there’s a major challenge that polling faces. And the good news is that we are really dedicated scientists. We work incredibly hard to hone our craft and improve our craft in a scientifically valid way. And improvements, you know, and challenges had been met every decade that something has come up that has sort of tested the validity of polling. Those challenges have been overcome. And I think we’re doing that again today. Maybe not as fast as we would have liked, you know, be it the results in, say, 2016. But we learned, you know, APPOR, the American Association for Public Opinion Research, every year has a very large postmortem report, were they work through all the different.

Kimberly Adams 

That thing was like four inches thick. I printed the 2016 one.

David Dutwin 

You know, so we look inward, and we try to really learn from the challenges that we face and improve our craft on an annual basis. And it doesn’t mean that there still won’t be moments where we have these India moments where the polls say, a large victory, and turns out, it’s more of a smaller victory. And again, you know, it’s a never-ending cycle right of improvement assessment, and refining the science of polling for the next election cycle.

Kai Ryssdal 

David Dutwin is the senior vice president of strategic initiatives at NORC. That’s at the University of Chicago, also a former president of the American Association for Public Opinion Research. David, thanks a lot. Appreciate your expertise.

David Dutwin 

My absolute pleasure being here.

Kimberly Adams 

You know, it strikes me that polling is very similar to earnings reports, in that the news is when there’s a miss and not when it’s right.

Kai Ryssdal 

That’s right. Yeah, that’s right. That’s totally true. Good point.

Kimberly Adams 

And the reason I printed out that 2016 report was because I used to teach about it when I was teaching a college class.

Kai Ryssdal 

Oh, come on. You were just reading it, you nerd.

Kimberly Adams 

Maybe. Maybe. Anyway, but one of the things that we discussed was this idea. You know, he was just talking, David was just talking about how these are serious scientists who take this work very seriously. They approach it from the scientific method. Yes and. There are also political pollsters who are not, who have a financial vested interest in working for campaigns, or who are working for the shops that sell candidates and campaigns services based on what their own polls say. And sometimes I see those polls coming out. And I’m just like, is this? A campaign will talk about its own internal polling. For example, you’ll hear a lot. The Biden campaign is constantly talking about, “oh, don’t believe the polls. Our own polling shows this. Our own polling shows that.” Well, that polling is coming from people and businesses selling you services based on what that polling says. Now, that’s not to say that it’s not real or that those people are not also scientists, but I think it gets a little different when there’s a financial vested interest. I was listening to the 24sight Podcast the other day, and they were talking to some Republican and Democratic pollsters and one of them said that they are struggling to get young voters of color, particularly young Black men was what they were talking about and Latino men. Not because they weren’t participating in the polls, but because they have these test questions that they throw into a poll just to make sure that people are paying attention, and that they’re not just blindly checking things in order to get whatever reward they were going to get at the end for participating in the poll. And they’re getting their results kicked out on the test question at higher rates than other groups. And so, something as simple as that is skewing the data, but it’s a very interesting topic.

Kai Ryssdal 

No, look. It totally is. I’m just. I’m just making a little bit of fun. Just a little bit of fun.

Kimberly Adams 

It’s fine. Look, I let my nerd flag fly. I’m proud of it. All right. We’d love to know what you all feel about polls because surely you dedicate as much time as me to thinking about polling and the science behind it. Do you pay attention to them? Do you trust them? We’d love to know. Do you find yourself often at the receiving end of being asked to fill out polls. Our number is 508-827-6278, also known as 508-U-B-SMART. We will be right back.

Kai Ryssdal 

And news, go.

Kimberly Adams 

So, speaking of polling. One topic polling very high this election cycle as important to voters in surveys and polls is immigration. Of course, I was looking at some Gallup polling. Gallup surveys, I should have asked, there’s like a difference there.

Kai Ryssdal 

I mean, we honestly could have gone another half an hour truly.

Kimberly Adams 

Yeah, for sure. But anyway, Gallup does surveys all the time. And the partisan shift in viewing immigration as a key issue 2014 to 2024, huge spike in the last year with Republicans viewing it as a much more serious issue than Independents or Democrats. This was from end of April, 48% of Republicans saying that immigration is the most important problem facing the country today, versus 25% of Independents and 8% of Democrats and all of that coming in very interesting timing. Not interesting at all, obviously, very political timing, that there’s news today that Biden is going to be out with new executive orders related to immigration, including being willing to shut down the border when crossings top more than 2500 a day over a seven-day period. That threshold has already been hit this past, over the past week, which means the border is probably about to get shut down. There are a bunch of exceptions to this, but this is one of a series of steps that the Biden administration is trying to take to get this issue under control. They know it is a weak spot for them in this campaign, and that it’s as an issue starting to not just be an issue for Republicans, but it also is starting to really be of concern for Democrats. And I find it very interesting that the narrative from the Biden campaign through the whole administration about immigration was that Congress needs to do something. Congress has to be the one to do something. Congress must take action. And they’ve said, oh, you know, the president can’t act unilaterally. But there have always been steps that the President could take, and we are seeing some of those now. And I imagine that that is going to be the response on the Republican side like see, this whole time. He could have done something if he wanted to.

Kai Ryssdal 

Yeah, I think I saw Mike Johnson today saying exactly that.

Kimberly Adams 

Yeah. Yeah. And, and they kind of set themselves up for it.

Kai Ryssdal

Yep. Yep. Absolutely. Absolutely.

Kimberly Adams

That’s my news. What do you got?

Kai Ryssdal 

So, mine is a little more it’s a bad news, good news thing. So, we got the JOLTS report today, job openings and labor turnover survey. The good news, and I mean, this is it. There are fewer job openings in this economy right now. That’s exactly what the Fed is looking for. And more importantly, the slowdown in hiring and opportunities is everywhere, right? Health care, manufacturing, government jobs, hotels, and food services, all kinds of stuff. So, so it’s really, really good news, not for individuals who are looking for jobs, although there are still plenty jobs in this economy. But in the very macro sense, this is exactly what the Fed wants to see. And for all the talk that there has been the last six or so weeks about, oh my god, it’s going to be rate hike, not a rate cut. This is what you’re looking for. And I will also just, you know, put on your calendars: Friday morning, 8:30, Washington time, the May monthly unemployment reports coming out. So, this is, you know, it’s good news on the labor front, which is, which is what we want to see.

Kimberly Adams 

Isn’t it so weird when we report on economics, the things that count as good?

Kai Ryssdal

Oh, I know.

Kimberly Adams

Fewer job openings, which is, you know, good news for what the Fed is trying to do. It is good news for businesses that you know, have what they need and don’t, you know, aren’t trying to scramble to find workers. But if you’re out there trying to find a job, it’s obviously not great news for you. But yeah, it gets so bizarre sometimes being in this business, and it turns you topsy turvy thinking like well, bad for individual people, good for the economy, good for, you know, good for individual people bad for the economy. Anyway, that is it for the news. Let us move on to the mailbag.

Mailbag

Hi Kai and Kimberly. This is Godfrey from San Francisco. Jessie from Charleston, South Carolina. And I have a follow up question. It has me thinking and feeling a lot of things

Kimberly Adams 

Last week, we talked about how corporate consolidation in the food industry has affected our food system, including what you and I eat. And we asked what you all think about Big Food and we got this response.

Reginald

My name is Reginald from Scotts Bluff County, Nebraska. I’m calling from the seat of my tractor, planning the last of my summer crops. I’ve devoted my second career to countering Big Food. Although for me, it’s more about making better food than it is about sticking to the barons who have monopolized our food supply chains. I’m a retired Navy submarine officer who returned to the family farm to build a flour mill on that farm that will mill and distribute stone ground flour. It turns out flour made on a stone mill preserves nutrition that is so necessary for healthy living. So, I’m now building my own farm company to help solve the Big Food problem in at least some small way.

Kai Ryssdal 

Oh, that’s awesome. That’s very cool. Very, very cool. That’s great. Okay, one more and, okay. Well, you guys decide. So, last week for President Trump was of course, convicted on Thursday. And everybody was out there saying it’s a historic day. It’s a historic day. And I said no people. I went on Twitter, and I said, it’s an historic day, not a historic day. Got a little traction on the socials.

Kimberly Adams 

You got some trolling.

Kai Ryssdal 

Well, look, we’re going to get some more trolling too. Here it comes. This is Melissa in Los Angeles.

Melissa

Hey, Make Me Smart. One thing that I learned when I was working as a book editor that I didn’t know before, was that it is actually a historic day in the United States. And that’s because we don’t pronounce it historic, like the British. And if we did that, then it would be an historic day. But here in the US, it’s a historic day because we pronounce the H hard at the beginning of the word. So, we don’t need the an to indicate that it’s a vowel sound at the beginning of the word.

Kai Ryssdal 

Well, so, Melissa, first of all, thank you. I appreciate that. Second of all, I will say language is a changing beast, and we all use it in our own ways. Also, it’s GIF. Not GIF.

Kimberly

Okay, no Kai. You can’t say. You cannot sit here and say, language changes and we all use it in our own ways while getting out there on Elon Musk’s platform and telling other people that they were wrong about saying a historic day. You can’t have both of those things.

Kai Ryssdal 

I can do whatever I want.

Kimberly Adams 

I had multiple people send me your tweet be like, really your co-host is on this issue today. And then, I look at the comments and I saw a mix of people with that same idea like, how can you be so pedantic on such an important day like today versus the grammar folks who were literally going at it about this.

Kai Ryssdal 

There was no small amount of support for my position as well.

Kimberly Adams 

There was some support. There was some support, but I was very entertained by the whole thing but you enjoy your time on X. I will keep my distance. You all will find me on Mastodon and Bluesky, I’m rarely there, and LinkedIn for what it’s worth. Anyway, before we go, we’re going to leave you with this week’s answer to the Make Me Smart question, which is what is something you thought you knew, but later found out you were wrong about? Let’s hear it.

Grant

Hello Kai and Kimberly. This is Grant in Somerville, Massachusetts. Something I thought I knew, but later found out I was wrong about is Kyla Scanlon’s vibecession. Since its first mentioned on the Marketplace main show, I always assumed that the term being used was vibe, like the mood and session, like a session IPA. It wasn’t until the interview on May 30, that it hit me that the term being used was actually vibecession like recession, which after thinking about for more than two seconds, makes a lot more sense. Thanks for making me smart.

Kai Ryssdal 

There’s a word for that phenomenon, right? Where you hear something and you kind of get it wrong. Yeah, somebody’s going to write in and tell us about it.

Kimberly Adams 

I also don’t know what a session IPA.

Kai Ryssdal 

Oh, session IPA is lower alcohol and you can you know, sort of sometimes they’re called all day IPAs, you can drink them, you know, it’s like 4% ABV and you can drink it without getting you know, too over the top as it were.

Kimberly Adams 

Huh, I didn’t know that. I learned something new. I appreciate that.

Kai Ryssdal 

Every day on the pod. Anyway, Kyla Scanlon, financial educator now author. She’s got a book out called; I think it’s called “In This Economy.” It’s her lessons and observations about how to think about this economy. We will put on the show page obviously, interview runs about seven minutes and you should you should subscribe to all her socials too, because she’s super funny. It’s @kylascan, I think. And let’s see, what else? If you need to get a hold of us, 508-827-6278 is how you can do that. 508-U-B-SMART.

Kimberly Adams 

Make Me Smart is produced by Courtney Bergsieker. Ellen Rolfes writes our newsletter. Today’s program was engineered by Juan Carlos Torrado with mixing by Jake Cherry. And our intern is Thalia Menchaca.

Kai Ryssdal 

Ben Tolliday and Daniel Ramirez composed our theme music. Our senior producer is Marissa Cabrera. Bridget Bodnar is the director of podcasts. Francesca Levy is the executive director of Digital. Marketplace’s Vice President and General Manager, but only on Tuesdays, is Neal Scarbrough. So, that’s the test of whether or not he listens to the end of the pod.

Kimberly Adams 

I’m sure you’ll hear about it later.