Why cellphones — and trust — may be affecting polling data

There was a time when pollsters went door to door to figure out what people were thinking. Gallup did that for almost 50 years before switching mostly to phones by the mid-’80s.

Phone polling was cheaper but still reliable. That is, until the cellphone came along. Gallup stopped doing presidential horse-race polling in 2012, but still asks Americans about their views on the sitting president as well as topics ranging from immigration to inflation.

Marketplace’s Lily Jamali asked Jon Clifton, CEO of Gallup, about the complexities of accessing polling subjects. They also talked about the “collapse of trust,” in Clifton’s words, and the many people who just want to be heard.

The following is an edited transcript of their conversation.

Jon Clifton: All the methodologies that we use are what we call probability-based. So we’re trying to get as close as we can to true random probability. And phones many decades ago got us almost exactly there, although we still had to pick somebody within a household once we randomly dialed up a household. But today, we can do that with address-based sampling. We can do it with phones. We can also do it with web, but only after we’ve recruited from those other samples.

Lily Jamali: Well, so help me understand what’s happening with response rates on polls. So again, you don’t do presidential horse-race polling, but you do do approval ratings on the president and all these other issues, polling around those things. So what are response rates like? We’re seeing pretty dismal data from other sources on this.

Clifton: Yeah. When you say, what are response rates like, the next question is where? Because you’re talking about the United States, they’re abysmal. But if you talk about countries in Africa, they’re very, very high. And in fact, when you also kind of explore issues about, well, what kind of methodology are you using?

Jamali: I want to go back to “abysmal” for a second. Abysmal here in the U.S., why do you say that? How bad is it?

Clifton: It’s bad. I mean, it’s so bad that much of the underlying data that we consume as Americans, even things like unemployment, come from survey research. A lot of people think that the unemployment number that gets provided to us, you know, through a CNN news break alert, through Axios, that kind of thing, that information on a monthly basis is a massive survey of 60,000 people. But everyone’s suffering from this problem because people are harder to reach. The other thing is that people are less trusting of organization[s], so it can cause them to be less likely to participate.

Jamali: Is this because of what’s happening to our phones? Ninety-seven percent of Americans have a cellphone, a quarter of us now have a landline. Is that why response rates are so abysmal, whether it’s for you guys at Gallup or the government agencies gathering the jobs data and inflation data we all look at?

Clifton: Well, it’s a number of factors. I mean, technology also changes our behavior. When we used to have landlines — it’s actually quite funny if we remember how it was, because if you called somebody at 3 a.m., they just picked up the phone no matter who it was that was calling. But now we can have a filter for that, which is don’t have your ringer on, and a lot of times people don’t answer their phone. So it is one of the reasons that it is harder for us to reach respondents.

Jamali: I mean, how significant would you say the phone shift from landlines to cellphones has been when you use that word “abysmal,” which is a pretty profound word, how much of it is attributable to that?

Clifton: Well, I think a large part of it is that. I do think there are other factors, like the fact that Americans are overwhelmed with the amount of people trying to reach out to them. I think there are a lot of people that are wondering, is the other person on the end of the line going to try to take advantage of me? So no matter what institutional guards that you have around that and protections that you can assure, and the law actually provides some of those things, it just makes things more challenging.

One of the things that we try to do is disclose everything about our methodology to anyone that’s reading whatever our results are. But when it comes to combating misinformation, I think one of the challenges is that when you look at the collapse of trust that’s taken place in the United States, one question we have to ask ourselves is are we wrongly attributing that sort of dissemination of misinformation to that collapse in trust? And I say that because there are other wealthy economies that are also experiencing social media and the rise of this misinformation, and they don’t see the same collapse in trust that we have.

Jamali: It’s worse here from what you say?

Clifton: Yes, that’s correct.

Jamali: Why do you think that is?

Clifton: Well, I think there are a few challenges with it. And I think it starts by defining “trust.” And trust ultimately comes from, what did I expect of you and what did you do? And the gap in between those things says, either you’ll reinforce the trust that I had in you or it breaks down. And oftentimes when there’s a gap, whether it actually was corruption or whether it was incompetence or a mistake, people often just assume that it was something nefarious that actually took place.

Jamali: And who’s the “you” in that sentence?

Clifton: I think largely the American public. I mean, right now —

Jamali: You’re talking about they have lost trust in officials, the media, all of the above?

Clifton: We’ve tracked for 55 years trust in institutions across the United States. And that trust has collapsed. On average, 50% of Americans said that they actually had trust in a host of institutions ranging from religious institutions, ranging from the Supreme Court to the presidency, everything that we measure, even journalism.

Jamali: So it’s 50% when?

Clifton: The average is 50% in the 1970s. It’s now collapsed down to 25%, on average, but all of them are falling, even things that Americans have traditionally had a great deal of confidence in. And we’ve even seen recent declines in trust in the military. So this is a collective decline. And I think, although it is a cause [of] misinformation, I would be surprised that based on the comparative data from other countries that is the proximate cause.

Jamali: And is there a feedback loop where the explosion of misinformation makes it harder for you to do your job at Gallup?

Clifton: I think the explosion of misinformation is challenging everywhere. And the only way that we can confront it is, again, by sincerely disclosing what it is that we did in our research. And again, I think one challenge that people have is when they’re consuming data is they believe that data is proving cause for whatever it is that they’re discussing. And I think that’s a problem because it’s not that we have direct evidence when there’s data. We have circumstantial evidence in order to advance something that we may believe in. So I think it’s very helpful that when people that are consuming information, I think this is largely true for a lot of individuals, that they’re looking at it and saying this is just yet another data point, even if it’s a collection of a great deal of data points, something that fits into an overall narrative.

Jamali: When people hear it’s the Gallup poll calling, what do you think goes through their mind? If you’re actually able to get through to that person on the phone, they pick it up and the person says they’re from Gallup — I’m assuming that’s how this works — do you feel like people are like, “OK, no, I still believe in this company”?

Clifton: Well, we take a lot of pride in the fact that we have tested that. And so we do see that we have slightly higher response rates. So that makes us proud of the fact that there is some trust that is maintained from the respondents when they participate. We’re also asking many respondents, not just the U.S., we’re trying to do some recruitment for panels around the world. And there are many that are still agreeing to participate. And I think that’s really helpful — all of us are trying to get closer and closer to the truth. And their participation means a lot to us. I know that even when government agencies are collecting their information, that data is incredibly important to make better-informed policy.

Jamali: Do you worry that we have lost our sense of ground truth in this country?

Clifton: No, no, I think we’re finding our way to seeing if it’s, and when, it’s OK for us to question things. And for whatever reason, I think the more that we say it’s not OK to question things, it actually makes things worse. But Americans, largely they understand — especially when you’re talking about their own communities or their own lives — they have a really good fix on what’s going on. And I know that a lot of times Gallup surveys are weaponized against the public. And they do so because we’ll come out with this survey, ask favorability or unfavorability of a country like North Korea, and 10% of Americans will say that they have a favorable opinion of North Korea. And they’ll say this is evidence that Americans don’t know what they’re talking about. But their perception of North Korea might be Koreans, it might be the actual physical country, not necessarily the leadership. And then you might have some that just have their own perceptions. Or maybe they don’t know. Interestingly enough, there are a lot of people that will even offer that when we ask them particular questions. They’ll say, “You know what? I don’t know.” And they’re fine with that. But again, when we’re asking about people’s lives, when we’re asking about their communities, for the most part they have huge awareness of what’s going on and what’s impacting their families.

Jamali: Tell me about the U.S. versus rest of world, as it’s often referred to in the tech community anyway. How that in terms of response rates, specifically, is it better abroad and why?

Clifton: It’s better abroad. I think one of our hypotheses is that they have not been inundated by a whole bunch of, potentially, phone calls coming in. But also people want their voices heard. And so there are a lot of places where we’re still doing face-to-face interviewing. In fact, in two-thirds of the world, we do face-to-face interviews, and not just in capital cities, but truly nationally representative surveys. Meaning we go out to the furthest parts of the world where there are very sparsely populated communities. And so I think the reason that their response rates are so high is because they want their voices heard. In fact, we’ll have other members of the community approach our interviewers and say, “Can I join in? Will you interview me as well?” And unfortunately, because of our sampling procedures, that isn’t a possibility. Sometimes if it’s a local leader, we might still do it for qualitative purposes. But again, it just goes to show that there are a lot of people in the world that just want to be heard.

Stories You Might Like

The good, the bad and the ugly of election polling

Why the definition of “employer” really matters

Measuring well-being in the U.S.

Job quality is getting worse during COVID

Does the U.S. have a modern poll tax?

A speed limit on Germany’s Autobahn? It’s a discussion again.