Can You Trust the Polls? Insights from Micah Roberts of Public Opinion Strategies

[00:00] Neil Hughes: So thank you for joining me on the show today. Can you tell me a little about who you are and what you do? [00:05] Micah Roberts: Yeah, thanks for having me, Neil. I'm Micah Roberts. I'm a partner at a firm called Public Opinion Strategies. This is a public affairs and public opinion firm. We do political polling and we do public policy polling. And I'm a Republican pollster. Say that 5 times fast. That— so we work on the Republican side of issues and then also tend to work selectively on projects with Democratic pollsters when that need arises. I've been doing this work for nearly 20 years with this firm, and I've been a partner since 2016 with this firm. And we are arguably the largest, most successful Republican polling firm in the United States. [00:50] Neil Hughes: And I suspect if people are, I don't know, scrolling on their news feeds here in the UK or the US, we all know about opinion polls. They used to drive electoral messages and sway voters, etc., but we don't know too much about what goes on under the hood. So let's just begin with the basics. What is public opinion polling and how does a pollster turn the views of a relatively small group sometimes into a reliable picture of a much larger population? [01:21] Micah Roberts: That's a great question and one I get all the time. So public opinion research goes beyond really just the surveys that we do. It also involves sitting one-on-one or in a group doing focus groups and listening to voters or to constituents or to people, customers, what have you, whatever the project requires. So there's a lot of face-to-face stuff that supports the opinions, the views of public opinion that we understand as pollsters. But most people just see the public polling that's out there, whether that's from NBC News or CNBC, which are 2 polls that I mentioned first because those are 2 that I have involvement, personal involvement with. I've been working on them for each of those for over a decade with my firm and my partner, Bill McInturf. And those are also bipartisan polls. But you can see, you get polls from, I'm sure the BBC does them over there. But here, you know, Fox News, CNN, Quinnipiac, New York Times, Washington Post. I mean, it's, uh, there seems to, there seems to be new numbers coming out all the time that the public can see. And I think what people may not know, Neal, is that the stuff that they see from the public, uh, whether it's newspaper or cable news channel or what have you, that public polling is probably just the tip of the iceberg in terms of the amount of polling that gets done. For campaigns, organizations, associations, corporations, what have you. So that's a very small part of the work that actually happens. And, but all of that informs us as pollsters about what's happening in the country, or if we're in a statewide race or ballot measure, helps us, helps us understand the views and opinions and where people are, where voters are, where people are. across the country. So hopefully that's a little bit of a broad brush. What did I miss? [03:29] Neil Hughes: Now, I would just say one of the things I was really excited to ask you is you've been in the heart of this space for a considerable amount of time now. And just for people listening, you have been part of the bipartisan NBC News polling team since 2010, Republican half of CNBC's bipartisan economic polling team since 2013. A phenomenal space that you've been in here. And I'm curious, there's been so many changes in the world in that time. So, how have the questions been asked, the people responding, and the way results are interpreted? How has everything changed during your time in this space? [04:05] Micah Roberts: Yeah, it's been a very exciting time to do what I do for a living because, you know, I started at the very end of the Bush presidency. We got to see the Obama presidency, 2 terms of that, the rise of Trump, the craziness of 2020. Everything seems to change all the time. [04:28] Neil Hughes: Yeah. [04:28] Micah Roberts: Think about COVID and all of the public opinion data that we collected then that was just radically different than anything we'd ever seen before. Oh, and by the way, in 2008, starting my career during the Great Recession. Economic attitudes were, have kind of yo-yoed over the last 20 years. I think though, from, if I'm looking back and taking a broad view, Neal, the thing that's changed the most is that actually opinion is able to change much quicker than it ever has before. And I attribute that a great deal to the prevalence of social media and the very distinct ability of social media to move opinion because people are getting information in a very restricted silo. You know, their algorithms typically teaching or telling them, unless you tell it not to or try to force it not to, telling you what you already know and also surrounding you with a lot of voices that that believe what you believe and then inching you forward down different paths to deepen those beliefs. And so once something takes hold, I think about things like the trans movement in America in 2020, before, right before COVID in terms of a broad national conversation about that issue, it was almost nonexistent. There was maybe 5% of people who even had heard of that. And now fast forward 6 years, 6 years, and it's a national conversation that almost everybody has a hardened opinion about. That's just fascinating and very, very peculiar when you think about how opinion has moved in the past. So that's just one example. You could talk about all sorts of different movements. I mean, you know, the MAGA movement in particular didn't exist, just did not exist. And then now it's, you know, it's probably the most consequential label you can slap on somebody in American politics. And most recently, you know, the Democratic Socialists of America. This is a very fast-moving set of opinion data that's very concentrated, very concentrated, of course, among Democrats because it's Democratic Socialists. But the idea of socialism being a net good for America is a rapidly growing perception among a very significant part of our country. And so the ramifications of all of that have, gosh, you know, if you go, if you stretch that 20 or 30 years forward, if they were to persist as they are now, it just changes the face of America. And it's something that we get to watch as pollsters, unfold from, you know, basically infancy. So that's very interesting. It's very exciting work that I do. And it's also super important, I feel. So I feel like the work that pollsters do is super important to inform and to give people a sense of kind of where the country is. [07:46] Neil Hughes: Yeah, and it's a universal theme as well. And I suspect everybody listening will have encountered polling through election headlines, approval ratings, but Where else is it being used to influence public policy, corporate decisions, messaging, public affairs campaigns? Where else do you see it being impacted too? [08:06] Micah Roberts: I would say almost everything that you see, read, or hear outside of, and maybe even, you know, straight news sources that are objective, if you can find them, are influenced in one way or another by actual scientific testing of what works, what moves opinion, what grabs people's attention, what is new information that people are learning about a subject, a candidate, or a movement. So it's hard to imagine what sphere is not influenced by this sort of thing. I mean, I think that's part of what makes it important, but also part of what makes it extremely important, I guess, is the same word, very important to protect and to make sure it's, it's, uh, it's got integrity, uh, the polling that you, that you look at has integrity and is, is being conducted correctly. [09:05] Neil Hughes: And I suspect also that no matter what generation you are, every single person here will come across polls. We all get our information in different places. Some people, it'll be old-school newspapers and TV, as it will be online and social feeds. But I'd love to get a message across to every single person when they read a headline, no matter where it is, about a poll. When they see, I don't know, a figure like 52% of voters support an issue, the issue itself is not important. What should they examine before deciding whether that figure is actually credible or they're being manipulated or led down a path? [09:41] Micah Roberts: Yeah, great, great question. I think you always have to consider the source when you're looking at polling. So this is maybe some broad advice for anybody who wants to or is following polling on a regular basis. Find polling that's been generally accurate. Polling is not perfect. It's very difficult. It's very expensive. So finding polling that has been generally accurate over a long period of time and then tracking that poll, right? So if you like the Fox News poll, it's a great poll. It's very good. It's bipartisan. The guys who run it are great. They have great questions that I'm always going, dang it, I wish I asked that question in my poll. It's a fantastic poll. NBC News, the fantastic gold standard poll. But the idea is that you'd have a pollster that does the same methodology consistently over time. Okay? if you're tracking with the same methodology consistently over time and you see a movement, that's what you want to react to. Not, oh, this poll said this, but then this other poll that was conducted, you know, by this other firm said another thing. They're not doing— they're not trying to be consistent between those 2 polls. You know, Quinnipiac and Fox News are not looking at each other saying, how do we replicate What Crony Point of Piak did. So the idea is find one or two, maybe three polls that you that you trust and feel are accurate. And trust doesn't mean like you agree with it all the time. Trust means okay, that's something where you can you know you the people have a a brand or reputation behind it. The pollsters have a brand or reputation behind it. They've been generally accurate, not perfect. That's, but generally accurate and they're consistent, meaning their numbers don't jump all over the place all the time. That's, that would be something that, those are a few things that I would look for as a consumer of polling. Don't look, here's the things not to look for. You don't look for the biggest cell sizes. Oh, this, this one, you know, this poll has whatever, 15,000 interviews completed. That only matters if it's 15,000 good, you know, good, well-conducted actual people taking the poll. If it's 15,000 people that are kind of unengaged and were not even real, sometimes that can be the case, it's not worth the paper it's written on. So you have to be a pretty discerning consumer to, I think, to use polling correctly. [12:24] Neil Hughes: And I would also imagine reaching a representative sample has become much harder as Fewer people are willing to answer unknown calls, and younger audiences famously communicate completely different. I'm curious, looking back at your career here, how have polling methods changed in response? [12:42] Micah Roberts: When I first started in 2007, we did— it was like kind of a novel idea to reach people on their cell phones. So less than 20 years ago, there was almost no cell phone representation in a polling sample. I mean, even the most You know, gold standard polling was done off of a landline sample. Maybe there was 7 to 10% of cell phones. Now it's completely reversed in the span of 20 years. 7 to 10% at most are from landlines. Then you're getting the large lion's share of completes or interviews from cell phones. And, you know, like voice cell phones. And then we're also seeing and using, effectively using text-to-web, basically text messages that are inviting people to take surveys and people answer those. Or you could get them, you can send it to your email, but most likely it's, most often it's to your smartphone or text. So it's radically changed in terms of methodology and that's an important underpinning considering the data that we've gotten over time has been very consistent. So those are small changes over 20 years. So we go from 7% to 10% to blah, blah, blah, blah, blah. And now we're all the way up to 90% or so of survey responses not being collected on a landline. The other, really quickly, the other methodological change for most public polling is that we're now using voter files National voter files, and we have much more access to large datasets and are able to pull voters, actual voters from their voter file. We know a lot about them, where they live and their names and their ages and basic information, demographic information. And before, most polls were conducted just kind of like a phone book, random phone book. It's called random digit. Sorry, random digit dialing. So basically just randomly punching in landline phone numbers. So it's become much more sophisticated, but much more difficult. And all of that means much more expensive. It's one of those things that has actually grown, you know, to do it right has become much more expensive over time because of the restrictions in how to reach people. [15:10] Neil Hughes: You raise so many good points there because technology now plays a big role in finding participants, conducting research, analysing responses, identifying patterns that maybe a human researcher might miss. But on the flip side of that, we are in an online world of bots and AI-generated responses and automated online activity. Has it become harder to distinguish between human behaviour and how can pollsters confirm that they are actually hearing from a real, a real person? Is that a challenge yet? [15:41] Micah Roberts: Yeah, especially if you're conducting polling through what we call consumer panels. But typically what we'll do is we'll ask trap questions. So not only trapping, not only kicking out people, not kicking out non-people, meaning AI or bots, but also identifying people who aren't paying close attention to the survey. They're just kind of racing through it for whatever reason. Maybe there's some incentive for them to take the survey that's built into a panel, or they're just not paying attention in general. And so polling does use those types of trap questions. For instance, you'd give them 4 options and one of them would be a color. The other ones would be objects that are not colors. And you'd say, okay, which one of these is a color? It's kind of like the CAPTCHA that we have to go through in order to prove we're human. Before we, you know, buy a ticket on Ticketmaster or whatever. Everybody's experienced that sort of experience. But yeah, so there are some traps, some trap questions that we'll do as a matter of course. That's just good practice. And then of course, open-ended responses, open-ended responses where we ask people to kind of explain what they think or feel about an issue is a really, really strong way to weed out Again, bots, AI, and even, you know, just people who are just not paying close attention or can't really put together a cogent thought. I mean, you know, people have different levels of communication skills, but what we find is people answer simple questions in very, very nonsense ways. And those would be removed from the sample. And so that adds again to the difficulty of reaching people, but also the expense because you've got somebody on the phone or online who you think is a completed interview and oops, they're not up to snuff. And so you have to kick them out and replace them. [17:40] Neil Hughes: And even though I'm 4,000 or 5,000 miles away from you today, our news has a lot of coverage on the midterm elections in the US. And looking at the changes happening today and everything we're talking about, What will polling need to do differently to maintain public trust and remain useful to policymakers, businesses, journalists, and indeed voters, of course? [18:02] Micah Roberts: That's a good question. I think, you know, we've seen the erosion, and this is part of what we do. We're measuring public opinion all the time. And over, you know, the span of 20 years, but even just very recently, the erosion of public support in our institutions, Whether that's the news media, the federal government, sometimes even state government agencies, depending on where you live. Congress is woefully underwater in terms of their approval. So there's, there's a broad lack of trust in institutions and the information that we get as Americans. We're very skeptical. We can be very skeptical of the information we get. And then, you know, and that, and part of that is, again, social media. You know, if we're hearing things we don't quite agree with, we can kind of tune it out or think, well, that's just biased or skewed or whatever you want to say. So the best thing that the polling industry can do to get to your question finally is be accurate, continue to do the difficult work to get as representative sample as possible. You know, we're a very unique industry where we We have to kind of answer to the principal every 2 or 4 years. So if you're a political pollster, you know, you know that test is coming. And so we're constantly innovating, constantly looking at our data to make sure that it is accurate. And if we miss, and when we have missed in the past, there is like a deep postmortem to say what went wrong, where did we miss the Miss this vote or miss this election. And then we carry those things forward to make sure that we're not doing it as a part of our— we're changing our practices in order to reflect and adapt to a changing world. So we have to continue to do that, and we have to continue to do that. And then also we've got to really prosecute anybody who would use polling for ill purposes, for negative purposes, abuse the public trust. Through their use of polling, and we've seen recent examples of that here in the U.S. where outfits have published false data, meaning they just made it up and put it out there, and it was never a survey. It was never a survey they conducted. They just kind of created a number on a major race and reported it so that to see, they say, as a social experiment, to see if. news organizations would pick it up and run with it. And a lot of them did. So there's, I think there's many kind of cooks in the kitchen to make sure that public trust is maintained about the— and of course, the news media who report the, you know, and carry these stories and treat this data as factual also have a role to play to be responsible users and conveyor of information. [21:02] Neil Hughes: 100% with you. And finally, as a pollster, what are you going to be following most when you're looking at the midterm elections? What are you looking out for? And again, for people listening that are fed up with noise and bias and untruths and stuff, what should they be looking out for as well? Wow. [21:22] Micah Roberts: Yeah, I mean, there's so many things that I'm kind of involved with across the country, but on a broad scale, To see how much the old kind of trope about the economy, we used it as Republicans very effectively in 2024 against the Biden administration. And now it seems the Democrats are going to have an opportunity to use the same messaging that we used back against us, whether it's inflation or cost of living concerns, energy concerns. And so to see how well that works in reverse will be very interesting. I'm very, I'm fascinated by the rise, scared but fascinated by the rise of this embrace of socialism. It is a, I think it's a dangerous ideology, if that's not too frank from a pollster. Usually I try to stick to the facts, but this is my opinion, a dangerous ideology. And The idea that it's growing in such a rapid clip is frightening. So to see how well that plays out in a general election, not just a Democratic primary where you're looking for the hardest core person that agrees with you the most, but in a general election to see how many of these, you know, DSA candidates actually make it through will be I think not just interesting to watch for this election, but foreboding or foreshadowing some future change in our politics. [22:57] Neil Hughes: One thing's for sure is we are going to see a lot of big headlines over the coming weeks. And for anybody listening that would love to keep up to speed with your coverage, your work, and what you are doing, or find out more information about you and connect with you and even work with you, Where would you like me to point everyone listening? [23:15] Micah Roberts: They can go to our website, pos.org, or you can find us on X. I am @MicahCRob, that's M-I-C-A-H-C-R-O-B, or our firm, which is @POStrategies. And we post a lot of really fascinating data on a regular basis on our X account. But yeah, those are the most main places they can find us. And always, Tune in and check out the NBC News and CNBC All-America Economic Poll, which are really important parts of the work that we do for public consumption. [23:50] Neil Hughes: Well, as we said at the very beginning, public opinion polling is something that impacts everyone all around the world in different ways. And understanding how it works, it's just been fascinating for me. I've certainly learned a lot. I'll be adding links to everything you mentioned there to the show notes. I encourage people to check you out and continue this conversation. But more than anything, thank you for starting it today. Really appreciate your time. [24:14] Micah Roberts: Thank you, Neil. This has been fun.