This year, we’re asking a series of other questions that are easy and don’t seem like you’re going to get judged harshly for answering them. Our first goal is to minimize the social desirability effect. And you do that by giving them a great sense of anonymity. The more anonymous they think they are when giving answers, the more honest they tend to be.
It’s kind of like the people who have two Twitter accounts — the one where they tweet out pictures of their pets and children, and one where they just go give everybody a fit. Well, that “troll” account is their real emotion. And the persona that runs that troll account is the one in the ballot booth. That’s who I’m trying to get to.
The results in 2016 really hurt people’s willingness to trust polls. You’re seeing it now: Democrats say, “Biden is leading, but the polls showed Clinton winning in 2016, and she lost.” Among Republicans, it’s sort of the opposite: “The polls in 2016 didn’t reflect Trump’s strength, but he won and will win again.” So how should people look at the polls over the final days of this campaign?
Cahaly: One, they should ask themselves these questions: Do you know someone who is going to vote for Trump — someone who maybe confided that fact in a few people, but didn’t share it widely? Do you think that person, if called on the phone by a stranger — a live person who knows who they are — would tell them? If the answer is yes, then you should be skeptical of polls that are given with a live person.
And ask yourself, would you answer a survey that took 20 or 25 minutes on a Tuesday night when you’re feeding your family? If the answer is no, then you should look with skepticism at polls with long questionnaires.
Kapteyn: I think it’s good to add some nuance to the idea that polls didn’t do so well in 2016, because after all, if you look at the national polls, they actually weren’t very far off when it came to the popular vote.
Another thing that may be a little underappreciated: One of the things that was quite clear just from looking at the data is that there were events late in the election season in 2016 that had an effect — for example, [FBI Director James] Comey’s announcement that he was reopening the investigation of Hillary Clinton’s emails. That moved the needle by, like, two percentage points or so. I could see that in the data. And that’s a big number, given how tight the election was. So I think there were some reasons why the polls seemed worse than they perhaps were — and why they couldn’t be more accurate, because some major events happened very late.
Cahaly: I’m a little different on that one, because we saw the Trump numbers the whole time. Nothing was new about them to us.
Did the gap between polling in 2016 and the results affect the way either of you think about polling?
Cahaly: I became fascinated with why there was denial that social desirability bias was in play and important. It made me realize just how critical the assurance of anonymity is to getting an honest answer.
Other people started using our “neighbor” question, as Arie pointed out. And that caused us to think of some new questions we thought would be more revealing. And this time, we decided we weren’t going to share them with the world.
Kapteyn: In that sense, we are at opposite ends of the spectrum. We [USC’s Dornsife Center for Economic and Social Research] are not a polling firm; we’re a research firm. We happen to have this Internet panel where we ask people all sorts of questions, so why not also ask them about politics? For us, this is largely an experiment. That’s why we ask about this in different way: We want to see what works best.