The Big Story: Why Polling Chaos Could Create Big Problems for American Politics
“Polls are traditionally used to measure public opinion, not mold it.”
When the Tail Is Wagging the Dog
By Jed Lam and David Shifrin
3-minute read
Every few years it happens. Someone – whether an insider proving a point, an irritating agent of chaos or a lazy/opportunistic professional – will poke at a system like computer science or the peer review process in biomedical research to find out where the weak points are, either to warn of the danger or to exploit it.
This year’s version is apparently political polling: “LOL just kidding! It was all a big social experiment to see if we could manipulate political levers and guess what it worked!” That was more or less the proclamation from the group responsible for a trio of fake but attention-getting polls (right before they disappeared).
Though sometimes valuable to highlight weaknesses and help those working in that field fix problems, the cost of these incursions is the continued erosion of public trust. Trust in both the results or metrics themselves and in the media or leaders charged with policing them. Case in point is the campaign of one candidate for LA mayor, which as PBS reported, had touted one of the fake findings on its social channels: “The Bass campaign has since deleted its social media post promoting the poll, but defended itself by saying that it ‘was reported on by multiple news outlets.’”
The recent spate of bad and/or fake polls has again raised the skepticism that has long surrounded the somewhat opaque process by which real people are contacted by phone, web or text and asked their opinions about any number of issues and who they’re going to vote for. Problem is, of course, the complexity of reaching enough real people and extrapolating their opinions to the general public makes it easier for bad actors – or even just biased ones – to create the outcomes they want.
Traditionally, polling would be used to measure voting preferences, figure out why people felt that way and then adjust campaign strategy based on those findings. Sure, touting a favorable poll has always been part of the political toolkit. But that’s subtly (and sometimes not so subtly) different than using a favorable poll as a weapon to push people. Similarly, getting results that don’t go your way should lead to asking why. Not discarding the data and shopping around for numbers that look rosier.
We’re seeing why both of those extreme approaches are a problem today. The desire to see what we want to see is strong. Without a careful process in place to work through data – and, yes, to be willing to accept negative results – we run the risk of cherry-picking to the point of creating or commissioning or accepting something fake that reinforces our position.
It’s a human phenomenon, not a partisan one. As The New York Times notes, both the red and blue sides of the aisle have “pointed to dubious polling to advance their political goals.”
Check bias at the door
It’s also a very frustrating phenomenon for those who are doing good work in the research field, whether high-stakes political polling or quantitative market research. As one researcher commenting on the polling controversy said, “This is an industry built on trust. I have to trust that people are answering my questions in good faith.”
The healthcare industry is also deeply dependent on trust. The public trusts healthcare leaders to make decisions in the best interest of their community, and those leaders have to trust the information they’re basing decisions on. Data on things like where to invest, workplace pulse checks, patient experience and brand reputation. If data is bad or biased, the consequences are real.
But how to find sources that you can trust, and what’s the right way to use it? Here are a few thoughts for vetting data like polls and survey results that may cross your desk – especially if they land there to be part of your organization’s strategic decision-making. All of these come down to two things: Trust, but verify. And beware your biases.
- Do your homework. Not every poll or survey is reliable. Check the source. Look for a track record. Ask questions of the team running the work. Ask around for references and case studies.
- Look at the methodology. How much is disclosed? Sample sizes? Margin of error? Data security and verification details? If you can’t find that information anywhere or the research team dodges the question, it’s likely because it doesn’t exist. Or at least isn’t a priority.
- Ask about AI. First, any use of AI at any stage of the research process should be disclosed in the methodology. Second, fake data and synthetic data are two different things. AI is being tested and used to replicate opinions and responses from actual people to rapidly and cheaply gain insights on a topic. Like so many uses of AI, synthetic data holds significant promise, yet it must be viewed with caution. Most importantly, it should only be used to supplement real-world data, not to replace it. Because, well, it can’t replace humans.
- Don’t just post it, pressure test it. You know the drill: If something seems too good to be true… The point of research is to inform. Obviously. Sometimes that means revealing something new and unexpected. Other times, it might confirm something we know or suspect. Many findings aren’t earth-shattering but are still valuable. Gut instinct is a powerful tool, and you have both permission and encouragement to ask yourself whether a finding makes instinctive sense. If it doesn’t, don’t discard it right away. Use it as a cue to dig deeper.
- Resist the temptation to use polling as peer pressure. Use data to inform, not instill fear – don’t weaponize it. We won’t begrudge you the occasional “See, I told you!” but that’s different. In some of the political examples above, data was used to manipulate donors. What is your version of that? Again, use research to inform strategic decision-making, not engender fear.
Public opinion research is hard. Margins of error exist for a reason. That’s not an excuse for sloppy work from a pollster or researcher, but a reminder that numbers should inform decisions, not make them for us.
Whether you’re analyzing election polls or patient sentiment, data can point you in the right direction. It can’t replace judgment. Because numbers are a tool, not a god. The best leaders neither worship data nor ignore it. They interrogate it. Challenge the findings. Test assumptions. Refine thinking. Then use it to make better decisions.
Contributors: David Jarrard, Emme Nelson Baxter, Tim Stewart
Image Credit: Shannon Threadgill
