Summary of We try to fix election polling (update)

Overview of We Try to Fix Election Polling (Update)

This Planet Money episode revisits NPR’s earlier experiment to improve election polling and explains why polling has become so difficult in the modern era. The story follows Marist College pollsters as they train callers, struggle to get people to answer, and test new “wisdom of crowds” questions meant to supplement traditional horse-race polling. The update shows that while the original fix was not a magic bullet, some of the ideas are still being used and remain relevant as pollsters try to adapt to lower response rates, changing voter behavior, and a less predictable electorate.

Why Election Polling Is in Crisis

• Polling has gotten harder because far fewer people answer phone surveys.
• Response rates for telephone polls have collapsed to below 1 in 20 in many cases.
• Pollsters especially struggle to reach:
  • younger voters
  • rural voters
  • people who screen unknown numbers
• If the people who do respond are systematically different from those who do not, polls can become less representative and therefore less accurate.

The History of Polling Blunders

The episode uses three major polling failures to explain how the field evolved:

1. 1936: The Literary Digest disaster

• The magazine polled millions of people and predicted Alf Landon would beat FDR.
• It was wrong by a lot.
• The problem: its sample was too wealthy and not representative.
• Lesson: more responses do not help if the sample is biased.

2. 1948: “Dewey Defeats Truman”

• Pollsters predicted Truman would lose.
• They tried to build representative samples by hand, but the process introduced error.
• Lesson: representative samples are hard to build manually.

3. 2016 onward: modern polling misses

• Polls increasingly underestimated Republicans and missed key election outcomes.
• The golden age of phone polling faded as:
  • landlines disappeared
  • caller ID and spam concerns increased
  • cell phones became harder to dial legally at scale
• Lesson: the old methods no longer work as well in a mobile, low-response world.

How Pollsters Used to Work

• During the “golden age” of polling, random telephone sampling worked well.
• Pollsters could call random numbers and get enough responses to produce accurate estimates.
• Marist director Barbara Carvalho describes this as a period when polls could sometimes match election results “to the decimal point.”
• But that era is largely over because the random representative sample is now much harder to achieve.

What the Planet Money Team Tried

During the original 2022 story, Jeff and Nick visited Marist College and tried doing live polling themselves.

Lessons from pollster training

• Use calming, conversational phrases like:
  • “Let’s try a few and see how it goes.”
  • “Every opinion matters.”
  • “There are no right or wrong answers.”
• The most important advice: smile while you dial.
  • Even through the phone, a friendly tone makes people more likely to engage.

The reality of calling voters

• Most calls ended in:
  • hang-ups
  • voicemails
  • answering machines
  • refusals
  • silence
• After many failures, they discovered how emotionally draining and inefficient telephone polling can be.
• One successful interview came only after a long string of rejections.

Ways Pollsters Are Trying to Fix the Problem

The episode highlights three main strategies:

1. Move beyond the phone

• Some pollsters now use text-based surveys instead of calls.
• The goal is still randomness, but without depending on people answering their phones.

2. Weight the results

• If certain groups are underrepresented, pollsters adjust the data statistically.
• Examples of weighting:
  • race
  • education
  • geography
  • psychographic factors
• This helps, but it is still an attempt to repair a sample that may already be flawed.

3. Ask better predictive questions

Researchers testing “wisdom of crowds” approaches found that these questions may predict outcomes better than asking directly who someone will vote for:

• Who do you think will win?
• Who do the people around you think will win?

Why these work better:

• The first question taps into the respondent’s overall sense of the race.
• The second pulls in information from a person’s social network, effectively broadening the sample.

The Update: Did Their Fix Work?

• Marist’s 2022 polling performed very well in predicting the national House vote.
  • They estimated Republicans would lead Democrats by about 3 points.
  • The final result was 2.8 points.
• The Planet Money team’s experimental questions were not as successful in that particular test.
• Still, Barbara Carvalho said the point of science is to test ideas and learn from the results.
• The “wisdom of crowds” questions are still being used in some form and continue to be studied.

What the New Questions May Help Predict

The update notes that pollsters are now exploring whether these questions can also help predict:

• who will actually vote
• turnout probability
• whether voters are influenced by:
  • a bandwagon effect
  • an underdog effect

Barbara also said that, in current polling, some of the most useful predictors include:

• opinion of President Trump
• party identification
• strength of party identification

Main Takeaways

• Polling is not “broken” in a simple sense, but it is under major stress.
• The core challenge remains the same: getting a truly representative random sample.
• Traditional phone polling is much less reliable because people no longer answer the phone.
• Pollsters are responding with:
  • new modes like texting
  • better weighting
  • experimental “wisdom of crowds” questions
• Public polls still matter because they can:
  • reduce spin
  • inform voters
  • influence policy debates
  • reveal what the public is actually thinking

Notable Insight

• Barbara Carvalho’s warning from 2014 proved prescient: even a poll that looks perfect one cycle may not stay that way.
• Her broader point: polling is always changing, and pollsters must keep adapting if they want to remain useful.