News

Can We Still Trust Polls?

Despite giant miss in 2013, BC pollsters argue results can be believed.

By Brent Holmes 24 Apr 2017 | TheTyee.ca

Brent Holmes is completing a practicum at The Tyee.

image atom
‘You don’t change everything you’ve done historically because of one negative outcome,’ says pollster Kyle Braid. Photo: Twitter.

It’s been four years since the 2013 B.C. provincial election and pollsters still haven’t decided why the polls were so wrong.

The NDP, according the polls, had the election won after 12 years of BC Liberal government. Eleven polls conducted in the final two weeks of the campaign all had the NDP ahead by an average of almost seven percentage points.

Then the votes were counted. The BC Liberals took 44.1 per cent of the popular vote to the NDP’s 39.7 per cent and won a majority.

Everyone seemed to be wrong about the 2013 election results — except the BC Liberals, who claimed their internal polling had showed them in the lead.

Pollsters are divided on the reasons their numbers were so wrong.

“It was more related to people changing their minds,” Insights West vice president of public opinion Mario Canseco said. “I know it’s anecdotal, but along with a steady barrage of messages asking me to do something else with my life there was one from a lady in Kerrisdale who said, ‘I took your survey, I am part of the online panel that you used for this. And I told you I was going to vote for the NDP, but when I got to the polling station I changed my mind.’”

Ipsos pollster Kyle Braid disagrees. He said one problem was that NDP supporters believed the outcome was inevitable and didn’t vote.

“When everybody believes that a certain outcome is going to happen, I think that does influence who does turn out to vote... NDP voters just didn’t turn up to vote on election day because they thought the election was in the bag.”

These aren’t the only theories as to why the polls were wrong in 2013.

And there is no definitive answer. “We don’t really know which one it was,” Braid admitted.

Who you gonna call: Landline versus online polling

A major challenge for pollsters is simply reaching a representative sample as new technologies disrupt their traditional methods.

UBC professor Richard Johnston, who recently published an article analyzing polling in B.C.’s 2013 election, said polling faces major challenges if companies can’t get large enough representative samples of respondents.

Pollsters used to rely on calling people on their home phones. But since 2006, the percentage of households with landlines has dropped from 94 per cent to 76 per cent.

“The percentage of eligible voters who can be reached through a landline has diminished,” Johnston said. “The other thing is that people have been less willing to agree to be interviewed and so that kind of conventional technology, which is as close as we’ve ever come to a truly random selection, that’s really challenged.”

Most polls now are online, Johnston said. The response from online panels can be influenced by a range of factors, including using the same group too frequently or not often enough. These polls need a massive sample size in order to be reliable, he said.

At Insights West, an online panel has the same scientific nature as a phone panel with a group of volunteers that is robust enough for the pollsters to gather information. Volunteers will have a profile so they don’t have to repeatedly fill out demographic information when they respond to email questionnaires, speeding the process up. The poll is completed once enough respondents have replied and appropriate weights are applied.

Canseco believes online polling actually makes polling easier. People said he was crazy when he started using online polling, he recalled. In the 2008 federal election, the company he was with at the time was the only one to do online panels. By 2015, there were only three companies doing exclusively over-the-phone polling.

Ipsos and Insights West focus on a mix of online and telephone polling.

The real challenge for pollsters is weighting the response to reflect the population.

Pollsters don’t contact a number of people and report the results, Johnston noted. They adjust the findings to compensate for ways in which the respondents differ from the general population. If a polling sample is equally male and female, but women represent more of the population, pollsters will adjust their results to reflect the difference.

Johnston explained that a smaller sample size can mean weighting becomes more significant as the likelihood that the sample reflects the population declines.

Both Johnston and pollsters are skeptical of the accuracy of polls that rely on heavy weighting.

Braid has previously said that he incorrectly weighted the polls during the 2013 election. Toward the end of the campaign, Ipso’s polls had the Liberals ahead among voters 50 and older, he told a conference.

“But we didn’t make any adjustments for that,” Braid said after the election. “We know that older folks make up about a third of the electorate, but they account for about 50 per cent of the voters.” Ipsos didn't give enough weight to these older voters.

“The industry continues to work on techniques to model turnout and predict who’s going to turn out... but the experience in trying to predict that turnout has not been overwhelming so far,” Braid said.

Faith in polling took a big hit in 2013. It fell even further after the failure to predict the results in the Brexit vote and the 2016 U.S. presidential election.

Canseco, however, disagrees with the assessment that these perceived failures show polling is unreliable. The company doing the polling is more responsible for the result than the industry.

“I keep hearing ‘Brexit! Brexit!” Everybody missed Brexit,’” Canseco said. “There’s three companies in the U.K. that got Brexit right. There’s five that got it wrong. It’s this bandwagon effect of ‘well polling is dead because Brexit wasn’t called by five out of eight pollsters.’ I don’t think we can definitively say that’s the way to go.”

In the case of Trump’s election, Canseco points out that the polls weren’t entirely wrong. The numbers looked wrong on election night, but when the final vote tally came in it showed many pollsters predicted the popular vote accurately, he said, but those results don’t matter to the public when the outcome has been determined two weeks earlier.

Pollsters are focusing on the issues in 2017 rather than just tracking party support, according to Canseco. The focus will be on how the public is feeling about key election issues and which candidates they believe best offer solutions to those problems.

Ipsos and Insights West are doing only two polls on party preference, at the start and end of the campaign.

It’s important to conduct the final poll as close to the election as possible to reflect changes in voter attitudes in the final days of the campaign.

Canseco said during the 2015 federal election, which was on a Monday, Insights West decided not to publish polling results from the Friday before election day because the numbers would not reflect how the electorate might change over the weekend.

When the vote was counted, the results were proved right. The poll showed 35 per cent support for the Liberals in B.C. and that’s what they got.

“Do I go out there with numbers that are three days old and get all the media coverage or do I wait until the final moment of the election campaign, publish the findings and then the next day remind people that you got it right?” Canseco asked. It’s better to err on the side of caution than release outdated information and risk a wrong prediction, he said.

Canseco said Insights West has correctly predicted 22 election outcomes including the 2015 federal election and several races in the United States.

And, Braid notes, one failure — like the 2013 provincial election — doesn’t justify tossing out established polling methodologies.

“You don’t change everything you’ve done historically because of one negative outcome,” Braid said.

Johnston remains skeptical.

“That’s a defensive reaction by the industry,” Johnston said. “They’re actually right. What we notice is the big errors, but that’s only natural. Polling miscalls have been frequent enough and they’re not getting rarer... it is not wrong for voters to feel insecure about the information they’re getting. A number of polling companies are anxious themselves that they’re less and less inclined to make predictions.”

Insights West and Ipsos will not be counting themselves among those companies.

“Wayne Gretzky said you miss 100 per cent of the shots you don’t take; well, you miss 100 per cent of the elections you don’t call,” Canseco said.  [Tyee]

Read more: BC Election 2017, Media

Share this article

The Tyee is supported by readers like you

Join us and grow independent media in Canada

Get The Tyee in your inbox

LATEST STORIES

The Barometer

What are you doing about reconciliation?

Take this week's poll