Pollster Kyle Braid thinks he's figured out how he missed so badly in the May 14 election.
Like almost every other pollster, Braid, a vice-president at Ipsos Reid, came up with results on the eve of the election that suggested a healthy victory for the New Democratic Party. When that turned into a healthy victory for the BC Liberals, Braid and the other pollsters were left with egg dripping from their faces.
The last Ipsos poll, taken May 13, suggested the NDP had the support of 45 per cent of decided voters, compared to 37 per cent for the BC Liberals. The final results were NDP 40 per cent, Liberals 44.
Braid told an audience at the Spur Festival in Vancouver last week that he thinks he adjusted his data when he shouldn't have and failed to adjust it when he should have.
Every polling company adjusts, or weights, its samples. That's because samples rarely look like the general population. They may, to take a simple example, contain more women than the electorate as a whole.
We know that women vote differently than men, so, to make such a sample more representative, pollsters will weight the data -- giving a higher value to men in this example.
Such weighting often contains a large dollop of intuition along with the statistical science. Ideally, proper weighting will make a sample match the demographics of the people who actually turn out to vote -- which are not the same as the demographics of the general population.
We know that a person's age, income and education all predict whether they will vote. Older people are a lot more likely to vote than young folks. (See below.)
At the end of the campaign, Ipsos's polls had the Liberals ahead among voters 50 and older, "but we didn't make any adjustments for that," Braid said. "We know that older folks make up about a third of the electorate, but they account for about 50 per cent of the voters."
Ipsos didn't give enough weight to these older voters.
Said Braid: "If we'd just done something as simple as reweighted our results based on age, the Liberals would have gone up immediately two or three points."
The past vote trap
An even bigger problem came when Ipsos weighted its data based on who respondents said they voted for in 2009.
Ipsos's samples tended to be heavy on people who said they voted for the Liberals last time, Braid said. To account for this, the company gave less weight to their opinions.
Not a good idea, as it turned out.
Asking people who they voted for comes with some dangers, Braid said. You're relying on their memories: they might have forgotten how they voted, they might not want to admit they didn't vote, they might not want to admit they voted for the losing party or they might not want to admit they voted for a currently unpopular party.
You don't know how many of your respondents fit which of those categories, and that makes weighting by past vote a dangerous practice. Braid said he did it because when Ipsos switched from telephone polling to online polling a few years ago, the online results didn't "match up" with phone results.
But when he weighted by past vote, they did match up, he said.
"I was comfortable with that," Braid said. "I did that with the HST referendum in 2011. Hit the number almost bang on the head and said this is the way to do it."
Part of that false sense of comfort during the May 14 campaign came from the numbers released by other pollsters. "Our results looked like everybody else's," he said. "Nobody... was telling a different story."
One thing Braid forgot, he said, is that further you get from an election, the less reliable past-vote declarations are.
Weighting by past vote "tended to lift the NDP up three [percentage points] and the Liberals down two or three points" in the Ipsos polls, Braid said.
On election day, Ipsos polled people who had just voted. The results line up almost perfectly with the election outcome, Braid said -- but only if you don't weight for respondents' memories of their 2009 vote.
Eleventh hour Liberal shift?
Some pollsters and commentators have speculated that the giant May 14 poll fail was caused at least in part by a huge last-minute shift to the Liberals that took place after the pollsters stopped polling. Braid said exit polls taken on election day suggest that about a quarter of B.C voters made up their minds in the last week of the campaign.
But, it appears, they didn't go en masse to the Liberals.
"They voted a little bit more for the Liberals," Braid said. "Maybe it accounts for one point of the seven points of our miss."
While Braid thinks he's unravelled the mystery of his own election-night failure, other pollsters have told him they don't see the same issues behind their own results.
BC Liberal pollster Dimitri Pantazopoulos, whose internal telephone polls called a Liberal majority, told the Spur Festival audience that he is extremely wary of weighting by anything that relies on a respondent's memory. Weighting should be restricted to "empirical data" such a person's age or gender, he said.
Pantazopoulos recalled polling for Stockwell Day and the Canadian Alliance during a turbulent period in 2002. Day's support at the time was in the single digits. When Pantazopoulos asked respondents who they voted for in the last federal election, those results were in the single digits, too -- even though Day had taken 26 per cent of the vote in the 2000 election.