The 'Dirty Secrets' of Pollsters

The media polls that are driving this election may contain hidden flaws, veteran pollster Angus Reid warns.

Reid says that a combination of factors - including reduced media budgets for polling and the increasing reluctance of Canadians to talk to pollsters - could be harming the reliability of election polls.

He's particularly worried about the growing tendency of many Canadians to screen their phone calls and refuse to talk to pollsters - which can mean that a pollster has to make 10,000 phone calls to complete 1,000 interviews.

Such high refusal rates not only increase the cost of polling, but they may mean that pollsters end up interviewing people who are not representative of the general public.

"It's the big dirty secret of the industry," Reid said.

Cheaper to publish others' polls

Reid's concerns come in the midst of the most poll-driven election in recent history, as the apparent Liberal collapse and Conservative surge have set the agenda for media coverage across the country.

Reid, who founded the company now known as Ipsos-Reid, said the media "really don't have any money for polling."

Twenty years ago, the Southam chain of newspapers would spend $250,000 on election polling, said Reid, who has been polling for three decades.

"I'll bet that CanWest-slash-National Post is not spending $50,000 in this campaign."

The result is that "no one's paying for really good high-quality polling, yet there's a lot of polls being conducted." That creates "a tension there that I think is bound to come up and bite someone where it hurts."

What they don't say about 'refusal rates'

Media polls should state their refusal rates, said Reid, who is now works as an independent consultant but is not polling during this election.

Refusal rates have been an issue of concern in the polling community for several years. The Professional Marketing Research Society has found that refusal rates - defined as the number of people who refuse to talk to a pollster as a proportion of all people spoken to by the pollster - have been climbing steadily for the last decade.

The refusal rate for all telephone polls in Canada now stands at about 80 per cent, the society recently reported. That means that for every interview a pollster conducts, four respondents have refused to participate.

In addition, thanks in large part to answering machines and call blockers, almost half of all numbers dialled by pollsters never reach a live human being.

Pollsters worry that this could mean that the people who do agree to be interviewed are different from the people the pollsters aren't talking to. The margin-of-error statements in news reports of polls are based on an assumption that the people interviewed make up a random sample; that is, everyone has an equal chance of being interviewed. That may not be as true today as it once was.

"Maybe Conservatives, being perhaps older or more cantankerous or whatever, are more likely to refuse than Liberals," said Reid. "In which case, the industry is going to be somewhat confused on election day."

Good polling key to good journalism

Reid said it is unfortunate that the CBC is not polling during this election. The network has stated that, as a matter of principle, it will not conduct "horse race" polls during this campaign.

"The CBC used to spend more on their polling program than any of the private operators," Reid said.

"For them to assume that good journalism doesn't involve polling is, I think, a little naive.. I think responsible polling is a very, very, very central part of an election campaign and responsible media should be paying some money to get it done right."

Reid was also critical of how the media report poll results. Reporters often highlight small shifts in poll results, he said - changes that could be due to nothing more than the fact that numbers will always tend to bounce up and down from poll to poll within the margin of error.

"You've got the Globe and Mail looking at a drop of four points or three points and saying this is a major drop," he said. "For all we know, within the normal margin of error, there was no change..

"I think the consumers of polls should be skeptical about that."

New techniques aim for better results

Steve Mossop, Ipsos-Reid's managing director for Western Canada, conceded that refusal rates are rising at the same time that pollsters are under pressure to do more with less.

But he denied that this hurts the quality of the media's election polls.

"It's not as dramatic as maybe Angus has proposed," Mossop said.

Ipsos-Reid, which is polling for the Globe and CTV during this election, is examining new techniques to offset the effect of rising refusal rates, he said.

As for media outlets cutting back on what they spend on polling, Mossop said, "Yes, we're getting a squeeze to produce more. It's not a challenge that's going to have a negative impact on the predictability of the election results."

Mossop added that Ipsos Reid and its predecessor, the Angus Reid Group, have successfully predicted every election outcome for the past 25 years. Current sampling methods are more sophisticated and accurate than in the days when Reid headed the company, he said.

Conrad Winn, whose Compas Inc. conducts polls for CanWest-Global, said there are techniques that pollsters can use to reduce refusal rates.

"Response rates are always something to be concerned about and clients get what they pay for," Winn said. "You can greatly increase response rates, as we know, by a combination of technical skill and training of interviewers."

Is this a telemarketing call?

Part of the problem is that people confuse pollsters with telemarketers, he said.

"It's also driven partly by cost-sensitive clients who make it difficult for the research suppliers to do what's necessary to boost response rates."

"It could well be" that media outlets aren't spending as much on polls as they once did, he said.

"On the other hand, what is the reality? People are not reading newspapers the way they used to."

Still, readers and viewers shouldn't be concerned with the quality of the polling they're given, he said.

"The proof is in the pudding.. The media don't do any worse job than they used to. They probably do a better job than they used to."

John Willis, senior consultant with Strategic Communications, which is polling for the NDP during this election, said he is not too concerned about the dangers posed by high refusal rates. Pollsters employ sophisticated statistical techniques to counteract the negative effects of high refusals, he said.

Willis did agree with Reid that the media often make too much of small shifts between polls.

"As an insider to a campaign, I wish there wasn't quite so much of that going on out in the media because it heightens our sense of 'Oh my God, the Conservatives are up two points.' It has a feedback for the political parties as well as a feedback for the public that I don't think is necessarily all that helpful. It tends to obscure deeper issues."

Smaller samples can mean 20-point swings

Willis and Vancouver pollster Evi Mustel agreed that readers and viewers can get the wrong impression from news accounts of regional support for parties.

A standard national poll will usually be based on about 1,000 interviews from coast to coast, which means that between 100 and 150 interviews will be conducted in a province the size of B.C. What the media often don't explain, however, is that the results for B.C. will therefore be much less reliable than the national results.

Smaller samples mean larger margins of error and a sample of 100 carries a margin of error of plus or minus 10 percentage points, 19 times out of twenty. That means that if the "true" support for the Conservatives in B.C. is, say, 45 per cent, you can expect a well-conducted poll based on 100 interviews to give you a result for the Conservatives anywhere between 35 per cent and 55 per cent. Not quite as precise as the national results.

Mustel says that all this means that if a national poll says the spread between two parties in B.C. is as large as 10 percentage points, it still doesn't mean anything. (Some local media outlets will pay extra to get a larger regional sample as part of a national poll. If the poll you're reading or hearing about has done that, it should say so.)

But should the media be reporting "horse race" poll results at all?

"If you look at campaign coverage from the '50s," Reid said, "before the advent of the kind of polling we see now, election campaigns were all about parties trying to trick the journalists into who was on first and who had more support."

Or consider these words from American journalism professor Philip Meyer, the former president of the American Association for Public Opinion Research:

"The most interesting fact about an election is who wins. The most interesting thing about a campaign at any given moment is who is ahead.. If

you can find out, ahead of time, who is going to win, that is news by definition."

Tom Barrett is a veteran political reporter and a contributing editor to The Tyee.