We value: Our readers.
Our independence. Our region.
The power of real journalism.
We're reader supported.
Get our newsletter free.
Help pay for our reporting.

Can You Really Poll Online?

Bad news numbers have Liberals wondering.

By Richard Warnica 14 Mar 2007 | TheTyee.ca

Richard Warnica is a senior editor at The Tyee.

image atom
Angus Reid: online poll pusher.

Everyone knows the maxim about blaming the messenger, but in this story, it's the horse that's taking the heat.

Two weeks ago, Angus Reid Strategies released a poll that showed the federal Liberals tanking. The poll, put out March 3 and conducted over seven days in late February, had the governing Conservatives 16 per cent ahead of their rivals and, according to the accompanying press release, "on the brink of majority."

The results were surprising, but not much. The Liberals have bled support since Christmas. A series of Conservative ads attacking leader Stéphane Dion only aggravated their wounds. So if the size of the Tory lead was unexpected, it was miles far from implausible. Still, that didn't stop some prominent Liberals from laying into the numbers.

Unlike traditional opinion polls, the Reid one wasn't conducted on the phone. No random calls at dinnertime, no random Canadians answering long strings of questions about their political preferences. Instead, the survey was done entirely online. And that's where, for those aforementioned Liberals, the problems start.

Blogger Jason Cherniak, maybe the best-known non-professional Grit on the web, was first out of the gate. "I was already pretty sure that online surveys can only be extrapolated to the online community," he wrote. "Now I am convinced."

Warren Kinsella, one of the architects of the Liberal's 2000 election victory, wasn't far behind. "Among those online (and as the blogosphere demonstrates, daily), important demographic groups are significantly under-represented: older people, rural residents, those with lower incomes, and women," he wrote. "And, despite what some of even the respected pollsters claim, pollsters aren't being terribly scientific in their recruitment efforts."

Now, there's nothing wrong with attacking a bad poll. We've done our fair share of that here at The Tyee. But one suspects the people at Angus Reid might not entirely agree with the Liblogger perspective on their scientific methods. So, aiming to do some unrepresentative sampling of my own, I phoned up Reid (the company, not the man) and spoke to President and Chief Operating Officer Edward Morawksi.

Morawski was pretty dismissive of the bloggers' concerns. No poll, or polling method is perfect, he told me, but there's no evidence that a well selected online sample is any less perfect that a telephone one. In fact, he argued, in some countries the online versus phone debate was settled years ago; the Internet won hands down. "In the U.K. today, the most respected pollster is YouGov," an entirely online operation Morawski said. And in the U.S., it's Harris, also all web based.

Polling, Morawski says, is all about the sample. To accurately predict how a larger population feels about something -- a brand, a politician, even Conrad Black -- you need to survey a balanced chunk of the whole. (The opinions of 3,000 white males from Regina won't tell you much about how the entire country feels about anything.) Angus Reid selects their sample randomly from a larger group signed up on the Reid website. The results are then calibrated to take into account things like age, gender and voting preference.

The main criticism that Cherniak and Kinsella had was that by picking their sample online, AR Strategies excluded all those Canadians who aren't, thus preventing a truly representative slice. Morawski acknowledged that was a problem. But, he countered, there are ways to work around it. And what's more, the sample problems for telephone pollsters are just as bad and maybe worse.

The big issue for telephone polling is refusal rates, something Reid himself called "the big dirty secret" of the polling industry in a 2004 Tyee article. Put simply, every year more people refuse to take part in surveys. Many more simply don't pick up the phone for unknown callers, and some don't have home phones at all. In another Tyee article from 2005, Tom Barrett reported that for a typical B.C. election poll, a pollster must dial 6,000 to 10,000 numbers to get a representative sample. That, according to Morawski and Reid, means building a representative sample by phone gets tougher every year.

The Angus Reid online polls try to work around that by building in incentives for people to participate, Morawski told me. Each selected member of the sample is paid between $1 and $4 for each survey they fill out. Morawski said they are also conscious about keeping the surveys short and, as much as possible, enjoyable.

At the end of our interview, Morawski came back to the idea that no method for polling is perfect. "We get it, we know it, there are concerns," he said. "Is the Internet perfect, no.... It is just a new method and a new technology."

Related Tyee stories:


Share this article

The Tyee is supported by readers like you

Join us and grow independent media in Canada

Facts matter. Get The Tyee's in-depth journalism delivered to your inbox for free.


The Barometer

Tyee Poll: Are You Concerned About Ticks?

Take this week's poll