Marking 20 years
of bold journalism,
reader supported.
Mediacheck

Can You Really Poll Online?

Bad news numbers have Liberals wondering.

Richard Warnica 14 Mar 2007TheTyee.ca

Richard Warnica is a senior editor at The Tyee.

image atom
Angus Reid: online poll pusher.

Everyone knows the maxim about blaming the messenger, but in this story, it's the horse that's taking the heat.

Two weeks ago, Angus Reid Strategies released a poll that showed the federal Liberals tanking. The poll, put out March 3 and conducted over seven days in late February, had the governing Conservatives 16 per cent ahead of their rivals and, according to the accompanying press release, "on the brink of majority."

The results were surprising, but not much. The Liberals have bled support since Christmas. A series of Conservative ads attacking leader Stéphane Dion only aggravated their wounds. So if the size of the Tory lead was unexpected, it was miles far from implausible. Still, that didn't stop some prominent Liberals from laying into the numbers.

Unlike traditional opinion polls, the Reid one wasn't conducted on the phone. No random calls at dinnertime, no random Canadians answering long strings of questions about their political preferences. Instead, the survey was done entirely online. And that's where, for those aforementioned Liberals, the problems start.

Blogger Jason Cherniak, maybe the best-known non-professional Grit on the web, was first out of the gate. "I was already pretty sure that online surveys can only be extrapolated to the online community," he wrote. "Now I am convinced."

Warren Kinsella, one of the architects of the Liberal's 2000 election victory, wasn't far behind. "Among those online (and as the blogosphere demonstrates, daily), important demographic groups are significantly under-represented: older people, rural residents, those with lower incomes, and women," he wrote. "And, despite what some of even the respected pollsters claim, pollsters aren't being terribly scientific in their recruitment efforts."

Now, there's nothing wrong with attacking a bad poll. We've done our fair share of that here at The Tyee. But one suspects the people at Angus Reid might not entirely agree with the Liblogger perspective on their scientific methods. So, aiming to do some unrepresentative sampling of my own, I phoned up Reid (the company, not the man) and spoke to President and Chief Operating Officer Edward Morawksi.

Morawski was pretty dismissive of the bloggers' concerns. No poll, or polling method is perfect, he told me, but there's no evidence that a well selected online sample is any less perfect that a telephone one. In fact, he argued, in some countries the online versus phone debate was settled years ago; the Internet won hands down. "In the U.K. today, the most respected pollster is YouGov," an entirely online operation Morawski said. And in the U.S., it's Harris, also all web based.

Polling, Morawski says, is all about the sample. To accurately predict how a larger population feels about something -- a brand, a politician, even Conrad Black -- you need to survey a balanced chunk of the whole. (The opinions of 3,000 white males from Regina won't tell you much about how the entire country feels about anything.) Angus Reid selects their sample randomly from a larger group signed up on the Reid website. The results are then calibrated to take into account things like age, gender and voting preference.

The main criticism that Cherniak and Kinsella had was that by picking their sample online, AR Strategies excluded all those Canadians who aren't, thus preventing a truly representative slice. Morawski acknowledged that was a problem. But, he countered, there are ways to work around it. And what's more, the sample problems for telephone pollsters are just as bad and maybe worse.

The big issue for telephone polling is refusal rates, something Reid himself called "the big dirty secret" of the polling industry in a 2004 Tyee article. Put simply, every year more people refuse to take part in surveys. Many more simply don't pick up the phone for unknown callers, and some don't have home phones at all. In another Tyee article from 2005, Tom Barrett reported that for a typical B.C. election poll, a pollster must dial 6,000 to 10,000 numbers to get a representative sample. That, according to Morawski and Reid, means building a representative sample by phone gets tougher every year.

The Angus Reid online polls try to work around that by building in incentives for people to participate, Morawski told me. Each selected member of the sample is paid between $1 and $4 for each survey they fill out. Morawski said they are also conscious about keeping the surveys short and, as much as possible, enjoyable.

At the end of our interview, Morawski came back to the idea that no method for polling is perfect. "We get it, we know it, there are concerns," he said. "Is the Internet perfect, no.... It is just a new method and a new technology."

Related Tyee stories:

 [Tyee]

  • Share:

Facts matter. Get The Tyee's in-depth journalism delivered to your inbox for free

Tyee Commenting Guidelines

Comments that violate guidelines risk being deleted, and violations may result in a temporary or permanent user ban. Maintain the spirit of good conversation to stay in the discussion.
*Please note The Tyee is not a forum for spreading misinformation about COVID-19, denying its existence or minimizing its risk to public health.

Do:

  • Be thoughtful about how your words may affect the communities you are addressing. Language matters
  • Challenge arguments, not commenters
  • Flag trolls and guideline violations
  • Treat all with respect and curiosity, learn from differences of opinion
  • Verify facts, debunk rumours, point out logical fallacies
  • Add context and background
  • Note typos and reporting blind spots
  • Stay on topic

Do not:

  • Use sexist, classist, racist, homophobic or transphobic language
  • Ridicule, misgender, bully, threaten, name call, troll or wish harm on others
  • Personally attack authors or contributors
  • Spread misinformation or perpetuate conspiracies
  • Libel, defame or publish falsehoods
  • Attempt to guess other commenters’ real-life identities
  • Post links without providing context

LATEST STORIES

The Barometer

Do You Think Naheed Nenshi Will Win the Alberta NDP Leadership Race?

Take this week's poll