Marking 20 years
of bold journalism,
reader supported.
Mediacheck

Hey, Careful Where You Swing that Poll!

More muddled election math.

Tom Barrett 6 Dec 2005TheTyee.ca
image atom

If there were no real winners in the last federal election, there was one big group of losers: the pollsters.

At the end of a campaign that saw an unprecedented emphasis on media horse-race polls, the pollsters all had the Liberals and Conservatives in a dead heat. Seat projections based on the pollsters' numbers had most of the media predicting a Conservative minority government.

Instead, the Liberals beat the Conservatives by more than seven percentage points - a result that fell well outside all the major pollsters' margins of error. The pollsters argued that their numbers had been right but that the voters, especially in Ontario, had changed their minds at the last minute, after the last polls had been taken.

They were probably right. We might have hoped, though, for a little more caution in the interpretation of those last polls, given everything the pollsters have said since about the volatility of the voters.

'Aggressive' polling

But whatever the reason for the red faces in 2004, you might think that the pollsters - and the media that breathlessly reported their findings during the campaign - would vow to be more careful this time.

To a certain extent, they are. Pollsters are promising to stay in the field later this time around. They're interviewing more people. The Globe and Mail, for example, is promising bigger and better polls this time, what editor-in-chief Edward Greenspon calls "our most aggressive ever public-opinion program."

But anyone who read last Wednesday's Globe might be left wishing for a little more care and a little less aggression.

Under the headline "Liberals losing allure among minorities, poll suggests," the Globe described how immigrants and visible minorities appeared to be switching to the Conservatives.

The 20-column-inch story, which was accompanied by three pie charts, talked about how "the Liberal Party's traditional stranglehold on support from immigrants and visible minorities may be vulnerable to a sustained Conservative assault focusing on government corruption…"

This was based on a finding in a Strategic Counsel poll that the Conservatives were supported by "just under 30 percent" of visible minority respondents. (According to one of the pie charts that accompanied the article, the figure was 27 percent.) A further 38 percent of visible minority respondents supported the Liberals, placing the Conservatives "within striking distance" among this demographic.

The story didn't say what previous polls had reported about visible minority and immigrant voting intentions, but it did suggest that this was a sizeable gain for the Conservatives.

Never mind

Various reasons were given why minority groups might be abandoning the Liberals: same-sex marriage, the sponsorship scandal and Conservative organizing efforts.

Various experts were interviewed to speculate about the results. The recent B.C. municipal elections were factored in. The attitude of the Bloc Quebecois towards visible minorities - and vice-versa - was assessed.

Then, in the second-to-last paragraph of the story, the Globe ran into an expert who knew something about polling. Richard Johnston, the head of the UBC political science department "cautioned against reading too much into the Strategic Counsel poll because the statistical sample of minorities is necessarily smaller than the larger survey sample."

"Indeed," the story continued, the Strategic Counsel had noted that the stuff about visible minorities "was added as an area of interest only, and is not considered a scientific poll on par with the larger survey."

In the words of Saturday Night Live Weekend Update correspondent Emily Litella, "Never mind."

Turns out the survey in question interviewed only 180 visible minority persons across the country. That means that the sampling error for any result from this group would be greater than plus or minus seven percentage points, 19 times out of 20.

At the outside limits of the sampling error, the Conservative support among visible minorities could be as low as 20 percent and the Liberal support could be as high as 45 percent. Or the Conservatives might be ahead. You don't know.

The fact is, a national sample of 180 just isn't going to tell you much of anything.

Intrigue without substance

The point here is not to dump on the reporter in question, or on The Globe. Lots of media make similar mistakes; at least this story admitted at the end that the numbers weren't particularly trustworthy. (Although they were described as "intriguing.")

The point is that, as a news consumer, you shouldn't put too much stock in what national polls say about who Nova Scotians plan to vote for, or what Green Party voters think about private health care.

The media make plenty of mistakes when it comes to interpreting polls. Giving too much weight to the results of small groups within the overall sample is one of the most common.

Tom Barrett is a contributing editor to The Tyee.  [Tyee]

  • Share:

Facts matter. Get The Tyee's in-depth journalism delivered to your inbox for free

Tyee Commenting Guidelines

Comments that violate guidelines risk being deleted, and violations may result in a temporary or permanent user ban. Maintain the spirit of good conversation to stay in the discussion.
*Please note The Tyee is not a forum for spreading misinformation about COVID-19, denying its existence or minimizing its risk to public health.

Do:

  • Be thoughtful about how your words may affect the communities you are addressing. Language matters
  • Challenge arguments, not commenters
  • Flag trolls and guideline violations
  • Treat all with respect and curiosity, learn from differences of opinion
  • Verify facts, debunk rumours, point out logical fallacies
  • Add context and background
  • Note typos and reporting blind spots
  • Stay on topic

Do not:

  • Use sexist, classist, racist, homophobic or transphobic language
  • Ridicule, misgender, bully, threaten, name call, troll or wish harm on others
  • Personally attack authors or contributors
  • Spread misinformation or perpetuate conspiracies
  • Libel, defame or publish falsehoods
  • Attempt to guess other commenters’ real-life identities
  • Post links without providing context

LATEST STORIES

The Barometer

Do You Think Naheed Nenshi Will Win the Alberta NDP Leadership Race?

Take this week's poll