Mediacheck

Hey, Careful Where You Swing that Poll!

More muddled election math.

By Tom Barrett, 6 Dec 2005, TheTyee.ca

globe

If there were no real winners in the last federal election, there was one big group of losers: the pollsters.

At the end of a campaign that saw an unprecedented emphasis on media horse-race polls, the pollsters all had the Liberals and Conservatives in a dead heat. Seat projections based on the pollsters' numbers had most of the media predicting a Conservative minority government.

Instead, the Liberals beat the Conservatives by more than seven percentage points - a result that fell well outside all the major pollsters' margins of error. The pollsters argued that their numbers had been right but that the voters, especially in Ontario, had changed their minds at the last minute, after the last polls had been taken.

They were probably right. We might have hoped, though, for a little more caution in the interpretation of those last polls, given everything the pollsters have said since about the volatility of the voters.

'Aggressive' polling

But whatever the reason for the red faces in 2004, you might think that the pollsters - and the media that breathlessly reported their findings during the campaign - would vow to be more careful this time.

To a certain extent, they are. Pollsters are promising to stay in the field later this time around. They're interviewing more people. The Globe and Mail, for example, is promising bigger and better polls this time, what editor-in-chief Edward Greenspon calls "our most aggressive ever public-opinion program."

But anyone who read last Wednesday's Globe might be left wishing for a little more care and a little less aggression.

Under the headline "Liberals losing allure among minorities, poll suggests," the Globe described how immigrants and visible minorities appeared to be switching to the Conservatives.

The 20-column-inch story, which was accompanied by three pie charts, talked about how "the Liberal Party's traditional stranglehold on support from immigrants and visible minorities may be vulnerable to a sustained Conservative assault focusing on government corruption…"

This was based on a finding in a Strategic Counsel poll that the Conservatives were supported by "just under 30 percent" of visible minority respondents. (According to one of the pie charts that accompanied the article, the figure was 27 percent.) A further 38 percent of visible minority respondents supported the Liberals, placing the Conservatives "within striking distance" among this demographic.

The story didn't say what previous polls had reported about visible minority and immigrant voting intentions, but it did suggest that this was a sizeable gain for the Conservatives.

Never mind

Various reasons were given why minority groups might be abandoning the Liberals: same-sex marriage, the sponsorship scandal and Conservative organizing efforts.

Various experts were interviewed to speculate about the results. The recent B.C. municipal elections were factored in. The attitude of the Bloc Quebecois towards visible minorities - and vice-versa - was assessed.

Then, in the second-to-last paragraph of the story, the Globe ran into an expert who knew something about polling. Richard Johnston, the head of the UBC political science department "cautioned against reading too much into the Strategic Counsel poll because the statistical sample of minorities is necessarily smaller than the larger survey sample."

"Indeed," the story continued, the Strategic Counsel had noted that the stuff about visible minorities "was added as an area of interest only, and is not considered a scientific poll on par with the larger survey."

In the words of Saturday Night Live Weekend Update correspondent Emily Litella, "Never mind."

Turns out the survey in question interviewed only 180 visible minority persons across the country. That means that the sampling error for any result from this group would be greater than plus or minus seven percentage points, 19 times out of 20.

At the outside limits of the sampling error, the Conservative support among visible minorities could be as low as 20 percent and the Liberal support could be as high as 45 percent. Or the Conservatives might be ahead. You don't know.

The fact is, a national sample of 180 just isn't going to tell you much of anything.

Intrigue without substance

The point here is not to dump on the reporter in question, or on The Globe. Lots of media make similar mistakes; at least this story admitted at the end that the numbers weren't particularly trustworthy. (Although they were described as "intriguing.")

The point is that, as a news consumer, you shouldn't put too much stock in what national polls say about who Nova Scotians plan to vote for, or what Green Party voters think about private health care.

The media make plenty of mistakes when it comes to interpreting polls. Giving too much weight to the results of small groups within the overall sample is one of the most common.

Tom Barrett is a contributing editor to The Tyee.  [Tyee]

42  Comments:

Login or register to post comments

  • Chris H

    6 years ago

    Comments on "Hey, Careful Where You Swing that Poll!"

    "Under the headline "Liberals losing allure among minorities, poll suggests," the Globe described how immigrants and visible minorities appeared to be switching to the Conservatives."

    You can read a lot into the headlines. I wonder how many people just read the headlines and don't bother with the whole article. I'm guessing a lot. There is definitely a bias that comes out from the publication by what kind of headline they use. Are they chosen to entice the reader to actually read the article, or are they the message itself?

  • darcy.mcgee

    6 years ago

    Quote:
    In the words of Saturday Night Live Weekend Update correspondent Emily Litella, "Never mind."

    Odd, pedantic reference. You couldn't have come up with "Never mind" on your own? Does anybody watch Saturday Night Live anymore?

  • darcy.mcgee

    6 years ago

    Incidentally, it's my contention (based on watching polls closely last time) that you essentially need about 15 polling groups to come up with any meaningful poll of Canadians.

    You need a sample from each province, large enough to be representative of that province. You then need a sample from major urban centres within provinces or regions -- Vancouver & Kelown, the GTA, Montreal & Quebec, Halifax, Fredricton etc.

    You could probably combine the Atlantic urbans and the central Canada urbans. BC, Alberta, Ontario, Quebec need their own. I may be wrong about combining...I change my mind on that every once in a while.

    Let's not pretend that the release of results from Ontario didn't have an effect in the west; it did. Elections Canada should do the smart thing and lock down all information release until all polls nationwide have been closed.

  • BC Mary

    6 years ago

    There's "definitely a bias" that comes out of polls, too. Why isn't it significant, in assessing the accuracy or usefulness of polls, to examine the pollsters? I don't accept their claims of neutrality -- not without a lot more work on their part, to maintain a neutral starting position. And to shut the heck up, between elections, about pushing their own choices.

    Allan Gregg is my favourite im-pollster. A dyed-in-the-wool Conservative, formerly part of Lyin' Brian's PMO, he has the Strategic Counsel polling tool. With these clearly biased credentials, he then pontificates regularly, posing as a genuine political pundit. But he's clearly a Conservative shill. I mean, come on ... Gregg's Strategic Counsel wouldn't be able to see an NDP gain if it bit him in his boxers.

    Was Strategic Counsel responsible for that erroneous "poll" published by the Globe and Mail on the eve of the last B.C. Election ... I forget ... but it's what bothers me a lot about polls. Polls are not impartial measures of public opinion ... they are political instruments serving whoever paid the pollster to create them.

  • Elliot

    6 years ago

    'Allan Gregg is my favourite im-pollster. A dyed-in-the-wool Conservative,'
    which allan gregg have you been watching? he's the cbc's favourite conservative basher. but i guess seeing him next to all those other lefties he may appear to be blue. and mulroney was not a conservative, he was a progressive conservative. big difference there. this will be the first true conservative gov't we've had for a very long time.

  • Ron Erwin

    6 years ago

    BC Mary, I know Allen Gregg, he is not now a conservative. He was when he was writing speeches for Alberta Conseavatives 25 years ago, but once he moved to Ottawa, he became part of the liberal establishment.
    Beware of his views, they are very liberal now.

  • darcy.mcgee

    6 years ago

    Quote:
    Polls are not impartial measures of public opinion ... they are political instruments serving whoever paid the pollster to create them.

    Polls CAN be impartial measures, it's jut hard to know when they're one or the other sometimes...

    Why bash Allan Gregg as a conservative (inaccurately, as has been pointed out) and leave David Schreck (who writes for the Tyee) or Rafe Mair out? Both of these guys are so biased, they can't hide it even when they try.

    Or David Beers for that matter.

    Bias is a fact of life, and exists in a multitude of places. Do not judge people who pretend to be impartial by the nature of their bias, judge them by their willingness to disclose it.

    It's also worth pointing out that you should not judge people by your personal biases and viewpoints, but rather judge them by their actual actions and words.

  • BC Mary

    6 years ago

    OK, Ronnie, who I never thought I'd speak to in this millenium, tell me everything you know about your friend's music company which was such a disaster it threatened the careers of some of the best artists performing today.

    And are you honestly telling me that while working for Lyin' Brian, the wonderboy Allan Gregg wasn't still some sort of Reform Alliance CCRAP Progressive Conservative (not Communist Party of Canada) CPC conservative?

    Right now, I know, he's clearly a weasel. But that'd make him a conservative more than a liberal.

  • BC Mary

    6 years ago

    Oh. And Darcy, I am not judging "people", I am judging pollsters and their personal effects, manipulations, and dishonesty on polls. Big difference there too.

  • BC Mary

    6 years ago

    P.S. Does David Schreck do polls? Or Rafe Mair? I don't think so. But neither have been the least bit shy about divulging their political biases. Crikey, don't you recall the uproar when Rafe declared himself a Leftie?? But I repeat: neither of them do polling. What they state is personal opinion, with sources provided. Big, big difference.

  • Ron Erwin

    6 years ago

    BC Mary, what I am telling you is that Allen Gregg is a part of the Ottawa elite. Whoever is in power. It's bben the Liberals for years,so you won't see him stepping on any Liberal toes. He will be doing his best to spin the Lib's back into power so he won't have to make a whole bunch of new friends.

  • Skip Tracer

    6 years ago

    Hey Ron, if you know Allan Gregg so well, why do you keep mis-spelling his name?

  • lenin's ghost

    6 years ago

    there was absolutely nothing progressive about mulroney. that prick should be hung for treason!

  • Wallace

    6 years ago

    I believe it was Diefenbaker who said polls are for dogs. I may be wrong in the attribution, but whole heartedly agree with the sentiment. The misuse of polls generally carries the problem of bias, and push-polling has been a particular favourite of the Straussian neocons.

  • BC Mary

    6 years ago

    Tom Barrett sums up the whole story on polls:

    Quote:
    The point is that, as a news consumer, you shouldn't put too much stock in what national polls say ... the media make plenty of mistakes when it comes to interpreting polls. Giving too much weight to the results of small groups within the overall sample is one of the most common.

  • darcy.mcgee

    6 years ago

    I meant my comments about judging people by your own biases in a more general sense, and I apologize if you took it personally Mary. It wasn't really intended.

    I've just gotten quite used to people on both sides of the political spectrum jumping to instant conclusions and assumptions which really reflect more of their personal bias. It grows tiresome, and is no more acceptable from the left or the right.

    As for Schrek, Beers, Mari et al. They all profess to provide journalism on a regular basis, not just "opinion." I've seen many a Schreck article printed with pro-NDP statements that did not explicitly disclose his past status as an NDP MLA and his close association with the party. That I know this doesn't mean it shouldn't be disclosed: many many others do not, or may forget.

    If it were up to me, ever publication would have a personal "bias-meter" posted. The Tyee for example, should wear a "hard left" badge. I don't consider this problematic (I read The Walrus, The Atlantic and Harpers as often as possible) but it would help to orient new users and continually put commentary from its "journalists" in the proper perspective.

    As an example, I would no more expect to see a fair evaluation of hospital privatization in The Tyee than I would in a Fraser Institute publication; the former would begin with the premise/belief that it is bad, while the latter would being with the premise that it is good. I evaluate each's commentary within that framework.

  • allan

    6 years ago

    Did someone really mean to say Brian Mulroney was a progressive conservative?

    Is that a joke?

    He stuck a sharp knife into the back of the last progressive conservative leader in Canada.

  • burner

    6 years ago

    lyin' brian was and is a mulroneyite.

    anyone who believes otherwise is badly uninformed.

    he thinks he was the best pm ever, yet seems oblivious to the fact he alone burned the pc party to the ground.

    no phoenix has risen from the ashes.

    just a turkey called steve, who takes his cues from a turkey called lyin' brian.

    in regard to polls - one should examine who is paying the piper, for it is they calling the tune.

  • Chris H

    6 years ago

    darcy.mcgee,

    The funny thing with these "far left" and "far right" biases is that they are never static. Many people have opinions on different topics that span the spectrum of political debate. Some might say that my opinions are very "left" on one topic while "right" on another. Why should myself, or any publication, get lumped in with a label that may not always be true? Better to look at individual arguments and articles to see how credible the speaker or author is. I know that takes time and effort, but why bother coming to the Tyee if not to read the articles, and perhaps, learn something.

    I have noticed that people, in general, hate being called "far right" much more than "far left." Why do you think that is? People with opinions that are "far right" seem to have a chorus of chants that include: "We are the majority." "We are the mainstream." "We are objective." "we are fair and balanced." "We are reasonable thinking persons." I say they protest a tad too much.

  • darcy.mcgee

    6 years ago

    I do read the Tyee's articles, although I don't find myself learning very much. The Atlantic or Harper's is filled with more ideas, and the Tyee tends to be a left wing analysis of the BC Political scene. It's bias skews the value of its articles.

    There are exceptions: the 100 Mile Diet series was well done, and informative.

    In general, I feel that it's more politically acceptable to be labeled "far left" than "far right" which tends to connote "American."

    In reality, there is little objective difference between these biases, although the subtance of the arguments will be different. People who are driven by them are ideologues, and are unable to see beyond the blinders that they are wearing.

    An example: for a variety of reasons, I supported some conservative party candidates in the last election. When this came up in conversation at dinner with a friend of a friend - the first time I had ever met this person - a palpable glaze camer over her eyes. When she found out that I was a registered BC Liberal, the vitriol began to spew. Me? I didn't want to talk about it - I was trying to have a social night with my friend's friends.

    At a subsequent event 3 months after the last provincial election, the same person persistently insisted that Geoff Plant was the A-G. My corrections (It's Opal) went unheard.

    She was so absorbed in her own dogma (self-ackowledged very left wing) that she didn't see past the label, and wasn't even current (it was important to her job that she know who the A-G was.)

    There are people on the left and right who claim to be balanced and are not; there are also reasonable, thinking persons on the left and right, and it's the discussion of ideas between these types of people which moves society forward.

    The discussion of ideas between people who are obsessed with backroom politics gets us nowhere.

    The screeching back and forth of far left dogma vs. far right dogma gets us nowhere.

    If we're not trying to develop a better society, what's the point of government? Isn't it ultimately a social contract that we all share, not just special interest groups?

  • Fiat lux

    6 years ago

    Polls shouldn't be allowed, at least not in publications, during election campaigns, as many people, who have no idea of the issues, tend to vote for the apparent, predicted "winners"

    The way the polls swing around show that a very large percentage of the so called "swing voters", people who are without a clue on what's going on, just sway around with the breeze, all over the place.

    Ed Deak.

  • BC Mary

    6 years ago

    darcy. This article and this thread are about polls and polling. In this discussion, it's irrelevant whether or not various journalists have a bias when they write articles.

    But it's powerfully distorting when a pollster acts out his self-serving bias by means of polls which are touted as being "accurate" ... now and then ... usually ... 19 times out of 20 ... within 3 or 4 percentage points ... and only on Thursdays when the moon is full.

    Political decisions are made, people vote, and the trend of the nation's development are increasingly influenced by polls. Far too often, people will depend upon a poll for their take on what's really happening and how to vote ... when often, the poll is pointing in a false direction.

    Polls are exerting an enormous influence on our lives. Imagine Allan Gregg pontificating (and being well paid to do it) on a weekly basis, nation-wide, on his bogus view of what's real and what's wrong.

    It's damwell time we took a close look at who is creating the polls we're depending upon.

  • dononmain

    6 years ago

    I have a poll question for you.
    Are you going to vote for the guys who skim off the top, or the guys who are being led by a nut?

  • Frank

    6 years ago

    I'm going to vote for the guys that steal rings because I believe in honesty and integrity in government.

    Quote:
    If we're not trying to develop a better society, what's the point of government? Isn't it ultimately a social contract that we all share, not just special interest groups?

    Gov't keeps us from killing each other in the street. Everything else about that contract we argue about.

  • Skip Tracer

    6 years ago

    Frank sez:

    Quote:
    Gov't keeps us from killing each other in the street.

    No. It doesn't. It can't even provide clean drinking water for some of its marginalized citizens.

  • Ron Erwin

    6 years ago

    Skip Tracer, oh, that Allan Gregg, sorry old boy. I am only a stupid neo-con.
    I can't help myself, you know what it's like.

  • Skip Tracer

    6 years ago

    Ron Erwin dropped to the ground and, with tears welling in his eyes, embarked on a program of self-flagellation by declaring:

    Quote:
    I am only a stupid neo-con.
    I can't help myself, you know what it's like.

    Jeez, Ron. I was just wondering about your spelling in relation to your close association with Al. And don't you mean Neo-Liberal? ;-)

  • ROBBINS Sce Research

    6 years ago

    thought you folks might want some fresh meat!

    December 6, 2005

    ROBBINS Sce Research (1998)
    robbinsscereseach.com

    For immediate release-

    Harper position on health care trumps Layton’s 3/2.
    Harper GST tax relief trumps Martin income tax 3/2.
    Harper most trusted with tax dollars
    Harper, Layton and Martin dead heat on trust relating to health care.
    Harper breaks open slight lead over competitors in B.C.

    Question #1-Which of the following two choices most suitably depicts how you perceive access to health care in Canada?

    (a) I would do whatever was necessary to ensure proper and immediate health care for my loved ones- (61%); (b) I am aware there are difficult choices but I do not believe in buying one’s way to the front of the health care line- (39%)

    Question #2-Which of the following two forms of tax relief do you prefer?

    (a) Cuts to the Goods and Services Tax (GST)-(65%); (b) Personal Income Tax relief- (35%).

    Question #3-Which of the following political leaders do you trust most with your tax dollars?

    (a) Stephen Harper- (41%); (b) Jack Layton- (29%); (c) Paul Martin- (30%).

    Question #4-Which of the following leaders do you most trust with Health Care?

    (a) Stephen Harper- (35%); (b) Jack Layton- (32%); (c) Paul Martin- (33%)

    Question #5-Which federal political leader and party are you currently leaning toward voting for on January 23, 2006?

    (a) Paul Martin and federal Liberal Party- (29.5%); (b) Jack Layton and NDP Party (32%); (c) Stephen Harper and Conservative Party (37%); Other- (04%). * Undecided (4.5%) (Numbers statistically adjusted on Question #5).
    Respondents- Voting Age to 45-(37%); Above 45 Years of Age- (63%)
    BC Age Demographics (Actual)-Voting Age to 45-(48%); Above 45 Years of Age- (52%)

  • ROBBINS Sce Research

    6 years ago

    Trying to fit into limited space 1100 or so respondents.

  • lynn

    6 years ago

    Quote:
    I'm going to vote for the guys that steal rings because I believe in honesty and integrity in government, wrote Frank wryly.

    Well, even though I'm covered in flour and sparkly cookie decorations just gotta interrupt my holiday baking to say that the above quote really made me chuckle...so very true and so very funny, Frank.

    That is, unfortunately, the real state of honesty and integrity in government these days...

  • allan

    6 years ago

    Robbins etc., would you be so good as to state who paid for your polling.

    I'd really prefer actual names so we can make our own decisions rather than hearing from you that your polling is funded by a group of interested citizens who do not wish to be identified.

    Lynn and Frank, I agree. Your reasoning ought to be framed and hung in Parliament Frank so that it is a constant reminder to those who believe they have entitlement.

  • Skip Tracer

    6 years ago

    Quote:
    thought you folks might want some fresh meat!

    Actually, we'd like to know who framed the questions.

    Even the Fraser Institute is not behind your man on the GST cut.

  • Skip Tracer

    6 years ago

    Oh, one more thing; the question on healthcare is useless in determining the public's feelings vis-a-vis medicare and the various solutions being floated to "fix" it.

  • herbie

    6 years ago

    I always tell the pollsters I'm voting Conservative. It helps get three people elsewhere to make sure they get out and vote Liberal. It's also fun to get Alberta's hope up and listen to them gripe afterwards...

  • dangrice.com

    6 years ago

    herbie, I'm on a cell phone, so the Pollsters leave me alone. But when I had a land line, I found it was fun to ask pollsters what they thought on the issues. They never seemed to want to answer their own questions.

  • Frank

    6 years ago

    Thanks allan and lynn.

    herbie, as I said last election, tell the conservative pollsters that not only are you one of theirs but you need a ride to the voting booth during that cold January day. I see no reason the Conservatives can't buy some of our gas back that they to the Yankees in order to take me to the booth.

  • Frank

    6 years ago

    insert "gave away" before "to the Yankees"

  • BC Mary

    6 years ago

    Well done, guys. The pollster has indeed shut the heck up.

  • darcy.mcgee

    6 years ago

    Funy how this:

    Quote:
    darcy. This article and this thread are about polls and polling. In this discussion, it's irrelevant whether or not various journalists have a bias when they write articles.

    and this:

    Quote:
    Well done, guys. The pollster has indeed shut the heck up.

    Go together.

    This article is about bias in a supposedly objective industry. Polls wouldn't matter if the media didn't report them, and most people - the vast majority learn about polls through the media. Media bias is actually relevant.

    My essential argument is "don't fight against the bias, use it -- instead -- as a guide in how to inerpret what you're reading."

    This is true of polls, journalism, and people who are trying to scream louder than you in order to get you to "shut the heck up."

    That's not discussion, that's dogma.

  • Mooney

    6 years ago

    No discussion of polls should forget the poll that had Kim Campbell winning a majority gov't by something like 44% before she was blown out of the water and the conservatives got wiped off the map by Jean Cretien. The so-called yesterday's man.

  • BC Mary

    6 years ago

    Darcy, old chum (and how did you get the name of one of Canada's Fathers of Confederation, and a very respectable one at that??)

    ... I assuredly do not want everybody to "shut the heck up" (well, maybe Little Ronnie once in a while) ... just that annoying pollster who is supposed to be neutrally counting the public's opinions not yelling at them about how they ought to be thinking (according to him, a failed Reform leader).

    That's all. Just the contradictory, dishonest guy who is trying to pull the woolies over our eyes.

    And hey, if David Schreck set up a Polling Service and behaved that way, you'd very soon get the idea, too.

  • BC Mary

    6 years ago

    Oh, Darc, that wasn't meant to be "funy" either.

    • No best comments selected by an editor for this story yet. To see all comments, click the All Comments tab, above.
    • The discussion for this story is closed. No more comments can be added.