Hey Tyee Commenters! Scholars Studied You. Here’s What They Found

Many of us think that thought-provoking and constructive online news comments are rare and that comments do more harm than good. Uncivil online comments, especially those responding to articles about sensitive issues such as climate change, immigration or Indigenous people, erode the public’s trust in media and can lead to harm in the real world.

And yet, news organizations, journalists and media analysts believe in the power of comments to engage communities, create meaningful discussions and to include diverse voices in the public discourse.

Our research group at Simon Fraser University is studying online comments to distinguish the constructive from the toxic. We were pleasantly surprised to see people expressing different viewpoints, providing evidence to support their opinion, sharing personal stories and experience, attempting to inform, convince or better understand the other side, and overall engaging in a meaningful conversation. We have also, sadly, read many vile and insulting comments.

Our in-depth analysis of more than 1.5 million comments gives us hope, while at the same time providing insights for how to make comment sections better.

Constructive or toxic?

We define constructive comments as those that are relevant to the article, make well-reasoned arguments and add meaningful discussion. On the opposite end of the spectrum, toxic comments contain insults, profanity or attacks on the authors or people mentioned in an article.

Using automatic text analysis methods, we have classified online news comments from three Canadian publications: the Globe and Mail, The Tyee and the Conversation Canada. We found that the majority of comments are non-constructive, but also not toxic. They tend to be short and do not contribute much to the conversation. Versions of "I agree" or "I disagree" make up the vast majority of these comments.

Interestingly, we also found that a large number of comments are constructive and toxic at the same time. They contain well-reasoned arguments as well as hateful language. These comments sound condescending and often open with an insult ("You are all idiots" is a mild example), followed with reasoning.

Readers of The Tyee will be pleased to know that this publication has many more constructive comments than the average of the three. There are consistently more constructive than non-constructive comments (except for many of the "I agree" variety, which are neither constructive nor toxic). Few comments on Tyee articles have 80 per cent toxicity or above. There seems to be a good community of Tyee commenters participating in robust debate. Or perhaps there are good moderators at The Tyee. Or perhaps both.

Topics matter

We used topic modelling, a technique to discover the themes or topics in large bodies of data. We then checked which topics receive the most comments, across all three publications. Unsurprisingly, we found that the three top topics with the most comments are about politics at different levels (local, national, international). What was interesting, though, was that those topics tended to include mentions of individuals and abstract words, such as "talk," "problem," "life," "family" and "community." It seems that people comment more about politics than other topics, but they bring in personal experience and evidence when they do so.

Our next question was whether the topics that emerged from our data showed different levels of constructiveness and toxicity. The answer is "yes" for constructiveness. Some topics lead to more or less constructive debate, with general politics or international politics having more constructive comments, as opposed to local politics or policy. It’s possibly easier to be constructive when topics do not affect the commenter so directly.

For toxicity, what we found is that toxicity is an equal-opportunity offender. Unlike constructiveness, toxicity seems to pattern in roughly the same proportion across topics. On average, the toxicity across all topics and across all three publications appears to be about 23 per cent. In other words, the average comment is one fourth toxic in its content. We expected some topics to be more polarizing than others, but instead we found not much fluctuation between topics, suggesting that toxicity in comments is likely just a feature of online language.

In conclusion, we can say that a large proportion of comments do contribute to the conversation, participating in healthy debate. It is also somewhat encouraging to see that there are small numbers of toxic comments (with the caveat that all three publications moderate their comments).

What can news organizations learn from this? We believe it is useful to find that many commenters are contributing thoughtful discussion. Many comments seem to be well reasoned, but open with an insult, which probably deters engagement with the reasoning and only leads to engagement with the insult. Nudging commenters to think of constructiveness as a feature of commenting may lead to more productive discussion.

For further detail on our findings, please see the full report.