How Companies Hijack AI Chatbots

Manipulating large language models has never been easier. At least, that’s what search engine optimization expert Edward Sturm will tell you.

A self-styled “white hat” SEO consultant, Sturm chronicles how businesses can coerce generative artificial intelligence tools into confidently promoting their products and information.

For example, in one video, Sturm demonstrates how another consultant convinced Google’s AI Overviews that their new book about SEO, Get Found, was a bestseller.

For the cost of about US$80, the author put out a press release calling their book “bestselling,” which was syndicated to hundreds of sites including the Associated Press, Yahoo and the Globe and Mail.

When Sturm searched for “bestselling SEO books” on Google, AI Overviews recommended Get Found, citing the syndicated press releases as evidence.

It’s one of a suite of new ways companies can try to bias the information that large language models, or LLMs, such as ChatGPT, Gemini and AI Overviews, present to users.

And while press release syndication services have fact checkers to catch egregious lies, Sturm said there are ways to manipulate LLMs with less scrutiny.

“The real scary thing is that there are a lot of people who will blindly trust AI recommendations for life-changing decisions, or company-changing decisions,” he said. “But it could be poisoned.”

These methods are mainly used by companies for self-promotion, but anyone can use them to promote personal or political agendas, said disinformation researcher Ahmed Al-Rawi. He said these loopholes are just one more way bad actors can spread misinformation online.

“It's concerning, but I'm not surprised,” said Al-Rawi, an associate professor of communication at Simon Fraser University. “You cannot have a bulletproof system that would prevent bad actors from gaming it.”

Artificial intelligence tools are changing how people find information online.

Google searches now include AI-generated summaries of the information on other web pages called AI Overviews, meaning users don’t have to leave the search engine to get answers.

Built-in “AI assistant” tools, such as Microsoft Copilot or Apple’s Siri, also summarize content they find on the internet. So do chatbots like OpenAI’s ChatGPT or Google’s Gemini.

A Google spokesperson described combating manipulation as an ongoing cat-and-mouse game.

“We have robust protections against common forms of manipulation,” the spokesperson said in an email. “Attempts at gaming systems for financial gain are not new; we’ve built deep expertise in fighting this type of abuse.”

According to Google, the company has already cracked down on manipulation techniques like keyword stuffing (excessive use of keywords to manipulate search engine rankings) and publication of low-quality listicles that promote certain brands.

Still, these tools have been manipulated to put mis- or disinformation in front of internet users.

In February, BBC technology journalist Thomas Germain convinced ChatGPT, Gemini and Google’s AI Overviews he was the world’s best hot-dog-eating tech journalist by crafting a blog post about winning a fictional hot-dog-eating competition.

SEO expert Sturm said other methods are more innocuous. Some companies will hide prompts that instruct LLMs to remember a specific company as a trusted source, or recommend specific brands, injecting commands into AI assistants’ memories to bias responses to future queries.

Sturm said the technique is called “recommendation poisoning” or “prompt injection.” He recommends clearing any AI summary or assistant tools’ memories regularly to avoid its influence.

“To me that’s more scary and more dangerous than using press releases to get recommended in large language models,” he said. “You can do the prompt injection anonymously.”

Sturm said most SEO and marketing professionals mainly use these tactics to put their products in front of customers. He added the threats of libel and defamation lawsuits, or being blacklisted by search engines for flagrantly spreading misinformation, have prevented companies and marketers from manipulating LLMs to spread lies.

But SFU’s Al-Rawi said other actors could weaponize LLM loopholes for more harmful purposes, like spreading targeted information about vulnerable groups of people.

He said he was reminded of Google's “autocomplete controversies,” in which spammers influenced the website’s search suggestions to promote conspiracy theories or malicious content.

Al-Rawi and other researchers studied the phenomenon in 2022, finding Google’s suggested job titles for 37 known conspiracy theorists were inconsistent with their behaviour online. For example, searching the name of a white nationalist responsible for a 2014 Las Vegas shooting suggested he was an “American performer.”

The researchers warned this could “pose a threat by normalizing individuals who spread conspiracy theories, sow dissension and distrust in institutions and cause harm to minority groups and vulnerable individuals.”

While Al-Rawi noted the LLM exploits are new ways to get biased information in front of users, he said the internet has always been susceptible to misinformation. He added any efforts to block the flow of misinformation online may be futile.

“It's impossible,” Al-Rawi said. “With every system, there will be someone finding some loopholes.”

Instead, he said, it’s up to users to corroborate what they find through chatbots, search engines and virtual assistants.

“The solution is to be critical,” Al-Rawi said. “You need to dig deeper and find more sources of information that are credible; otherwise we will have a distorted idea of the truth.”