DISINFORMATIONAI Is Making Disinformation Harder to Spot—but We've Found a New Way to Catch It
As generative AI becomes more powerful, malicious groups are increasingly using it to produce and spread disinformation online. Automated accounts can flood social media with convincing comments designed to sow division, inflame political debate and undermine trust in reliable information. But our latest research offers a way to spot these attempts.
Do you ever see comments on social media that seem way off topic, but still manage to wrench the discussion around to divisive political debate?
A discussion about the cost of living suddenly becomes an argument about immigration. A conversation about the war in Ukraine turns into claims about government corruption. It can feel jarring – and sometimes this is deliberate.
As generative AI becomes more powerful, malicious groups are increasingly using it to produce and spread disinformation online. Automated accounts can flood social media with convincing comments designed to sow division, inflame political debate and undermine trust in reliable information.
But our latest research offers a way to spot these attempts. Rather than trying to identify whether a post was written by AI, we focus on something different: whether it’s trying to derail the conversation.
Until recently, identifying malicious accounts was often quite straightforward. Many campaigns relied on people writing in a second language. So, posts sometimes contained grammatical mistakes or unusual word choices. Detection systems could look for these patterns in the language used.
But generative AI has changed that. AI systems can now produce fluent, natural-sounding text that is much harder to distinguish from human writing. For example, patterns like use of em-dashes and the word “delve” used to be telltale signs of a text being generated by AI. But AIs are adapting, and these older systems are increasingly ineffective.
Trying to detect AI purely from the words people use is becoming a losing battle. We believe the better approach is to look at what a message is trying to achieve.
Looking for Signs
Attempts to spread disinformation often work by steering conversations away from their original topic, towards more polarizing issues. So, instead of analyzing individual words, we set out to build a system that could recognize this phenomenon in online discussions.
We analyzed comments posted beneath BBC News videos on YouTube, a platform that has previously been targeted by organized disinformation campaigns.
For example, imagine a comment about Ukraine’s president, Volodymyr Zelensky, interacting with senior UK political figures: “Zelensky must be wondering how many foreign secretaries the UK goes through.” Now, imagine another person responding: “Mind you, Zelensky has barely been president for four years. Maybe that’s why the little tyrant bans his opposition.”
Whether that second point is true or false is not the issue. Instead of responding to the original comment, it redirects the conversation towards a different, more divisive topic.
