AI Is Making Disinformation Harder to Spot—but We've Found a New Way to Catch It

This is known as a red herring: introducing an unrelated issue that distracts from the original discussion. These kinds of shift are difficult for conventional disinformation detection systems to identify, because they are not tied to particular words or phrases.

We manually analyzed more than 1,600 comments under BBC News videos, labelling them according to 25 different features of online discussion – and discovered some clear patterns.

We found that 36% of derailing messages had red herrings, 65% had leaps in logic known as “non sequiturs”, and 20% contained personal attacks. They were also much less likely to acknowledge previous comments or express empathy.

Spotting Manipulation
The next step was to see whether an AI system could recognize these patterns automatically. We used an AI to catch an AI.

For every genuine online comment, we asked an AI large language model to generate several reasonable, relevant responses. Returning to the example of UK foreign secretaries, the AI suggested replies such as: “The current situation in this country must come as quite a shock” or “One too many?”. Both responded directly to the original point.

The system then compares the real response with our AI-generated replies. If the actual comment differs substantially, it may indicate that someone is attempting to steer the conversation in a different direction. So, rather than searching for suspicious words, our system looks for unexpected changes in the flow of the discussion.

We tested this approach using our manually labelled dataset. In our second study, the system correctly identified derailing comments around 77% of the time.

That’s far from perfect, but no detection system is – particularly when analyzing something as complex as human conversation. However, our approach performed around twice as well as existing systems based on word-level sentiment analysis. It also achieved results comparable with the level of agreement between human researchers.

Our approach is effective because the AI learns what a typical response to a conversation looks like. When a reply unexpectedly changes the discussion, the system can identify that change and analyze patterns that earlier methods couldn’t detect.

Of course, going off topic isn’t necessarily a sign of malicious intent or disinformation. People naturally take conversations in unexpected directions, and there are many legitimate reasons why discussions evolve.

For that reason, this technology may act as an early-warning system rather than a replacement for human judgment. It could help moderators identify conversations that deserve closer attention – but any final decisions should remain with trained experts.

There are also important ethical questions to address. AI systems can reflect biases in the data they are trained on, and they still do not understand conversations in quite the same way that people do. Improving how AI represents and interprets human discussion remains a challenge.

As AI-generated content becomes increasingly difficult to distinguish from human writing, detecting disinformation requires more than simply searching for telltale words. It requires understanding how conversations work, how they are manipulated, and when someone is trying to quietly steer them off course.

Seán Roberts is Lecturer in Linguistics, Cardiff University. Kateryna Krykoniuk is Honorary Research Fellow in the School of Languages, Arts and Societies, University of Sheffield. This article is published courtesy of The Conversation.