Detecting Conspiracy Theories on Social Media

Social media platforms are concerned about malicious or harmful uses of their services, and as part of their effort to combat harmful content on their platforms, Google’s Jigsaw unit asked our RAND research team to help answer a difficult question:(1) How can we better detect the spread of conspiracy theories at scale? The scale of text on the internet is so vast that even large teams of humans can detect or flag only a fraction of harmful or malicious conspiracy theory language. Only machines can operate at that speed and scale. Jigsaw leaders further understood that tackling this issue is more than just an engineering challenge. Because the spread of conspiracy theories is a sociocultural problem, they wanted more than a black box and asked whether we could advance machine-learning (ML) applications to provide additional insight: How do online conspiracies function linguistically and rhetorically?

The ability to detect a variety of conspiracy theories at scale while understanding their functional and persuasive features is an important step in addressing the problem through evidence-based interventions. Given not only the harm posed by existing conspiracy theories, but also the proliferation of new ones—for example, that anti-fascist activists in the Antifa movement started fires in Oregon in summer 2020— we feel this report is both timely and urgent.

….

Four Conspiracy Theories For the research used in this report, we pulled data from Twitter that characterized four separate conspiracy theories about the existence of alien visitation, the danger of vaccinations, the origin of coronavirus disease 2019 (COVID-19), and the possibility of white genocide (WG). The alien visitation conspiracy theory offered a contrast to the others; it provides an example of an ideology that appears relatively benign

Understanding How Conspiracy Theories Work
To better understand how conspiracy theories function, we first conducted a literature review to capture the state of knowledge on this topic. We then conducted a mixed-method analysis of online conspiracy theory language, using computer text-mining to detect patterns in our conspiracy theory data sets along with human qualitative analysis to make sense of how those patterns function persuasively.

For this effort, we used the stance analysis capabilities in RANDLex.(2) Stance analysis is a text-mining approach used to determine how speakers represent the world linguistically—the style and tone that point to the sociocultural part of language. One example is certainty: a writer could choose to use hedging language (“I think,” “maybe,” “it’s possible that”) or to use epistemic certainty markers (“we know,” “it has been shown,” “there is”). Those are representational choices that speakers make in attempting to achieve social effects (such as persuasion) within cultural contexts (such as genre and setting).

Improving ML Detection Through Hybrid Modeling
Second, we built an ML model that would detect a variety of conspiracy theories. We innovated by creating a hybrid model that combined word embedding (semantic content) with linguistic stance (rhetorical dimensions). ML has already made great progress in recognizing the semantic content of text—for example, automatically detecting whether an article is about sports, hobbies, or world events. Word embeddings using a deep neural network (DNN) are an example of a powerful way to classify documents (one that accounts for words as they appear in context) and thus do a very good job of capturing the semantic meaning of documents.(3)

To capture stance in our model, we operationalized a taxonomy of rhetorical functions of language originally developed at Carnegie Mellon University. We have used stance by itself in previous modeling efforts and gotten good results—for example, when detecting Russian interference in elections solely through rhetorical style.

This hybrid modeling effort is important for two reasons. First, although ML is getting better all the time at recognizing text content, recognizing rhetorical dimensions has been challenging. It is one thing to identify an anti-vaccination topic; it is a very different thing for a machine to interpret the conspiratorial dimension of anti-vaccination talk, and the latter angle is critical if we want to distinguish between talk that promotes conspiracy theories and talk that opposes or simply addresses them. The second reason that hybrid modeling is important is that DNN models, although powerful and useful, are also black boxes. Stance in this context has an interpretable representation and is not so high-dimensional that it cannot be looked at by humans. Thus, a hybrid model using DNN’s semantic capability combined with stance would allow us to rank the importance of different rhetorical features and thus better understand how various conspiracy theories function rhetorically.

1. Jigsaw seeks to address technology threats and innovate for safer digital technologies.

2. RAND-Lex is RAND’s proprietary text and social media analysis software platform. It is a scalable, cloud-based analytics suite with network analytics and visualizations, a variety of text-mining methods, and ML approaches.

3. Classifying documents refers to assigning documents or text to a human-established set of classes. Also called human-supervised learning, this might, for example, mean inserting examples of threatening language, angry-but-not-threatening language, and neutral language into an ML model and then teaching the model to classify new documents into one of those classes.

Key Findings

·  The hybrid ML model improved conspiracy topic detection.

·  The hybrid ML model dramatically improved on either single model’s ability to detect conspiratorial language.

·  Hybrid models likely have broad application to detecting any kind of harmful speech, not just that related to conspiracy theories.

·  Some conspiracy theories, though harmful, rhetorically invoke legitimate social goods, such as health and safety.

·  Some conspiracy theories rhetorically function by creating hate-based “us versus them” social oppositions.

·  Direct contradiction or mockery is unlikely to change conspiracy theory adherence.

Recommendations

·  Engage transparently and empathetically with conspiracists.

·  Correct conspiracy-related false news.

·  Engage with moderate members of conspiracy groups.

·  Address fears and existential threats.