How Foreign Operations Are Manipulating Social Media to Influence Your Views
Model Misbehavior
The consequences of such operations are difficult to evaluate due to the challenges posed by collecting data and carrying out ethical experiments that would influence online communities. Therefore it is unclear, for example, whether online influence campaigns can sway election outcomes. Yet, it is vital to understand society’s vulnerability to different manipulation tactics.
In a recent paper, we introduced a social media model called SimSoM that simulates how information spreads through the social network. The model has the key ingredients of platforms such as Instagram, X, Threads, Bluesky and Mastodon: an empirical follower network, a feed algorithm, sharing and resharing mechanisms, and metrics for content quality, appeal and engagement.
SimSoM allows researchers to explore scenarios in which the network is manipulated by malicious agents who control inauthentic accounts. These bad actors aim to spread low-quality information, such as disinformation, conspiracy theories, malware or other harmful messages. We can estimate the effects of adversarial manipulation tactics by measuring the quality of information that targeted users are exposed to in the network.
We simulated scenarios to evaluate the effect of three manipulation tactics. First, infiltration: having fake accounts create believable interactions with human users in a target community, getting those users to follow them. Second, deception: having the fake accounts post engaging content, likely to be reshared by the target users. Bots can do this by, for example, leveraging emotional responses and political alignment. Third, flooding: posting high volumes of content.
Our model shows that infiltration is the most effective tactic, reducing the average quality of content in the system by more than 50%. Such harm can be further compounded by flooding the network with low-quality yet appealing content, thus reducing quality by 70%.
Curbing Coordinated Manipulation
We have observed all these tactics in the wild. Of particular concern is that generative AI models can make it much easier and cheaper for malicious agents to create and manage believable accounts. Further, they can use generative AI to interact nonstop with humans and create and post harmful but engaging content on a wide scale. All these capabilities are being used to infiltrate social media users’ networks and flood their feeds with deceptive posts.
These insights suggest that social media platforms should engage in more – not less – content moderation to identify and hinder manipulation campaigns and thereby increase their users’ resilience to the campaigns.
The platforms can do this by making it more difficult for malicious agents to create fake accounts and to post automatically. They can also challenge accounts that post at very high rates to prove that they are human. They can add friction in combination with educational efforts, such as nudging users to reshare accurate information. And they can educate users about their vulnerability to deceptive AI-generated content.
Open-source AI models and data make it possible for malicious agents to build their own generative AI tools. Regulation should therefore target AI content dissemination via social media platforms rather then AI content generation. For instance, before a large number of people can be exposed to some content, a platform could require its creator to prove its accuracy or provenance.
These types of content moderation would protect, rather than censor, free speech in the modern public squares. The right of free speech is not a right of exposure, and since people’s attention is limited, influence operations can be, in effect, a form of censorship by making authentic voices and opinions less visible.
Filippo Menczer is Professor of Informatics and Computer Science, Indiana University. This article is published courtesy of The Conversation.
