AIHow Poisoned Data Can Trick AI − and How to Stop It
The quality of the information that the AI offers depends on the quality of the data it learns from If someone tries to interfere with those systems by tampering with their training data –either the initial data used to build the system or data the system collects as it’s operating to improve –trouble could ensue.
Imagine a busy train station. Cameras monitor everything, from how clean the platforms are to whether a docking bay is empty or occupied. These cameras feed into an AI system that helps manage station operations and sends signals to incoming trains, letting them know when they can enter the station.
The quality of the information that the AI offers depends on the quality of the data it learns from. If everything is happening as it should, the systems in the station will provide adequate service.
But if someone tries to interfere with those systems by tampering with their training data – either the initial data used to build the system or data the system collects as it’s operating to improve – trouble could ensue.
An attacker could use a red laser to trick the cameras that determine when a train is coming. Each time the laser flashes, the system incorrectly labels the docking bay as “occupied,” because the laser resembles a brake light on a train. Before long, the AI might interpret this as a valid signal and begin to respond accordingly, delaying other incoming trains on the false rationale that all tracks are occupied. An attack like this related to the status of train tracks could even have fatal consequences.
We are computer scientists who study machine learning, and we research how to defend against this type of attack.
Data Poisoning Explained
This scenario, where attackers intentionally feed wrong or misleading data into an automated system, is known as data poisoning. Over time, the AI begins to learn the wrong patterns, leading it to take actions based on bad data. This can lead to dangerous outcomes.
In the train station example, suppose a sophisticated attacker wants to disrupt public transportation while also gathering intelligence. For 30 days, they use a red laser to trick the cameras. Left undetected, such attacks can slowly corrupt an entire system, opening the way for worse outcomes such as backdoor attacks into secure systems, data leaks and even espionage. While data poisoning in physical infrastructure is rare, it is already a significant concern in online systems, especially those powered by large language models trained on social media and web content.
