Perspective: AIAI Could Be a Disaster for Humanity. A Top Computer Scientist Thinks He Has the Solution.

Published 28 October 2019

Stuart Russell is a leading AI researcher who co-authored the top textbook on the topic. He has also, for the last several years, been warning that his field has the potential to go catastrophically wrong. In a new book, Human Compatible, he explains how. AI systems, he notes, are evaluated by how good they are at achieving their objective: winning video games, writing humanlike text, solving puzzles. If they hit on a strategy that fits that objective, they will run with it, without explicit human instruction to do so.

Stuart Russell is a leading AI researcher who co-authored the top textbook on the topic. He has also, for the last several years, been warning that his field has the potential to go catastrophically wrong.

Kelsey Piper writes in Vox that in a new book, Human Compatible, he explains how. AI systems, he notes, are evaluated by how good they are at achieving their objective: winning video games, writing humanlike text, solving puzzles. If they hit on a strategy that fits that objective, they will run with it, without explicit human instruction to do so.

Piper writes:

But with this approach, we’ve set ourselves up for failure because the “objective” we’ve given the AI system is not the only thing we care about. Imagine a self-driving car with an “objective” to get from Point A to Point B but unaware that we also care about the survival of the passengers and of pedestrians along the way. Or a health care cost-saving system that discriminates against black patients because it anticipates that they’re less likely to seek the health care they need.

Humans care about a lot of things: fairness, law, democratic input, our safety and flourishing, our freedom. AI systems, Russell argues in Human Compatible, care about only whatever we’ve put in as their objective. And that means there’s a disaster on the horizon.

Piper met Russell at UC Berkeley, where he heads the Center for Human-Compatible AI, to talk about his book and about the risks posed by advanced artificial intelligence. Here are excerpts from their conversation:

Kelsey Piper: What’s the case that advanced AI could be dangerous for humanity?
Stuart Russell: To answer that question, we have to understand: how are AI systems designed? What do they do? And in the Standard Model [of AI systems] you build machinery, algorithms, and so on that are designed to achieve specific objectives that you put into the program.

So if it’s a chess program, you give it the goal of beating your opponent, of winning the game. If it’s a self-driving car, the passenger puts in the objective: [for instance,] I want to be at the airport.

So that all sounds fine. The problem comes when systems become more intelligent. If you put in the wrong objective, then the system pursuing it may take actions that you are extremely unhappy about.