Ensuring Human Control over AI-Infused Systems

3. Time frames of control accommodate situations in which machines have immediate control but people have longer-term control. Homeowners are happy to let the thermostat manage heating and cooling systems on a minute-to-minute basis to adjust temperature, but they want to control the hourly or daily temperatures to deal with weather or schedule changes, vacations, or visitors. They can also make seasonal adjustments, such as switching from heating to cooling in the summer.

Time frames of control are used in the large systems associated with NASA’s Mars Rovers. They are autonomous for immediate tasks such as navigating around obstacles while pointing the antenna and solar panels for optimal performance, but NASA ground controllers manage longer-term exploration destinations, deal with equipment failures, and take advantage of unexpected opportunities.

And in aircraft operations there are at least four time frames of control: (i) automatic systems control the engine functions every millisecond; (ii) airplane pilots control the plane on a minute-by-minute basis; (iii) ground controllers and airline managers make hour-by-hour decisions about speed, altitude, or changes to flight plans; and (iv) airline managers make week-by-week decisions about maintenance based on flight data recorder information on engine performance for each individual plane and aggregate performance information on all aircraft of the same design.

1.     Evolution of control recognizes that, as devices are refined to be more reliable, safe, and trustworthy, their adoption and use change. For example, the evolution of automobile manual shifting to automatic transmissions took several decades. Early automatic transmissions performed poorly but were refined to be generally more effective than people in terms of smooth operation, energy conservation, and engine performance, and eventually gained widespread acceptance. But even with automatic transmission drivers have ways to take control for steep hills or icy roads. Part of the maturation process is training people to use increased automation and making refinements to deal with conditions that require human control.

These four types are a starting point. Engineers and designers may know or develop other ways to give people control over issues they care about while using automation to accomplish tasks for which machines are well suited.

Self-driving cars—which I prefer to call safe-driving cars—are a familiar commercial product for which types of control and automation are being actively explored. The Society of Automotive Engineers (SAE) describes automation from Level 0, full human control, to Level 5, full self-driving. Some auto manufacturing proponents are so committed to full self-driving that they do not provide a steering wheel or brake pedal for drivers.

An effective, fine-grained design distinguishes the role and extent of human control and automation for the many tasks that go into driving. It applies to (at least) five key driving tasks (these are for illustration; there are dozens of tasks to consider):

·  Route selection and changes: this is mostly done by humans but automation can show route choices

·  Emergency vehicles: their presence on the road requires full human control as they operate in unpredictable circumstances and may require the driver to follow hand signals or voice commands from police

·  Parking: this can be highly automated

·  Speed control: this is selected by humans, who may use cruise control to ensure that they maintain a steady and safe speed

·  Lane following: in some circumstances this could be fully automated to keep cars in a fixed lane or remain with a convoy of cars in a lane.

Current advanced driver-assistance systems (ADAS) provide increased automation for a growing list of tasks. These systems enable drivers to select the degree of control for many tasks, including collision avoidance and parking assistance. Drivers select their seat position, mirror positions, and steering wheel height. Drivers or passengers can also set some features such as interior temperature controls and then the car maintains that temperature automatically.

The main message is that in complex tasks, designers would be wise to consider requirements under different conditions which call for different degrees of control. Rather than aiming for full automation, it is far safer to allow human decision making over the degree of control for specific tasks (Shneiderman 2022). Tasks for which computers can be reliable, safe, and trustworthy should be automated, with the following caveats:

·  There is a danger of excessive automation, which can have deadly results, as was apparent with the Boeing 737 MAX, which produced two crashes that killed 346 people, thereby raising global awareness of the risks of excessive automation.

·  It should be possible for humans to override the machine; existing examples include the option for firefighters and furniture movers to temporarily control elevators, and camera users can change the autofocus to favor a specific item. In self-driving cars, drivers may need to override control features in circumstances such as bad weather, road obstructions, or emergencies.

·  There is a danger of excessive human control—there may be occasions for automation to prevent drivers from engaging in unsafe behaviors, such as driving at excessive speed or following a car too closely.

·  Human fatigue, distractions, and complacency can lead to loss of vigilance that is necessary in life-critical applications. AI strategies for engaging, alerting, and providing immediate feedback for drivers, equipment operators, and medical personnel about their performance can increase vigilance.

Real-world designers are often confronted with difficult trade-offs. For example, should cars be programmed to prevent driving more than 20 miles per hour above the speed limit? This would save many lives, but there are special cases like the parent who is racing to bring an injured child to the hospital. Would you buy a car that limited you from driving more than 20 mph over the speed limit?

Design of commercial products, especially with life-critical implications, is complicated and requires courage and creativity to resolve the difficult trade-offs around control. Design and life are imperfect, but effective engineers and designers understand the trade-offs and make careful choices that bring benefits to users of their products and services.

References

Fischer G. 2017. Exploring design trade-offs for quality of life in human-centered design. Interactions 25(1):26–33.

Schmidt A, Giannotti F, Mackay W, Shneiderman B, Väänänen K. 2021. Artificial intelligence for humankind: A panel on how to create truly interactive and human-centered AI for the benefit of individuals and society. In IFIP Conference on Human-Computer Interaction (pp. 335–39). Cham: Springer.

Shneiderman B. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction 36(6):495–504.

Shneiderman B. 2022. Human-Centered AI. Oxford: Oxford University Press.

Ben Shneiderman (NAE) is a distinguished university professor emeritus in computer science at the University of Maryland. The article was originally posted to the website of the National Academy of Science.