RoboCup 2018: Testing methods used to evaluate rescue robots

“This year’s challenge tasks used the same standard test methods as previous years, but had more difficult apparatus settings toward the final missions,” said Adam Jacoff of NIST, who has chaired RoboCup Rescue since 2001, “That’s how we use the test methods to push the state-of-the-science while maintaining the ability to measure and compare system capabilities in different locations at different times.”

What does this robot do?
The original test methods were developed to provide the Federal Emergency Management Agency (FEMA) and first responders a reference point for selecting robots to carry out different missions. The methods helped FEMA make better purchasing decisions, as they could validate effectiveness and distinguish robots that did their jobs well.

“We wanted to develop standards the whole community agreed upon, especially first responders who will be using them to disarm bombs, go into fires, etc.,” said Kai-Dee Chu, S&T Program Manager. “If the claim is that a robot can carry a victim out of a fire or inspect luggage aboard a plane, we design an environment to test it.”

Without standards and benchmarks for developing and testing machines, there are no clearly-defined rules by which the robots can be compared.

Before S&T and NIST standards were developed for public safety robots, it was more difficult to distinguish one robot from another. The design of a soccer-playing robot, for instance, can be guided by the rules of the game, but there were no preexisting ways to define excellence for rescue robots. Today, instead of each robot being its own unique set of wires, wheels, belts and bolts, the robots can be categorized and given specific requirements for tasks that support activities such as fire rescue or handling explosives.

Not long after the test methods were developed, FEMA realized some robots could only be used by well-trained operators, so different apparatuses were adopted as training grounds for first responders. Since then, various public and private partners have found them useful—the North Atlantic Treaty Organization has also adopted these methods for training.

The standard test methods have been received by different public and private representatives at the RoboCup and beyond. Their adoption continues to expand.

“If you’re in a major city, you can probably find someone employing our test methods,” said Chu.

The big leagues
S&T notes that now that the 2018 RoboCup has passed, the S&T- and NIST-developed apparatuses created by local Canadian sponsors and organizers will head to Canada’s Police College in Ontario and the Canadian Explosive Technicians Association. While many organizations have adopted some of the standards or housed the apparatuses after RoboCup events in past years, this is the first nationwide adoption of robot standards.

“They will set minimum acceptable levels of proficiency for bomb technicians across Canada,” said Jacoff, “like getting a driver’s license for emergency response robots. The test methods focus training and measure basic maneuvering and dexterity along with more operationally relevant tasks such as accessing buildings.”  

Last year’s apparatuses, from RoboCup Rescue 2017 in Nagoya, Japan, were donated to the Fukushima Test Field, set to open in 2020 for first responder training.

“The widespread adoption of robot standards demonstrates how innovation is not necessarily always in the hardware. The hardware, in this case, needed purpose. S&T and NIST provided it, and as a result, various law enforcement agencies are given unprecedented capabilities to help them in their missions,” S&T says.