Safer cyberspace through experimental cybersecurity research

“Experimentation is an inherent part of the scientific method and you can’t do research without doing experimentation,” said Douglas Maughan, Director of the Cyber Security Division at the Department of Homeland Security, Science and Technology Directorate. “This report is a critical first step to re-think what is needed in cyber experimentation before we build the infrastructure.”

Using the scientific method also requires peer review and repeatability. The report emphasized the need for infrastructure that supports and enables repeatable experiments by creating easy ways for researchers to test each other’s results.

Moreover, instead of uncoordinated, domain-specific studies — some related to denial of service attacks or password cracking, others related to critical infrastructure or automotive testing — researcher need common standards and ways to work across disciplines and domains.

“The adversary isn’t looking narrowly,” Benzel said, “and researchers can’t afford to either.”

Finally, the community needs to develop new approaches for sharing and synthesizing data in order to accelerate knowledge and community building across disciplines and organizations.

“We need a way that makes it easy for researchers, not only from different aspects of cybersecurity, but across different domains, to share their problems and draw from a library of experimental cyber components to put together a big problem,” Benzel said.

Recommendations for securing our cyber-future
Based on input from scholars, the authors synthesized five key observations which they believe, if followed, will yield transformational results.

First, research must be multidisciplinary. Whereas today, experts typically specialize in one area, in the future, individuals and teams must incorporate a wider range of knowledge and skills.

“We need to bring in different disciplines, from computer science, engineering, math and modeling to human behavior, sociology, economics and education,” said David Balenson, another of the lead authors and a senior computer scientist at SRI International.

Second, experiments must accurately model and incorporate human activity.

“Everything we do needs to be grounded in the real world and include the human element — users, operators, maintainers, developers and even the adversary,” Balenson said.

NSF Program Director Anita Nikolich said performing cybersecurity research “in an isolated, contained environment that doesn’t mimic reality is not conducive to discovering the nuances inherent in this sort of research. New approaches to testing are needed in order to produce useful, actionable results.”

Third, different experimental environments must be able to work together in a plug-and-play fashion by following common models of infrastructure and experiment components using open interfaces and standards.

“Without shared experimental infrastructure, researchers have to spend lots of money developing their own experimental infrastructure which takes away from their core research,” said Laura Tinnel, a senior research engineer at SRI International and one of the study’s authors. “People are reinventing the wheel.”

Fourth, experimental frameworks must allow reusable designs to better enable science-based hypothesis testing.

“In most other sciences, someone can come and repeat your experiment, but that’s not typically the case in cyber,” Benzel said. Hardwiring such capabilities into the structure of the experimental framework would allow researchers to do broader experiments, and also lower the barrier to entry and improve education and training.

Finally, any infrastructure that is built must be useable and intuitive, so researchers and students spend less time learning to use the infrastructure and more time doing critical scientific inquiry. Moreover, the community must adopt a more rigorous scientific model for research and supporting infrastructure.

“People have been doing things the same way for some time now, and trying to get them to work in a more community-oriented way is going to take some shifts in their thinking as well as cultural changes,” Balenson said.

NSF notes that the study’s authors believe that if the scientific community follows the recommendations, such a shift would not only change the balance of power between hackers and cybersecurity experts, but result in systems that are secure by design — something that long-discussed in the cybersecurity world but not yet successfully implemented.

“We can shift this asymmetric cyberspace context to one of greater planning, preparedness, anticipation and higher assurance solutions,” Benzel said.