CYBERSECURITYHow One of the World's Most Popular Open-Source Security Monitoring Platforms Was Developed

Published 23 February 2023

A tool from the internet’s early days keeps Microsoft’s users secure while supporting the open-source paradigm.

When Vern Paxson was a graduate student in Lawrence Berkeley National Laboratory’s (Berkeley Lab) Network Research Group in the 1990s, the term “cybersecurity” was not well known. But the software now known as Zeek, that Paxson developed at Berkeley Lab based on his general internet traffic research, has become one of the world’s most popular open-source security monitoring platforms. In October 2022 Microsoft Corporation announced Zeek’s integration into the Windows operating system, where it will help security teams have better visibility into their networks and respond more effectively to attacks. “This is an incredible tech transfer success for Berkeley Lab,” says Greg Bell, former director of Berkeley Lab’s Energy Sciences Network and Scientific Networking Division, as well as co-founder of Corelight, Inc., the company behind Zeek.

Most cybersecurity products focus on stopping malicious activity from entering a network or computer, by filtering the traffic with a firewall or blocking malicious files with antivirus software. Antivirus software, for example, “scans files entering your computer to see if they are malicious,” explains Jay Krous, head of cybersecurity at Berkeley Lab. “But if you don’t know that a file is malicious when it enters, you’ve missed your chance,” he adds. Zeek, in contrast, monitors network traffic and records and stores the traffic details in a condensed format. It does so without interfering with the network traffic, a requirement when moving the large data sets created by U.S. Department of Energy (DOE) science projects. Security teams can then use Zeek data to investigate potential attacks and understand what’s happening on the network, both in real-time and later in time.

Now seeking to bolster its own security systems with a robust and dynamic tool, Microsoft is adapting Zeek directly into an endpoint security product that ships on every version of Windows. And that represents a paradigm shift. Zeek has proved its worth for network watching, but individual client workstations, or endpoints, are equally susceptible to malicious activity. “The Zeek team realized cyber security professionals need to watch not just the network but also individual computers,” explains Krous. “If you have a version of Zeek that monitors inside the computer, and a version of Zeek that monitors the network, it allows more effective monitoring for malicious activity.”

Paxson says, “It’s incredible that this tool, which for most of its history has been strongly associated with making sense of network traffic, is now an endpoint tool.” Microsoft’s integration extends Zeek’s watchdog capabilities to a massive number of endpoints that are not on the corporate network. Moreover, Microsoft is contributing optimizations to Zeek – required so that the software can run efficiently on Windows – back to the open-source community. “Zeek was amazing 25 years ago and it’s still amazing today. It’s nice to see Microsoft recognizing the value in the approach Paxson created with Zeek,” says Krous.

Zeek’s Berkeley Lab Origins
Berkeley Lab’s unclassified research environment provided a unique setting where Zeek could evolve. The Lab’s high-performance and open network provided the opportunity to get visibility into attacks. And because of the Lab’s diverse science portfolio, network traffic from around the world enters the Lab network, where it can be recorded. When recording internet traffic for research purposes turned out to help with understanding attacks on the Lab, Paxson was inspired. He went on to develop a system custom-designed to analyze network activity to look for malicious behavior and produce a detailed record for future use.

In 1996, shortly after Paxson developed the software, Berkeley Lab put it into 24/7 production for in-house security use. But widespread deployment remained difficult. Because it was developed by and for expert users, Zeek at the time had no user-friendly interface and no documentation. With support from the International Computer Science Institute (ICSI), DOE, and the National Science Foundation, Paxson and his collaborators began to develop the tool for broader use. They disclosed the software to Berkeley Lab’s Intellectual Property Office in 2005 after which the copyrighted software was generally released under an open-source software license. In 2013, ICSI provided support for the team to found the company that eventually became Corelight, Inc. “After a bunch of exploration, my cofounders identified the sweet spot: Zeek-in-a-box, with a number of custom additions for high performance and usability. It’s taken off like a rocket since then,” says Paxson.

Microsoft’s endpoint adoption marks a new way to address the cybersecurity problems associated with a global network of clients. Bell concludes, “Zeek has had an amazing journey over the years. It was created by a grad student working out of Building 46A. Over the years this software, and the data-centric perspective on security it represents, has become a global gold standard. This is an unlikely hero’s journey, and a terrific example of the broad, cultural impact of DOE science.”