Power-Outage Exercises Strengthen the Resilience of U.S. Bases
The key is to not limit its scope. All facilities and missions, especially those that are critical, should be included, and service members are tasked with working through issues. To make exercises even more useful as an evaluation of readiness, some are modified with scripted scenarios simulating real-world incidents. These scenarios might challenge personnel to handle a cyberattack to control systems, shutdown of a backup power plant, or a rocket launch during an outage.
“We can do all the tabletop exercises in the world, but when you actually pull the plug, the question is, what actually goes on?” former assistant secretary of defense for sustainment Robert McMahon said at a joint House Armed Services subcommittee hearing about initial exercises. “Perhaps the most important lesson that I’ve seen is a lack of appreciation and understanding by our senior leaders at the installation level, all the way up to my level, of what we thought was going to happen versus what actually occurred, and then being able to apply those lessons learned.”
Illuminating Issues
The ERREs have brought to light common issues across bases. One of them is a reliance on fragile or faulty backup systems. For example, electronic equipment experiences a hard shutdown if it isn’t supported by a backup battery to bridge power transitions. In some instances, these battery systems failed or unexpectedly depleted due to age or generator issues. “We see a giant comms room drop out, and then phones and computers don’t work. It emphasizes the need for redundancies,” Lashway says.
Generators also present issues. Some fail because they aren’t regularly serviced or refueled through the long outage. Sometimes, personnel mistakenly assumed a generator would support their entire building, requiring reconfigurations after the fact. Air conditioning systems are often excluded from generator-supported emergency circuits, but rooms with a large number of computers generate a lot of heat, and overheated equipment quickly shuts down.
The exercises also unveiled interdependencies and chain reactions. In one case, a fire-suppression system accidentally went off, dousing a hangar in foam. The cause was a pressure drop at the same exact moment a switch reset.
“Executing an operation at this scale stresses how each of these factors need to work harmoniously and efficiently to ensure that the base, and ultimately missions, remain functional,” Lashway says.
Beyond resolving technical issues, the exercises have been valuable for practicing coordination and following chains of command. They’ve also revealed social challenges of operating through outages. For instance, some DoD guidance restricts the use of generators at daycare centers, so parents needed to coordinate care while maintaining their mission.
After an exercise, the laboratory compiles all findings in a report for the base. It provides time stamps of significant events by building, identifies links between issues, and summarizes common problems site-wide. It then provides recommendations to address vulnerabilities. “Our goal is to provide as much justification as possible for the base to get the resources they need to fix a problem,” Sack says.
The researchers also want to help bases prevent issues and avoid costly repairs. Recently, they’ve been using power meters to capture electrical data before, during, and after an exercise. These monitoring tools reveal power-quality issues that are otherwise hidden.
“Not all power is created equal, and standards must be followed to ensure equipment, especially specialized military equipment, operates properly and doesn’t get damaged over the long term. Power metering provides a view into that,” says Lashway.
Sparking Resiliency Ahead
Lincoln Laboratory’s ERRE campaign has resulted in legislation. In 2021, Congress passed a law requiring each military branch to perform at least five ERREs, or “Black Start Exercises,” per year through 2027. That law was recently reauthorized until 2032. The team has transitioned the ERRE process to two private companies, as well as within the Air Force and Army, to conduct exercises in the coming years.
“It’s very exciting that this got Congress’ attention and has scaled across the DoD,” says Nick Judson, who leads the portfolio of energy, water, and natural hazard resilience efforts within the Energy Systems Group. “This idea started out as a way to enable change on DoD installations, and included a lot of difficult conversations about turning the power off to critical missions, and now we’re seeing significant improvements to the readiness of bases and their missions.”
It may even be encouraging some healthy competition across the services, Lashway says. At a recent regional event in Colorado, three U.S. Space Force installations each vied to push the scope and duration of their exercises.
The team’s focus is now turning to related analysis, such as water resiliency. Water and wastewater systems are vulnerable to disruptions beyond power outages, including equipment failure, sabotage, or water source depletion.
“We are conducting tabletop exercises and workshops uniting stakeholders around the importance of water and wastewater systems to enable missions,” says Amelia Servi, who leads this work. “So far, we’ve seen great engagement from groups managing water systems who have been seeking funds to fix these aging systems, and from missions who have previously taken water for granted.”
They are also working on long-term energy planning, including ways for installations to be less dependent on the grid. One way is to install microgrids, which are self-sufficient systems that can tap into stored energy. According to Sack, microgrids are highly customized and complicated to operate, so one goal is to design a standardized system. The team’s recent power-metering data is providing useful initial inputs into such a design.
The researchers are also considering how this work could improve energy resiliency for civilians. Large-scale exercises might not be feasible for the public, but they could be conducted in areas important to public safety, or in places that rely on military resources. During one exercise in Georgia, city residents partially depended upon a base’s power plant, so that exercise included working with the city to ensure its resiliency to the outage.
“Striking that balance of testing readiness without causing harm is a big challenge in this field and a huge motivation for us,” Sack says. “We are encouraged by the outcomes. Our work is impacting the services at the highest level, rewriting infrastructure policy, and making sure people can better sustain operations during grid disruptions.”
Kylie Foy writes about science, technology, and community at MIT Lincoln Laboratory. This article is reprinted with permission of MIT News.
