Managing the data deluge for national security analysts

Tracktable code automates observation of motion, trajectories
Mark Rintoul, a Sandia data scientist, developed the Tracktable code along with Sandia researcher Andy Wilson and others to automate the observation of motion and trajectories. The code could be applied to any problem that involves movement, such as airliners, ships or people.

Current approaches to getting meaningful information from trajectories focus on comparing one trajectory to another. If you have millions of trajectories to consider, that could mean trillions of comparisons, which takes a lot of time and computer power, Rintoul said.

“We’ve developed a way to store and represent trajectories so that computers can compare them all at once in a very fast and effective manner,” he said. Instead of trillions of comparisons, the software does the same job in millions of comparisons, which is manageable.

An analyst concerned about the number of airliners stuck in holding patterns could ask Tracktable about aircraft trajectories that made a certain pattern of turns. Tracktable then calculates geometric features, such as the number of 90-degree turns an aircraft flew or the length of a straight line. By associating a type of motion with these features and assigning a number to each feature, the computer can quickly group flights that behave in similar ways and show them to the viewer for interpretation.

“If you have millions and you’re not interested in precise comparisons, but general groupings of them, this is very effective,” Rintoul said.

PANTHER also examined the predictive capability of the information buried in data. If an analyst looks at the first half of a flight, considers historical data about similar flight paths and then looks at the second half of the flight, any deviation from the pattern might cue an analyst to take a closer look. Finding that outlier from millions of flights that have flown before takes about a second with Tracktable, Rintoul said. The analyst is alerted because PANTHER team members are using the advances in cognitive science to design visual results that will highlight the odd behavior of the single aircraft. By studying how analysts use visual data, Sandia researchers are figuring out ways to make an outlier pop out of a screen full of detail to demand an analyst’s attention.

The team is now looking at integrating motion and trajectories into a system called GeoGraphy.

GeoGraphy helps analysts search for items of interest, shows changes over time
GeoGraphy, initially funded by the National Nuclear Security Administration, is a software system that converts remote sensing images expressed in pixels into nodes and edges in a graph to show changes over time and make the data searchable, said Randy Brost, a Sandia computer scientist who led the team that developed the software. Nodes are analogous to the beige hubs in Tinkertoys, while edges are the colored connecting rods.

GeoGraphy breaks the images into categories, such as buildings, trees or rivers. This pre-processing creates a graphic resembling a complex paint-by-number that shows the categories of everything in the image. The program uses nodes and edges to describe relationships between objects, such as distance or time, Brost and Czuchlewski said.

Sandia Lab notes that in addition to the imagery, the software package could include such information as phone books or county records, producing a single searchable database of all the information that shows what’s changed over time.

For example, to find a high school, the analyst tells the program to search for large buildings near regions that look like parking lots, football fields and tennis courts and defines those items. The analyst then can choose from among the results the computer provides.

The system is hierarchical, so once analysts identify high schools, they can ask the program to find high schools the next time without describing them. And should they doubt that something is a high school, the software makes the raw data available so they can verify the results, Brost said.

“The purpose of these codes — GeoGraphy and Tracktable — is to assist humans, not to replace them or to automatically do their jobs. It’s to enhance their ability to do their jobs well and to allow them to be more effective in dealing with large sets of evidence,” Brost said. “In the end, basically they are suggestion systems that say, ‘Hey, based on what you told me you’re interested in, you ought to look here, here and here.’”

The PANTHER team also included researchers focused on enhancing the viewer experience. Researcher Laura Matzen and others are conducting cognitive science experiments to learn how analysts’ expertise affects their visual cognition and to create a model of how top-down visual attention — when a user approaches an image with a goal in mind — works. The researchers hope to use the answers they find to such fundamental cognitive science questions to inform the design of new tools that will improve interactions between humans and computers, Matzen said.

The prototype products and ideas developed under PANTHER are ready for the next step in their development: to be tested in real-world environments, Czuchlewski said.

Sandia researchers have proposed research into new problems illuminated by PANTHER, while other agencies are solidifying the foundation PANTHER has developed. Other projects will use PANTHER’s ideas to address real-world problems, the researchers said.

“We went into PANTHER thinking we were going to do one thing, we’re going to improve the lives of image analysts,” Czuchlewski said. “And, in the research process, we did a whole lot more.”