Trend: Government demand for data-mining applications grows

Talk of searching for things which hide in plain sight: Since 9/11, U.S. intelligence agencies have invested millions of dollars on computer programs which search through financial, communications, travel, and other personal records of people in the United States and around the world for connections to terrorism. More recently, these agencies began to explore what information about terrorism could be gleaned from analyzing social networks. The software used in all these search activities is designed to find links between terrorism suspects and previously unknown people; track the international flow of money, operatives, and materials; and search for clues in the worldwide communications over phone lines, wireless connections, and Internet links.

Much of this activity is conducted with “black” budgets, so it is not possible to say precisely how much the U.S. intelligence community has spent to develop, purchase, and upgrade these type of crawlers and data-mining programs. From information which what is publicly available we can say that data-mining systems used by intelligence agencies include:

—Hardware and software from Dayton, Ohio-based Teradata, an NCR subsidiary; Teradata’s solutions are capable of storing and searching databases as large as four million gigabytes, or twice as much information as is held in all research libraries in the United States. All Teradata officials would say is that the company’s applications include searching financial transactions for signs of money laundering

—Draper, Utah-based Cogito has developed a program designed to identify members of terrorist networks and determine the most important members of these networks. The company sold the program to the National Security Agency (NSA) and other intelligence agencies

—The Defense Intelligence Agency (DIA) and DHS used software from Sunnyvale, California-based Verity to collect personally identifiable information about Americans from other agencies and commercial sources (Note: In December 2005 Verity was acquired by Cambridge, U.K.-based Autonomy; Autonomy’s motto: “Understanding the Hidder”)

In January 2002 the Defense Advanced Research Projects Agency (DARPA), the research and development agency of the U.S. Department of Defense, created the Orwellian-sounding Information Awareness Office (IAO) to bring together several DARPA projects focusing on applying information technology to counter threats to national security. The IAO mission, in the organization’s own words, was to “imagine, develop, apply, integrate, demonstrate and transition information technologies, components and prototype, closed-loop, information systems that will counter asymmetric threats by achieving total information awareness.” The IAO came under criticism from the start. Congress, fearing that the development and deployment of IAO technologies and programs [we will detail these programs in Monday’s issue] could lead to a Big Brother-like mass surveillance system, cut funding to the IAO in 2003 and the organizations was abolished. The IAO did not benefit from the fact that the administration appointed Admiral John Poindexter of Iran-Contra fame to head it, or from criticism of the organization’s seal (see below) as more befitting the Free Masons or similar illuminati organizations.

The TIA was abolished, but five data-mining programs developed under the organization’s auspices are among at least eight TIA projects which have continued under the supervision of the departments of defense, homeland security, and justice. These programs include four efforts to create software that searches through mountains of data for evidence of terrorists and three projects which allow intelligence analysts from different agencies to collaborate on computer networks. A contract to pull all of the new software together into a working system also remained active until at least last year, government records show.