How NSA processes all that intelligence data

Other

January 28, 2016

Ray Alderman

VITA Standards Organization

EVOLUTION OF WARFARE Blog: We’ve been through examples of how the NSA and the military intelligence groups intercept and collect massive amounts of information on our enemies in previous articles here. All that collected data must be decrypted and processed into usable intelligence, then analyzed and distributed, which is where computing comes in from pencil and paper to quantum computers.

Before computers

The first serious intelligence group, the 1919 “Black Chamber” run by Herbert Yardley, used pencils, paper, chalk boards, and the brains of a few smart people to break codes and analyze intelligence data. Shortly before World War II (WWII) began, military intelligence groups were working on breaking the German Enigma machines. In 1932, the Polish Cypher Bureau figured it out and handed over their work to the British at Bletchley Park in England.

The German Enigma machines were basically an elect-mechanical typewriter, and used three encryption wheels called rotors (later machines used four), along with a plug board to add another level of complexity to their code. The Enigma code was basically a series of S-boxes (Substitution boxes) and P-boxes (Permutation boxes), but the possible combinations were in the millions and millions. The Polish Cypher Bureau built a electro-mechanical machine that simulated the S and P-box sequence possibilities and decrypted some messages sent in Morse Code by the Germans. The British built similar but larger machines. American Intelligence built even larger simulators that took up entire rooms.

The breaking of the WWII Japanese Purple code followed a similar path. Their machine was the size of two typewriters and used four stepping switches, similar to the rotors on the Enigma machines, also with a plug board. American Intelligence broke the code in early 1942 using the ECM Mark III, another simulator machine, with commercial punched card readers and tabulators for input/output.

As you can see, very large and complex electro-mechanical simulators had to be built for each code, or for any major change in the code (i.e., additional encrypting rotors). That was both expensive and time consuming. These simulators were called RAMs (Rapid Analytical Machines) and were built for the intelligence services by Eastman Kodak, National Cash Register, Bell Labs, and IBM.

Computers come on the scene

In 1950, Engineering Research Associates delivered NSA’s first computer, named Atlas. A second one was delivered in 1953. These were vacuum-tube based computers. In 1952, the Army Security Agency (my old military intelligence unit) designed and built their own computer, named Abner. Army Intelligence had the most powerful computer in the world for many years.

In 1956, NSA decided to initiate Project Lightning, to build their next generation computer system named Harvest. It would use transistors, not vacuum tubes. The contractors used to build it were Sperry-Rand, RCA, IBM, Philco, and General Electric, along with assistance from MIT, University of Kansas, and Ohio State engineering departments. In 1952, IBM delivered the cornerstone processor of the Harvest computer project, named Stretch. The remainder of Harvest consisted of Stream Processing Units and input/output devices (card readers, and printers). With the addition of tape drives and disks, Harvest remained operational until 1976.

NSA began to buy and install commercial mainframe computers after Harvest. They consisted of IBM 360s, 370s, and 3033s. In 1976, NSA took delivery of the first Cray-1 Supercomputer. Cray X-MP’s were delivered in 1983. The liquid-cooled Cray-2 was delivered in 1985. In 1988, the Cray Y-MP supercomputer was delivered. Then, some Ncube massively-parallel machines were delivered with 1024 processors in each one. That was followed by some 8-processor Control Data ETA-10 machines. Along the way, NSA picked up some NEC SX-4 supercomputers. And, Thinking Machines delivered their massively parallel CM-5 in 1991. IBM RS-6000 SP systems arrived in 1999.

In 1996, NSA moved over 150 of the largest and fastest supercomputers in the world into the Tordella Supercomputer Facility at Fort Meade, Maryland. NSA does not count the number of monster supercomputers they have. They measure them by “acres." They had 5 1/2 acres of them in 70s, so they must have much more than that today.

Earlier, in 1984, NSA had created a Supercomputer Research Center, to manage all their computers and their algorithms. But after managing them for a few years, what SRC decided was to build supercomputers themselves, just like they did with RAMs. The commercial computer makers no longer had any interest in building monster machines, except IBM with Blue Gene and now Watson. Consequently, NSA has their own semiconductor fab on their campus at Ft Meade. They can build their own processors and crypto chips based on their success with certain software algorithms that have been proven on their supercomputers. They can take their mature software algorithms, put them into hardware, and make them run many times faster.

Quantum computing

What is NSA doing now? Well, they have been funding research and development of quantum computers since the 90s. Typical register-oriented sequential-execution processors are not good at code breaking, even with many of them working on the problem simultaneously. They are bounded by Amdahl’s Law, which says that parallel computing with traditional processors exhibits diminishing returns for each additional processor after eight or ten (depending on the characteristics of the code). Quantum computers are completely different animals. You’ll have to read about Quantum Entanglement and Quantum Superposition on the web. I don’t have the space to explain how they work here, nor do I understand it well enough to try. There are some good tutorials on the web, if you’re interested.

A quantum computer could break DES-encrypted codes in about five minutes or less. To factorize a 200-digit number with today’s supercomputers takes millions of years (this 200-digit number is the product of two large prime numbers, which are the keys to the encryption algorithm). Quantum computers could find those two large prime numbers in an hour or less. Brute-force code breaking, character frequency analysis, character proximity analysis, word frequency analysis, key discovery, and encryption algorithm techniques (logarithmic and elliptical code schemes) are perfect applications for quantum computers.

Google and NASA just ordered quantum computers from Dwave Systems -- their new 1,000 Qubit machines. So, you can bet that NSA has some Dwave machines among their acres of supercomputers. Or, maybe they have built their own. NSA’s stated objective in computing is to be five to 10 years ahead of the state of the art, not just ahead of what the computer makers are working on. Just imagine what could be inside the new NSA data center in Bluffdale, Utah. If you want to read more detail about the history of NSA’s computing capabilities, read chapter 14 of “Body of Secrets” by James Bamford.

At this point, we’ve broached the messy subject of cryptology, cryptography, and cryptanalysis, which was the topic of the previous installment. TO read it, click here. Early codes were more like interesting puzzles. Intermediate codes were driven by S-boxes and P-boxes: much more complex. Modern codes are driven by massive formulas and complex algorithms created by math and physics PhD's. I promise that I made it interesting and understandable for those who don't solve integral calculus problems and quadratic equations as a hobby.