Summer FEATURE 2010
The hype surrounding System X was reaching a crescendo seven years ago when Virginia Tech's Srinidhi Varadarajan presented the keynote address at the O'Reilly Mac OS X Conference, a hotbed annual gathering in Santa Clara, Calif., of the world's leading information technology researchers. The room was packed. Everyone wanted to know about System X.
Varadarajan, then-assistant professor of computer science in the College of Engineering, delivered. Back in Blacksburg, in an office building at the university's Corporate Research Center, was X, a vastly high-tech supercomputer just assembled by Virginia Tech faculty, staff, and students, comprising 1,100 Apple PowerMac G5 computers right off the assembly line. It was rumored to rank among the largest supercomputers in the world.
The New York Times and scores of industry publications already were publishing stories, building up anticipation as if X were a summer blockbuster film. Varadarajan, now associate professor of computer science and director of Virginia Tech's Center for High-End Computing Systems, was not expecting this attention. The supercomputer was not even fully operational at the time of his talk, and it was unknown whether the system would work in its entirety. The System X team had ordered crates of G5s on an act of blind faith, well before the G5 models hit the market. The near month-long process of assembly and testing was painstaking. "We were running first-generation hardware. They had never been connected before like that," says Varadarajan.
Srinidhi Varadarajan, the architect of System X, has seen the supercomputer assist countless research projects.
X's first power output numbers were 800 gigaflops, roughly one-twentieth of the supercomputer's potential output. The numbers leapfrogged, crawled, and leapfrogged again. The day a New York Times reporter called, X was at 7.5 Teraflops. When the story saw print the next day, X was at 8.5. Then it jumped to 9.5 teraflops for a week while Varadarajan was at the OS X Conference. A little more tinkering, pushing and problem-solving pushed the supercomputer to a then-resounding 10.3 teraflops.
"We were racing against time to submit the benchmark timing results in time for the fall 2003 release of the TOP500 list," says Cal Ribbens, associate professor of computer science and one of many original masterminds behind X, along with Erv Blythe, vice president for information technology; Hassan Aref, then dean of the College of Engineering; and Glenda Scales, associate dean of information technology for the college; and others. "And we were dealing with leading-edge technology--both hardware and software. So the system was not very stable at first. There were a few hardware failures. There were many software glitches."
The fallout would have been great if benchmarks were not met and the system failed to coalesce. "It would have been disastrous," Varadarajan says.
But it wasn't. Varadarajan's presentation was answered with thunderous applause. "I'm not used to getting a standing ovation at the end of a talk," said Varadarajan. But he got it. Adds Ribbens: "He was treated like a rock star." A rock star in a polo shirt and Khakis.
The intensive work paid off. In fall 2003, X landed at No. 3 on the TOP500, an international tabulation that has benchmarked the world's top high-performance computing systems since 1993. The machine achieved a sustained rate of 10.28 teraflops on the standard TOP500 benchmark calculation, ranking it behind only supercomputers in Japan and at the Los Alamos National Laboratory, both of which cost more than 10 times as much as System X. With a 2004 update, System X began running at a sustained rate of 12.25 teraflops, with a peak speed of 20.24 teraflops. "We had the fastest supercomputer machine in academia," says Varadarajan.
FINDING GENES AND SOLVING UNKNOWN PUZZLES
Wu Feng, an associate professor with Virginia Tech's computer science and electrical and computer engineering departments, has used X on numerous occasions. A 2005 study, just prior to his arrival at Virginia Tech, involved Feng and several other researchers focusing on parallel genomic sequencing. The findings resulted in a best paper nomination at the 2006 International Conference on High-Performance Computing Networking, Storage and Analysis.
Feng continues research into that field, having recently participated in a massive study with the Virginia Bioinformatics Institute that used X and six smaller supercomputers to locate genes in the genomes of microbes that were previously missed by gene annotation programs created by scientists.
"We gathered the sequences of 780 microbial genomes, searching these sequences against each other," Feng says of his work. "This part took roughly 12,000 processor cores about 10 to 11 hours to complete. If a DNA segment is similar to the sequence of known genes, then the segment is likely to be a coding gene with a similar function."
The work by Feng was akin to looking for a single page within a comic book series, but with an index that is missing page references to that particular page. The "missing" page is in the collection, but there is no way to readily locate it. "Instead we had to thumb through by hand and find the pages that we wanted. X, using this software program, found the genes that were known, but missing, hidden among so many like ones," Feng says.
Alexey Onufriev, associate professor in the departments of computer science in the College of Engineering and of physics in the College of Science, was one of X's earliest users and remains a strong advocate. He used X to crack a decades-old riddle that befuddled scientists: tracking the routes oxygen uses as it moves in and out of myoglobin, the oxygen-binding protein found in the muscle tissue that has been the guinea pig of molecular science for the past 50 years. It is the protein that gives fresh meat its nice red color in your local supermarket, and principles that govern its function often are applicable to other, more complex proteins.
In separate work, Onufriev used X to explore the full range of motions of a long DNA strand--long enough to form the fundamental unit of DNA packed in a single living cell, the nucleosome. The computer simulation, using only 10 percent of X's total heft, scuttled the long-held belief that DNA is a rigid building block of life, like a Lincoln Log. Instead, the computational experiment showed DNA is much more flexible than previously thought, Onufriev says. These findings may go a long way in helping scientists track the molecular makeup of human beings.
"The 'pieces' are chemical reactions, and we used System X to try out different ways to put together these reactions to form a biochemical switch," says Ramakrishnan. "It's actually a little bit more complicated because some of the pieces can be used multiple times in completing the jigsaw, some pieces need not be used, and each piece has many orientations. So there are millions of possible combinations, which is why we need a supercomputer to sift through all of them."
Paul, the associate professor of mechanical engineering, was attracted to Virginia Tech because of System X. He also conducts research under the flag of the College of Science's physics department, where he has used the supercomputer to conduct large-scale computational simulations to better understand the chaotic dynamics of the atmosphere and oceans. "We are exploring fundamental aspects of the chaotic dynamics that result when large systems are driven far from equilibrium," he says. "Examples include the weather and climate, the convection of biological organisms in the oceans, the efficiency of combustion and chemical reactions in complex flow fields, and fluid turbulence."
THE FAST TRACKWith a committee of just three members--Ribbens; Paul; and Daniel Crawford, professor of chemistry in the College of Science--approval time for using System X for a project is one or two days; and once an allocation of System X time is made, researchers on a given project can submit jobs to the machine with virtually no waiting time at all. Requests to use supercomputer resources at other installations can take anywhere from three months to a year--"and three months is incredibly lucky," says Varadarajan--while jobs often sit in a queue waiting to run for hours or days. The committee reviews 25-30 projects a year, declining only a handful of projects that don't justify the supercomputer's resources, Ribbens says.
From there, researches are given a project account. They log in from wherever they are in the world at an appointed time, enter and then run their data codes, and wait for the results. The amount of computing power and the complexity of the work determines the return rate, so a project involving the ignition of gases within an engine could take only a couple hours. But creating algorithms tied to, say, the formation of wildly infinitely complex protein or cells can take months, if not longer.
Virginia Tech researchers, including Onufriev, appreciate the fast approval and work flow. "I could have applied for time at national supercomputer centers. Problem is, the applications take a while, and then you have to stand in a queue to run your jobs," he says. "With the myoglobin project, it was really important to quickly test out many factors, so the on-demand access to fairly large chunks of X was important. I would have given up on the project if I did not have access to X, simply because it would have taken me years to do this work with ordinary computing power."
Multiple experiments and research projects can be handled simultaneously on X, as it is rare for any one project to consume the full blast of X's regular 12.25-teraflop computing power. "Many things can run at the same time, in parallel, like cars on a highway," says Crawford, one of the early consultants on X as it was being planned and built. "As scientists, we knew we'd be getting a lot of use out of it."
X's FUTURESupercomputer years make dog years look easy. Built in 2003, System X is ancient for its species. Among typical PCs, every 18 months computer processors can run twice as fast for roughly the same cost. For supercomputers the process is more drastic: performance increases by 50 percent every six months.
"We are looking for a successor," says Varadarajan. But the process may be grueling. Including upgrades, X has cost $6.5 million. A supercomputer now, to be competitive, would cost upward of $30 million to build and would need to run 1,000 times faster at peak power. As of publication, the No. 1 computer on the TOP500 list is the "Jaguar," peaking at 2,331 teraflops.
X and other supercomputers, now and in the future, will be vital tools for researchers such as Feng, Onufriev and Ramakrishnan. But they still are tools. They need cutting-edge users to ask the important questions.
"A supercomputer is a fast tool. It won't lead to discoveries, but it can help scientists and researchers discover things faster," says Feng. "One important thing to remember is a supercomputer is only as smart as its users. It does what it is told. That's it."
X is by no means ready for retirement. In the hands of Virginia Tech's world-class researchers, the supercomputer will continue to pave the way for groundbreaking research that will change the world.