Its not all about search: Yahoo! and cloud computing

tata_logo_03_3 News Yahoo! announced today a new collaboration agreement with Computational Research Laboratories (CRL), which is owned by India’s Tata Sons Limited, to utilize Tata’s giant supercomputer, the fourth fastest in the world, to “leverage CRL’s expertise in high performance computing and Yahoo!’s technical leadership in Apache Hadoop, an open source distributed computing project of the Apache Software Foundation, to enable scientists to perform data-intensive computing research on a 14,400 processor supercomputer”.  First, a bit about the computer:

Called the EKA, CRL’s supercomputer is ranked the fourth fastest supercomputer in the world – it has 14,400 processors, 28 terabytes of memory, 140 terabytes of disks, a peak performance of 180 trillion calculations per second (180 teraflops), and sustained computation capacity of 120 teraflops for the LINPACK benchmark. Of the top ten supercomputers in the world, EKA is the only supercomputer funded by the private sector and is available for use on commercial terms. EKA is expected to run the latest version of Hadoop and other state-of-the-art, Yahoo!-supported, open-source distributed computing software such as the Pig parallel programming language developed by Yahoo! Research.

 While Microsoft is trying to portray itself as “the little engine that could”, far behind in search and needing a Yahoo! acquisition to have a chance at competing with Google, Yahoo! has been investing heavily into research and development in cloud computing, an area Microsoft is known to be heavily involved in as well.  Yahoo!, while it is using open source tools like Hadoop and not Microsoft technology, still has significant assets in cloud computing research that would be highly beneficial to Microsoft. 

“We have made our leadership in supporting academic, cloud computing research very concrete by sharing a 4,000-processor supercomputer with computer scientists at Carnegie Mellon University for the last three months. With this supercomputing cluster, researchers were able to analyze hundreds of millions of Web documents and handle two orders of magnitude more data than they previous could,” said Ron Brachman, vice president and head of academic relations for Yahoo!. “Launching our cloud computing program internationally with CRL is another significant milestone in creating a global, collaborative research community working to advance the new sciences of the Internet.”

So while Steve Ballmer jumps around onstage and plays up Microsoft the underdog in search, the move to buy Yahoo! may well bring benefits down the road that aren’t readily noticed by those intent on watching a Google/Microsoft battle.