As a Computer Science Student, I like the idea of being able to run my code on the biggest, baddest computers there are. Unfortunately, most of my code doesn’t need more than my laptop. In fact, many of the programs I write would run just about as well on my phone, or an Apple II. However, sometimes I do write programs that require gobs of raw computing power. Last night, I conceived an idea for just such a program. Before I even started programming, I realized that my only hope for finishing computation on the large dataset I wanted to process (all the links on Wikipedia, many times) was to spread the computation across multiple computers. I needed a cluster. So I built one.
A cluster (short for computing cluster) for those who don’t know, is basically a bunch of computers connected via a network. You could think of the Internet as a computing cluster, but the important difference between the Internet and a useful computing cluster is that a useful cluster works toward a unified goal. As you might have guessed, part of the beauty of a cluster is that the computers involved don’t have to be all that powerful by themselves to be useful. Because my family doesn’t usually get rid of computers (precisely for occasions such as this,) we had three old laptops lying around, all of which I confiscated for my cluster. Add my laptop to that, and I have a cluster of six cores. It’s not a whole lot, but it may just be enough. I also have an old switch (what most people would call a “router”) that I saved from the trash during high school, which I’m using to connect the computers together. After installing Debian Linux, and puppet to manage the configurations on all the machines, I’m almost ready to run cluster computations!
In order to run a cluster, you need a server to distribute the data you want to process, and clients to actually do the processing. In this case, I’m talking about the server and client software, not machines. So I have to write a series of programs that will split up my data and process it on each of the cluster computers. It’s simple enough when you use Java’s RMI (Remote Method Invocation.) The server maintains a collection of the processed data, which is updated whenever the clients finish processing. Once the client has finished its data, it sends back its results and requests more data.
This strategy has a lot of advantages, including the fact that I don’t have to store the entire Wikipedia database on more than one machine. Since the data is sent in chunks, I can store it all on one of the two computers in my cluster that can handle that much data. Also, the clients can work asynchronously. Since the computers don’t have to wait for each other before getting new data to work on, my laptop can crunch along at 2.53GHz per core, processing as much data as it wants, while the poor little AMD Athlon slogs through whatever data it can handle.
My cluster is called “End of the World,” or EotW for short. I was trying to think of a theme for the hostnames of the computers in the network, and I settled on character’s from Haruki Murakami’s novels. Since there’s a “place” in his book Hard Boiled Wonderland and the End of the World called “End of the World,” I figured that would be fitting. The nodes are named toru, watanabe, and sumire.
How much did all this cost? $20 for a power strip. Even if I didn’t have all the laptops, I could have waited until the morning, gone to the recycling center, and picked up a bunch of computers and a switch for free. That’s what I love about cluster computing: with just a little recycled hardware, you can create a pretty fast computer. It’s true, my tiny computing cluster barely rivals a new, mid-line desktop computer with an Intel Xeon processor, but it’s still faster than just my laptop. And hopefully, with all those computers working full tilt for a few days, I’ll be able to crunch through my data*.
*As I said before, I’ll discuss what I’m trying to do at another time. Trying to explain it here would make this post too long, and I want some tangible results before I start talking about it.