Shake and Bake, Conduct a Bake-off

This article was originally published in September 2008 at

Have you ever held a Bake-Off to select a core technology for a project? Not an RFI, but an actual honest to god series of “real world” tests. Few things are as exciting as setting up a technology obstacle course that is somewhat indicative of what your business environment is like then having various vendors run through it. Several times in my past I’ve conducted these when emerging technologies like server UPS systems and VOIP telephony were new in order to shake out the posers from the players, evaluate “real-world” performance then determine value.
Few vendors post actual price and performance data on the web, let alone the methodology they used to arrive at those performance numbers. If only there were an independent third party that actually ran Netperf, Iperf, ntttcps, ntttcpr and other tools on all the available 10GbE NICs using the same test systems then posted the results for everyone to see. Some companies would never recover. For legal reasons, the vendors won’t, and in most cases do not want to, do it because the results would only help one or two companies and likely not theirs. Today all most consumers have to go on is the cost of the adapter, wouldn’t it be great if you knew the cost/Mbps of the adapter prior to buying it so you could easily compare between adapters. Some would argue that features like iWARP and TOE should be factored in, but today they are just marketing fluff and rarely deliver any significant end user value.
So how do you determine which NIC will perform the best and deliver the most value for your company, do a bake-off! If you can make the time and the project is big enough the cost to conduct the back-off should easily be offset by the savings, education, and performance gains you reap over time. Also, a well constructed and executed bake-off will demonstrate not only to you but your management, that you’re an effective individual and a good steward of the companies resources.
Finally, share the full set of results with the vendors that participated, some will moan and groan, while others will kindly thank you for the opportunity to compete and move on. If the race was close their reactions at this point might be your deciding factor. So pull on your oven mitts and start baking…

Leave a Reply