25/50/100GbE Facts

Several years ago the mega data center and Web 2.0 companies started looking for an alternative to the approved 40GbE standard which is the link speed offered between 10GbE, and 100GbE. They viewed the IEEE approved implementations of both 40G and 100G, which were simply multiple 10G lanes, as very cumbersome. These mega data centers seek to leverage High-Performance Computing (HPC) concepts and desire that their exascale networks utilize fabrics that scale in even multiples. Installing 25/50GbE at the servers, and (2x25G) 50GbE, or (4x25G) 100GbE for the switch to switch links is much more efficient. It turns out two groups formed in parallel: The 25 Gigabit Ethernet Consortium and the 2550100 Alliance (that’s 25, 50, 100 for those that didn’t see it) to develop & promote a single lane 25GbE specification as the next Ethernet. This approach would then be extended to two 25G lanes for 50G, and four 25G lanes for 100G. It should be noted that today the IEEE, the industry standards body, has not yet ratified a 25GbE standard (when it does it will be referred to as IEEE 802.3by). Once approved this standard will be used to create Ethernet Controller NIC silicon and compatible switch silicon. This work is underway, but won’t be completed until sometime in 2016.

The 25 Gigabit Ethernet Consortium was founded by Arista, Broadcom, Google, Mellanox & Microsoft. While the 2550100 Alliance is about fifty companies, and the ten most notable in this alliance are: Accton, Acer, Cavium, Finisar, Hitachi Data Systems, Huawei, Lenovo, NEC, Qlogic, and Xilinx. Interestingly absent from the above lists are key Ethernet product companies: Chelsio, Cisco, Emulex, HP, Intel, and Solarflare. The focus of this piece will be on the Ethernet NIC controller silicon, because if you can’t connect the server then the whole discussion is just switch to switch interconnects, which is a another class of problem for discussion in a different forum. Today there appears to be only two general purpose Ethernet controller NIC chips that support a version of 25/50/100GbE and they are Mellanox’s ConnectX-4 and QLogic’s cLOM8514. For NIC silicon to actually be useful though it must be delivered on a PCI Express adapter. At this point QLogic has only demonstrated their 25GbE publicly, but has not announced a date to ship a PCIe adapter. This means that Mellanox is the solitary vendor shipping a production 25/50/100GbE adapter today. QLogic has not formally stated when it will be producing adapters with the cLOM8514.

In the home Wifi networking market hardware vendors typically race ahead of IEEE standards, and produce products to secure “first mover advantage”, otherwise known to end users as the bleeding edge, but they can only do this because their products are for the most part stand-a-lone. Enterprise and data center markets are highly interconnected and shipping a product ahead of the approved IEEE specification is inviting an avalanche of support calls. Today there still remain significant open technical issues around 25/50/100GbE such as auto negotiation, link training, and forward error correction. The IEEE has yet to resolve these, but they are being discussed. At the end user level, interoperability is the key issue. If a company were to produce a stand-a-lone NIC product without an accompanying cable & switch ecosystem they would be flooded with support requests. The converse is also true, if a company were to build a switch around the Broadcom silicon without offering a bundled in server NIC it would quickly also become an interoperability situation. Those on the bleeding edge, would surely now understand the true meaning of the phrase.

So why haven’t the more traditional 10GbE NIC vendors jumped on the 25/50/100GbE band wagon? Simple, without an approved IEEE standard the likelihood of profiting from your investment in 25/50/100GbE is fairly low. Today, exclusive of R&D, producing an Ethernet controller NIC chip is a multi-million dollar exercise. So to justify spinning a 25/50/100GbE NIC chip in early 2015 for a “first mover advantage” one would require a plan that it would produce well into the tens of millions in revenue. Couple this with the interoperability support nightmare of getting one vendor’s NIC working with a second vendors cable, and third vendors switch, and any profit that might exist could quickly be consumed.

Enterprise customers want choice, which by definition implies multivendor interoperability based on mature standards. Once the IEEE 802.3by standard (25/50/100GbE) is ratified next year it is expected that all the NIC vendors will begin shipping 25/50/100GbE NIC products.

7/27 Update: Broadcom announced a 25/50GbE NIC controller today.