This article was originally published in May of 2011 at 10GbE.net.
Today for the umpteenth time I had to explain to someone that if you go optical to connect your server to your switch with 10GbE it could easily cost you twice as much. There is a secret at the end of this entry that MIGHT allow you to save some big time cash if you have enough muscle, but you have to read to the end of this entry.
For cable runs of seven meters or less you should always use Direct Attach (DA otherwise known as Twinax) cable if possible as it could easily save enough to basically connect the second server for free! Here are some actual numbers from earlier today.
First, some basic end user costs assuming a five-meter run, note these are rounded a little bit to keep the math simple:
10GbE Network adapters, roughly $400/port
10GbE Switches, roughly $500/port
10GbE SR SFP+ Optics from switch vendor $800/port
10GbE SR SFP+ Optics from NIC vendor $200/port
10GbE SR Optical 5M cable $80/ea
10GbE Direct Attach 5M cable roughly $140/ea
Now let’s build a solution between the server and the switch using optics:
10GbE Network adapter $400
10GbE SR Optic from NIC vendor $200
10GbE SR Optical 5M cable $80
10GbE SR Optic from Switch vendor $800
10GbE Switch port $500
Total $1,980 to connect a single server
Direct Attach (Twinax) Option:
10GbE Network adapter $400
10GbE Direct Attach 5M cable roughly $140/ea
10GbE Switch port $500
Total $1,040 to connect a single server
Let’s look more closely at the market dynamics going on here. First, only a handful of companies make 80% of the 10GbE Short Range (SR) optics that everyone uses today. These companies are typical: JDSU, Finisar, Agilent, etc… None of the switch companies or NIC companies make their own optics, we all source them from several of the above companies, and a few others, all of whom rebrand them for us and burn our company name and part number into what is essentially flash memory within the optic.
Here’s where it gets interesting. Myricom, the company I work for, sells it’s SR SFP+ optics online via CDW’s website for $185. Here are some of the more expensive SR SFP+ optics listed on CDW’s site:
HP Procurve: $1,498
Avaya: $1,350
Enterasys: $1,210
Cisco: $1,100
Juniper: $1,082
Brocade: $1,022
QLogic: $930
IBM: $920
Now remember under the covers we’re all sourcing these optics from the same competitive pool, so why the price spread?
First, remember that we each buy our optics with our manufacturer name and part numbers already burned into them by the optics makers mentioned above. Now here’s where it gets interesting the switch makers during switch initialization query the optic and if it does not return a valid company name and part number then it locks the optic out and reports the port as offline.
A Cisco switch requires a Cisco optic. If you were to use a Myricom optic it would see that the optic was made by “Myricom” with a part number “10G-SFP-SR” and it would lock that port out because it has an incompatible optic. Never mind that a valid Cisco optic and the “failed” Myricom optics may very well have been made by JDSU on the same assembly line, perhaps even on the same day.
Network adapter vendors, like Myricom, are optic agnostic. You can shove in an Arista, Cisco, HP, or Gnodal, we won’t care. We provide optics to offer a complete solution for our customers. Finally, we are not “in the optic business” so we pick them up, mark them up fairly, then offer them for sale. I can assure you we’re not buying them at the same discount that a Cisco or Juniper might be getting, yet our price is clearly so much more reasonable.
Now here’s the secret I promised. Most switch vendors have a patch for the switch operating system so that it will ignore the optic check and allow you to use anybody’s optics. If you have the buying power and the cojones, then insist that they provide the patch as a condition of buying their switch. It will save you big time. You can then take those savings, and buy a few more Myricom 10GbE adapters.