CNA – Converged Network Agora

This article was originally posted in May of 2012 at

With every generation of Ethernet, there is always a new crop of agile startups who race to silicon and deliver a variety of new ASICs for network adapters and switches.  As the market matures competition thins the herd and after several years only the best products remain.  This Darwinian process is what the market does best, and the process was in full swing in January of 2009 when I authored a post “Thinning the 10GbE Herd.”  In this article, I mentioned NetXen, Neterion, Tehuti, ServerEngines, NetEffect, & Teak Technologies all of which are gone three years later. 

Today three startups who have remained focused on high-performance 10GbE network interface card (NIC) silicon & software for the past six years still remain.  These companies are Chelsio, Myricom & Solarflare.  Don’t get me wrong, large public companies like Intel, Broadcom, Emulex & Qlogic ship far more 10GbE ports, but these are all well established public companies focused on the larger 10GbE market, and with significant silicon production capabilities.  To be complete there is also a cache of very niche players like Napatech, Endace, and HotLava but these firms leverage FPGAs or in HotLava’s case Intel silicon.
As someone who’s been selling 10GbE for the past seven years full time, it finally appears that 2012 will be the year of convergence.  Chelsio & Solarflare are both venture capital backed and by most estimates are reaching the end of their funding.  Myricom is privately held, and has been running on revenue for 18 years, also last week they announced a partnership with Emulex.
In the recent past, Intel gobbled up NetEffect & Folcrum so their appetite for 10GbE appears pretty well satiated.  Early on Broadcom acquired Siliquent, and they appear content.  So the game of musical chairs has started.  Emulex & Qlogic have made various plays, but they both still appear hungry, which leaves only two seats.  We have three remaining players (Chelsio, Myricom & Solarflare).  Later this year the music will stop, none may remain, and the convergence will be complete.  Two may win, and likely one will lose, the question is which?
Then in 2013, we get to start it all over again with 40GbE!

FPGAs on 10GbE NICs, An Idea Whose Time Has Passed

This article was originally published in April of 2012 on

A few months ago SolarFlare announced a new class of Network Interface Card (NIC), a hybrid adapter, that will be available in June. This hybrid combines their generic 10GbE ASIC with a Field Programmable Gate Array (FPGA) chip, some RAM and then they wrap all this hardware in a Software Development Kit (SDK). This will then be marketed as a complete solution for the High-Frequency Trading (HFT) market. Rumors exist that they’ll also try and sell it into the network security market, and perhaps others.

At the time of this writing high performance dual port NICs have a street price between $550 & $750, this new hybrid NIC is rumored to cost ten times this. So why would someone even consider this approach? Simple to reduce load on the host CPU cores. The initial pitch is that this hybrid will take on the role of the feed handler. Typically a feed handler runs on several cores of a multi-core server today. It receives trading data from all the relevant exchanges, then filters off all the unwanted information, normalizes what remains and then passes this onto cores running the algorithmic trading code. By freeing up the feed handler cores, through the use of a hybrid NIC, this processing power can be allocated to run more advanced algorithmic codes.
On the surface, the pitch sounds like a great idea. Use a proven low-latency ASIC to pull packets off the wire, send the boring packets on to the OS and the interesting financial stuff to the FPGA. It’s when you get into the details that you realize it’s nothing more than a marketing scheme. When this product was designed I’m sure it sounded like a good idea, most 1U and 2U servers had eight cores and systems were getting CPU bound. As this NIC hits the market though Intel has once again turned the crank and vendors like IBM and HP are now delivering dual socket 16 core, 32 thread servers that will easily pickup the slack. A nicely configured HP DL360P with 16 cores, 32GB memory, etc… is available today for $10K, adding one of these hybrid NICs will nearly double the hardware price of your trading platform. Note, this before you even crack open the SDK and hire the small army of consultants you’ll need to program the FPGA.
Typically we’ve found that the normal packet flow from multiple exchanges into a trading server is roughly 200-300K packets per second, with very rare bursts up to 800K. So if one were to set aside four cores for feed handling, with an average feed load of 250Kpps, and assuming the feeds were evenly distributed each core would have 16 microseconds per packet. On these new 2.2Ghz Intel E5 systems this translates to roughly 8K instructions per packet to filter & normalize. This assumes two threads per core and an average of four clock ticks per instruction.
Like TCP Offload Engines (TOEs) these hybrid NICs sound great when they’re first proposed, but on in-depth analysis and particularly after Moore’s law kicks in, they soon become solutions looking for a problem, a novelty. With Intel’s new E5s why would anyone seriously invest their time hardware & consulting budgets on an outdated approach?

World War III is Now in Cyber Space

This article was originally published in May of 2011 on

“We don’t know who struck first, us or them. But we do know it was us that scorched the sky.” Morpheus explaining to Neo how the final World War between the humans & the machines started.
Unlike the first two World Wars, or the final fictitious one is in the Matrix. WW III is being fought today exclusively in cyber space.  It is a silent war, rarely reported in mainstream media, primarily because there’s almost never a human casualty*, and most readers wouldn’t understand it.  Often the combatants have no idea who they’re fighting. If we look at one of the early major battles, Stuxnet, no one has officially claimed to have launched the attack, but the target was laser focused on Iran’s nuclear weapons program. The campaign was successful, and it delayed Iran’s weapons program by several years.
Now I’m sure you’re wondering, “Why is this being covered in a blog about 10GbE?”  Simple, all battles require supply lines. Today the supply lines for cyber warfare are moving from GbE to 10GbE.  Sure a lone hacker can use a Low Orbit Ion Cannon (LOIC) to take down an unsophisticated opponent using little more than a laptop, and a decent Internet connection. Frankly, this is synonymous in the real world to someone with a machine gun trying to shut down a store.
On the other hand governments, and corporations have substantially more resources.  As we saw with Stuxnet.  For those not familiar with Stuxnet, this was a very carefully engineered offensive cyber weapon delivered in virus form.  It had several different parallel attack vectors to ensure that it reached all the intended targets, which reports suggest that it did.  It contained a precise triggering system and a lethal warhead.   This battle would not have been possible with only a handful of intelligent people.  It took a collaborative effort by several governments, at least one corporation, and several unique pieces of equipment for testing to ensure the weapon’s success.  This went way beyond your garden variety LOIC class assault and defined a whole new category of offensive cyber weaponry.  While the LOIC is a machine gun obtainable by nearly anyone, Stuxnet, by comparison, is a state of the art cruise missile.
Now back to 10GbE.  Markets like HFT and HPC are breaking new ground in low latency, high packet rate & wire-rate bandwidth.  They are utilizing advanced 10GbE network adapters and switches, along with sophisticated algorithmic routines.  Today HFT’s are exploring lossless wire-rate 10GbE capture to do real time analysis and simulation.  They capture real trading data in buffers or spin up synthetic ones to emulate the market.  They then inject these buffers into an HFT trading engine to see how it responds.  Is this really any different than simulating 100s or 1,000’s of web surfers?  This technology is crossing over into cyber warfare.
All of the above was written over the holiday weekend.  This morning the Wall Street Journal reported that the US Government has announced that The Pentagon has adopted a new strategy that will classify major cyber attacks as acts of war, paving the way for possible real world military retaliation.  “If you shut down our power grid, maybe we will put a missile down one of your smokestacks,” said a military official as quoted in the WSJ.  So if you don’t believe the ongoing war in cyber space is real, then just consider it a prelude to it.
* John P. Wheeler III may very well be the first person slain in this cyber war.  He is a former Pentagon official and was a consultant for Mitre prior to his murder.  John was an outspoken proponent of Cyber defense.  In January 2011 he was found dead in a Maryland landfill, the investigation into his death is currently stalled.

Optical Lock Down

This article was originally published in May of 2011 at

Today for the umpteenth time I had to explain to someone that if you go optical to connect your server to your switch with 10GbE it could easily cost you twice as much.  There is a secret at the end of this entry that MIGHT allow you to save some big time cash if you have enough muscle, but you have to read to the end of this entry.

For cable runs of seven meters or less you should always use Direct Attach (DA  otherwise known as Twinax) cable if possible as it could easily save enough to basically connect the second server for free! Here are some actual numbers from earlier today.
First, some basic end user costs assuming a five-meter run, note these are rounded a little bit to keep the math simple:
10GbE Network adapters, roughly $400/port
10GbE Switches, roughly $500/port
10GbE SR SFP+ Optics from switch vendor $800/port
10GbE SR SFP+ Optics from NIC vendor $200/port
10GbE SR Optical 5M cable $80/ea
10GbE Direct Attach 5M cable roughly $140/ea
Now let’s build a solution between the server and the switch using optics:
10GbE Network adapter $400
10GbE SR Optic from NIC vendor $200
10GbE SR Optical 5M cable $80
10GbE SR Optic from  Switch vendor $800
10GbE Switch port $500
Total $1,980 to connect a single server
Direct Attach (Twinax) Option:
10GbE Network adapter $400
10GbE Direct Attach 5M cable roughly $140/ea
10GbE Switch port $500
Total $1,040 to connect a single server
Let’s look more closely at the market dynamics going on here.  First, only a handful of companies make 80% of the 10GbE Short Range (SR) optics that everyone uses today.  These companies are typical: JDSU, Finisar, Agilent, etc…  None of the switch companies or NIC companies make their own optics, we all source them from several of the above companies, and a few others, all of whom rebrand them for us and burn our company name and part number into what is essentially flash memory within the optic.
Here’s where it gets interesting.  Myricom, the company I work for, sells it’s SR SFP+ optics online via CDW’s website for $185.  Here are some of the more expensive SR SFP+ optics listed on CDW’s site:
HP Procurve: $1,498
Avaya: $1,350
Enterasys: $1,210
Cisco: $1,100
Juniper: $1,082
Brocade: $1,022
QLogic: $930
IBM: $920
Now remember under the covers we’re all sourcing these optics from the same competitive pool, so why the price spread?
First, remember that we each buy our optics with our manufacturer name and part numbers already burned into them by the optics makers mentioned above.  Now here’s where it gets interesting the switch makers during switch initialization query the optic and if it does not return a valid company name and part number then it locks the optic out and reports the port as offline.  
A Cisco switch requires a Cisco optic.  If you were to use a Myricom optic it would see that the optic was made by “Myricom” with a part number “10G-SFP-SR” and it would lock that port out because it has an incompatible optic.  Never mind that a valid Cisco optic and the “failed” Myricom optics may very well have been made by JDSU on the same assembly line, perhaps even on the same day. 
Network adapter vendors, like Myricom, are optic agnostic. You can shove in an Arista, Cisco, HP, or Gnodal, we won’t care.  We provide optics to offer a complete solution for our customers.  Finally, we are not “in the optic business” so we pick them up, mark them up fairly, then offer them for sale.  I can assure you we’re not buying them at the same discount that a Cisco or Juniper might be getting, yet our price is clearly so much more reasonable.  
Now here’s the secret I promised.  Most switch vendors have a patch for the switch operating system so that it will ignore the optic check and allow you to use anybody’s optics.  If you have the buying power and the cojones, then insist that they provide the patch as a condition of buying their switch.  It will save you big time.  You can then take those savings, and buy a few more Myricom 10GbE adapters.

Dualies Aren’t Just for Trucks

This article was originally published in April of 2009 at

One would think that after 30 years our industry would have developed a NIC naming convention for “dual-port.” Does a dual-port NIC mean your OS sees one or two interfaces? Do dual-port NICs mean that one port is active and the other is for fail-over? Can a dual-port run traffic through both port simultaneously? It all depends on who you talk to, and the product they’re selling.

With 10GbE we’ve seen three main approaches for building dual-port NICs:
Active/Active: this is what most people expect, a single OS interface with a driver that sprays traffic fairly evenly across both network ports and if one port fails the other picks up the slack until it can handle no more:
  • Chelsio’s N320E for $790 is an example of this type of card.
  • Intel’s AF DA card for $799 appears to be another example of this class of card.
Dual-NIC: two OS interfaces are presented to the OS and both interfaces run independently. This typically affords the best performance and the most flexibility:
  • Myricom’s 10G-PCIE2-8B2-2S+E for $995 appears to be the only example of this approach. Myricom utilizes two unique 10GbE controllers on the same PCI Express Gen2 NIC and a PCI Express bridge chip to break the slot into two unique NIC devices.
Active/Passive or Active/Fail-over: a single OS interface with a driver that monitors connectivity on the active port and if the connection fails the driver migrates traffic rapidly over to the second port:
  • Myricom’s 10G-PCIE-8B-2S+E for $795 is an example of this type of card. The fail over time is under 10 microseconds.
  • Chelsio’s B320E Bypass adapter for $3,483 is similar but it can detect an OS/BIOS/System failure and make a hard switch over to the second port.
Do the above categories cover it, or do we need more lingo? When looking for a dual-port NIC, what features do you require, and what do you expect? Please let us know.
P.S. As I brought this page back online I left off the links as most no longer apply, but from a historical perspective it is interesting to see how things have progressed.

Thinning the 10GbE Herd

This article was originally published in January of 2009 at

Thinning the herd.

In 2007 over one million 10GbE network ports were purchased. Many of those were for a switch to switch interconnects but some were to connect servers to networks via 10GbE. Natural selection is now taking effect in the 10GbE NIC market as the big dogs, Intel & Broadcom, start thrashing around in an effort to secure market share as 10GbE matures. Both want to dominate the 10GbE LAN on Motherboard (LoM) market. In the NIC market, four companies likely supply over 80% of the 10GbE NICs purchased and they are Chelsio, Intel, Myricom, and Neterion. The remaining 20% of NIC sales fall to companies like Broadcom, SMC, NetXen, ServerEngines, Tehuti, AdvancedIO, Endace, Napatech, etc… One should be wondering why Broadcom is in the second group, it’s because Broadcom’s focus is on selling 10GbE silicon to OEMs like IBM and HP for LoM projects positioning their silicon on high-end server mother boards and not retailing NIC cards. 

Officially the first documented victim is NetEffect, the leader in iWarp (Infiniband for 10GbE) NICs. NetEffect rose from the ashes of a failed Infiniband company, Banderacom, earlier this decade to apply their silicon development skills and Infiniband algorithms to the more stable Ethernet market as a new feature called iWarp. NetEffect in-fact led the iWarp charge, it was the self-proclaimed leader in low-latency iWarp 10GbE NICs. In August NetEffect filed for reorganization in US Bankruptcy Court. With the failure of NetEffect the market has cast its vote and drove a stake through the heart of iWarp, hopefully terminating this feature.
Rumors have been swirling around Teak Technologies, a maker of 10GbE NICs and a switch, for some time. It appears that Teak has not weathered the storm and has since faded away, their domain name is no longer resolving to an IP address. The domain was never transferred from the founder, and the founder announced this spring on Linkedin that he had moved on some time ago. Is it conclusive evidence, no, but would you buy technology from a tech company whose URL won’t resolve to a server?
It is a tough economic climate for start-up NIC companies, particularly those in the bottom 20% as they have likely never had a quarter in the black. Now is a challenging time to be out there seeking another round of capital from ones VCs. Several have been without an injection of new funding for over two years and lack the sales volume required to sustain their own existence much beyond year end. As such we’ve directly questioned one firm to see if they are alive, and another that is widely rumored in the industry to be in trouble, but their marketing departments are still bailing.