Author: scottcschweitzer

Digital Ocean Moves to 40GbE

scottcschweitzer networking

This week Digital Ocean announced SolarFlare as its worldwide technology partner for upgrading their data center servers from 10G to 40G Ethernet. So who is Digital Ocean? According to Netcraft, the leader in Internet Analytics, Digital Ocean is the world’s second largest hosting provider. Digital Ocean offers their customers a KVM based on-demand cloud infrastructure for hosting their site and applications. So what?

In the words of Ben Uretsky, CEO of Digital Ocean, “We are committed to providing software developers with a fast, secure, and reliable cloud environment to design their applications.” So when asked why Digital Ocean selected Solarflare’s 7042 series of 10/40GbE adapters Ben responded: “Solarflare’s technology roadmap and the company’s dedication to network performance and security made them an ideal strategic partner to meet the demands of our explosive growth.”

Well there you go, if Solarflare is good enough for Digital Ocean’s many thousands of servers worldwide, why not give them a try in your data center. Drop me a note using the Contact Scott block on the right, and I’d be thrilled to see if we can work out a no charge try & buy.

99.99999% Available + 2.7us = 1 Awesome Computer

scottcschweitzer HFT, networking, Time

What do you get when you put together a pair of dual socket servers running in hardware lock-step with a pair of leading edge, ultra-low latency OS Bypass network adapters all running RedHat Enterprise Linux? One awesome 24 core system that boasts 99.99999% uptime, zero jitter, 2.7 micro seconds of 1/2 round trip UDP latency, and 2.9 microseconds for TCP.

How is this possible? First, we’ll cover what Stratus Technologies has done with Lock-Step, and how it makes the ftServer dramatically different than all others. Then we’ll explain what jitter is, and why removing it is so critical for deterministic systems like financial trading. Finally, we’ll cover these impressive Solarflare ultra-low latency numbers, and what they really mean.

We’ve all bought something with a credit card, flown through Chicago O’hare, used public utilities, and possibly even called 9-1-1. What you don’t know is that very often at the heart of each of these systems is a Stratus server. Stratus should adopt the old Timex slogan “It takes a licking and keeps on ticking” because that’s what it means to provide 99.99999% up time, you’re allowed three seconds a year for unplanned outages. Three seconds is how long it takes me to say “99.99999% up time.” How is this possible? Imagine running a three legged race with a friend. Ideally, if you each compared your actions continuously with every step you could run the race at the pace of the slowest of the two of you. This is the key concept behind Lock-Step, comparing, then knowing what to do as one starts to stumble to ensure the team continues moving forward no matter what happens. Stratus leverages the latest 12-core Intel Haswell E5-2670v3 server processors with support for up to 512GB of DDR4. If any hardware component in the server fails, the system as a whole continues moving forward, alerts an admin who then replaces the failed component, then that subsystem is brought back online. I challenge you to find another computer in your life that has ever offered that level of availability over the typical 5-7 year lifecycle that Stratus servers often see.

So what is Jitter? When a computer core becomes distracted from doing its primary task to go off and do some routine house keeping (operating system or hardware driven), the impact of that temporary distraction is known as Jitter. With normal computing tasks, Jitter is hardly noticeable, it’s the computer equivalent of background noise. With certain VERY time critical computing tasks though, like say financial trading, even one Jitter event could be devastating. Suppose your server’s primary function is financial trading, and it receives a signal from market A that someone wants to buy IBM at $100, and on market B it sees a second signal that another entity wishes to sell IBM at $99. So the trading algorithm on your server buys the stock on B for $99, but then the instant it has confirmation of your purchase a thermal sensor in your server generates an interrupt. The CPU then that is running your trading algorithm goes off to service that interrupt which results in it running some code to determine which fan to turn on. Eventually, say a millisecond or so later, control is returned to your trading algorithm, but by then the buyer on market A is gone, and the new price of IBM has fallen to $99. That’s the impact of Jitter, brief often totally random moments in the trading day stolen to do basic house keeping. These stolen moments can quickly add up for traders, and for exchanges, they can be devastating. Imagine a delayed order as a result of Jitter missing an opportunity! Stratus Technologies has crawled through their server architecture and eliminated all potential sources of Jitter. Traders & exchanges using other platforms have to do all this by hand, and this is still as much art as it is science. That’s one reason why over 1,400 different customers regularly depend on Solarflare.

Finally, there’s ultra-low latency networking via generic TCP/IP and UDP networking. In the diagram below network latency is in blue. Market data arrives via UDP and orders are placed through the more reliable TCP/IP protocol. Here is a quick anatomy of part of the trading process showing one UDP receive and one TCP send. There are other components, but this is a distilled example.

click to make larger
Initially, the packet is received in from the wire, the light blue block, and the packet passes through the physical interface, electrical networking signals are converted to layer-2 logical bits. From there the packet is passed to the on-chip layer-2 switch which steers the packet to one of 2,048 virtualized NICs (vNIC) instances, also on the chip. The VNIC then uses DMA to transfer the packet into system memory, all of which takes 500 nanoseconds. The packet has now left the network adapter and is on its way to a communications stack somewhere in system memory, the dark blue box. Here is where Solarflare shines. In the top timeline, the dark blue box represents their host kernel device driver and the Linux communications stack. Solarflare’s kernel device driver is arguably one of the fastest in the industry, but most of this dark blue box is time spent working with the kernel. There are CPU task switches, and several memory copies of the packet, as it moves through the system, and thousands of CPU instructions are executed, all told this can be nearly 3,000 nanoseconds. In the bottom timeline, the packet is DMA’d directly into user-space where Solarflare’s very tight user space stack sits. This is where the packet is quickly processed and handed off to the end user application via the traditional sockets interface. All without additional data copies, and CPU task switches, and completed in just under 1,000 nano seconds a savings of about 2,000 nanoseconds or roughly 4,600 CPU instructions for this processor at this speed. All this, and we’ve just received a packet into our application, represented by the green blocks.
So in the two bars above the first represents market data coming in via Solarflare’s generic kernel device driver than going through the normal Linux stack until the packet is handed off to the application. The response packet, in this case, a trade via TCP, is sent back through the stack to the network adapter and eventually put on the wire, all told just over 9,000 nanoseconds. With Stratus & Solarflare the second bar shows the latency of the same transaction, but traveling through Solarflare’s OS Bypass stack in both directions, the difference here is that the transaction hits the exchange over 4,000 nanoseconds sooner. This means you can trade at nearly twice the speed, a true competitive advantage. Now four millionths of a second aren’t something humans can easily grasp, so let’s jump to light speed, this is how long it takes a photon of light to cover nearly a mile.
So if you’re looking to build a financial trading system with ultra-high availability, zero jitter & extreme network performance, you have only one choice Stratus’s new ftServer.

5 Reasons Infiniband Will Lose Relevance After 100G

scottcschweitzer Infiniband, networking, RDMA

Proprietary technologies briefly lead the market because they introduce disruptive features not found in the available standard offerings. Soon after, those features are merged into the standard. We’ve seen this many times in the interconnects used in High-Performance Computing (HPC). From 2001 through 2004 Myrinet adoption grew as rapidly in the Top500 as Ethernet, and if you were building a cluster at that time you likely used one or the other. Myrinet provided significantly lower latency, a higher performance switching fabric, and double the effective bandwidth, but it came with a larger price tag. In the below graph Myrinet made up nearly all of the declining gray line through 2010, by

which time the Top500 was split between Infiniband and Ethernet. Today Myrinet is gone, Infiniband is on top just edging out Ethernet, but its time in the sun has begun to fade as it faces challenges in five distinct areas.

1. Competition, in 2016 and beyond Infiniband EDR customers will have several attractive options: 25GbE, 50GbE and by 2017 100GbE along with Intel’s Omni-Path. For the past several generations Infiniband has raced so far ahead of Ethernet that it left little choice. Recently though within HPC 10GbE adoption has been growing rapidly, and is responsible for much of Ethernet’s growth in the past six months. During the same time, 40GbE has seen little penetration, it’s often viewed as too expensive. In 2016 we will see an IEEE approved 25GbE and 50GbE standard emerges, along with new & affordable cabling/optics options. It should be noted that a single 50GbE link aligns very well with the most common host server bus connection PCIe Gen3 x8 which delivers roughly 52Gbps/unidirectionally. For 100GbE we’ll need PCIe Gen4 x8. While 100Gbps could be done today with PCIe Gen3 x16 often HPC system architects leave this slot open for I/O hungry GPU cards. The second front Infiniband will be facing is Intel’s Omni-Path technology which will also offer a 100Gbps solution, but it will be directly off the host CPU complex designed to be a routable extensible interconnect fabric. Intel made a huge splash at SC15 with Omni-Path & switching which is a fusion of intellectual property Intel picked up from Cray, Qlogic, and several other Infiniband acquisitions. Some view 2017 as the year when both 100GbE and Omni-Path will begin to chip away at Infiniband’s performance revenue while 25/50GbE erodes the value focused HPC and Exascale customers Infiniband has been enjoying.

2. Bandwidth, if you’ve wanted something greater that 10GbE over a single link, you’ve pretty much had little choice up to this point. While 40GbE exists many view this as an expensive alternative. Recent pushes by two groups to flesh out 25GbE and 50GbE ahead of the IEEE have resulted in this standards group stepping up its’ efforts.  All of this has accelerated the industries approach toward a unified 100GbE server solution for 2017. Add to this Arista and others pushing Intel to provide CLR4 as an affordable 4-channel 25G, 100G optical transceiver, and things get even more interesting.

3. Latency, has always been a strong reason for selecting Infiniband. Much of its gains are the result of moving the communications stack into user space and accelerating the wire to PCIe bus connection.  These tricks are not unique to Infiniband, others have played them all for Ethernet delivering performance ethernet controllers and OS bypass stacks which now offers similar latencies when compared at similar speeds. This is why nearly all securities worldwide are traded through systems using Solarflare adapters leveraging their OS Bypass stack called OpenOnload while using standard UDP, and TCP protocols. The domain of low latency is no longer exclusive to RDMA as it can now be more easily, and transparently done using existing code, and via UDP and TCP transport layers over industry standard ethernet.

4. Single vendor, if you want Infiniband there really is only one vendor who offers a single end-to-end solution. End-to-end solution providers are great because they expose the single throat to choke when things eventually don’t work it. Conversely, many customers will avoid adopting technologies where there is only one single provider because it removes competition and choice from the equation. Also when that vendor stumbles, and they always do, with a single vendor you’re stuck. Ethernet, the open industry standard, affords you options while also providing you with interoperability.

5. Return to a Single Network, ever since Fiber Channel intruded into the data center nearly two decades ago network engineers have been looking ways to remove it. Then along came exascale, HPC by another name, and Infiniband was also pulled into the data center. Some will say Infiniband can do all three, but clearly, those people have never dealt with bridging real world Ethernet traffic with Infiniband traffic. At 100Gbps Ethernet should have what it needs in both features, and performance, to provide a pipeline for all three protocols over a single generic network fabric.

Given all the above it should be interesting to revisit this post in 2018 to see how the market reacted. For some perspective, in this blog back in December 2012, I wrote: “How Ethernet Won the West” where I predicted that both Fiber Channel and Infiniband would eventually disappear. Fiber Channel as a result of Fiber Channel over Ethernet (FCoE), which never really took off, and Infiniband because everyone else was abandoning it including Jim Cramer. Turns out while I’ve yet to be right about either, Cramer nailed it.  Since January 2013, adjusting for splits and dividends, Mellanox stock has dropped 14%.

1.44 us Full Round Trip Latency, Unlikely

scottcschweitzer HFT , , ,

Tuesday morning one of the guys on my team woke me with a text stating a competitor was claiming 1.44 microseconds for a full round trip (RT) using UDP.  Two things about this immediately struck me as strange: first it was reported as a full round trip number, and second, the number (excluding units) was oddly close to what I’d thought the theoretical 1/2 RT limit might be. You see in the ultra-low latency, high-frequency trading market, time is everything. One need only be a few nanoseconds faster than their competition to win the lion’s share of the business. So speed is everything, but in the end, physics sets the speed limit.

In an ideal world if one were to measure the time required for a UDP packet to enter a network server adapter, traverse the Ethernet controller chip, travel the host PCIe bus, through the Intel CPU complex and finally end up in memory they’d find that this journey was roughly 730 nanoseconds. Now it should be noted that this varies across Intel server families & clock rates. We could be off by as much as +/- 100 nanoseconds, measuring at this level is pretty challenging, but 730 nanoseconds is a reasonable number to start with. Also, it should be noted that this is with Solarflare’s current 7000 series Ethernet Controller ASIC.

Breaking this down further, the most expensive part of this trip is the 500 nanoseconds or so the UDP packet will spend in Solarflare’s Ethernet controller chip. This chip is arguably the most popular low latency Ethernet Controller ASIC on the market today, it includes a high-performance PHY layer, an L2 switch, and built-in PCIe controller logic, everything happens within this single chip.  Over 1,000 financial trading firms rely on this technology daily, most of the world’s financial exchanges and nearly all of their high-performance customers depend on Solarflare, and as such they’ve turned all the dials possible to squeeze out every available nanosecond. Add to this 150 nanoseconds, the time the packet will spend traveling across the PCIe bus using DMA to cache via DDIO (not RAM), and finally another 80 nanoseconds or so to store it in RAM, making your final total 730 nanoseconds to receive a packet to memory. Again, your mileage will vary considerably so please only use these numbers as rough reference points. For a 1/2RT you’ll need to double this number (a receive plus a send) which brings the 1/2RT total to 1,460 nanoseconds, or 1.46 microseconds. It should also be noted that receives and sends have different costs, sends often consume less time, so again your numbers will vary, and this number should, in fact, be smaller. That’s Solarflare physics.  Solarflare has a new 8000 series Ethernet Controller ASIC coming out soon which will further trim down the 500 nanoseconds spent in the ASIC, but by exactly how much is still a closely guarded secret.

So is 1.44 microseconds for a conventional (through to user space vs. done completely in an FPGA) full round trip possible today? Well, the PCIe and memory components of this total 920 nanoseconds (150 nanoseconds for the PCIe bus plus 80 nanoseconds for CPU to memory, and both times 4 to address a full round trip). This leaves 520 nanoseconds to traverse the Ethernet Controller logic four times, or 130 nanoseconds for each pass. Considering that the most popular low-latency Ethernet controller chip on the planet requires 500 nanoseconds, doing it in 130 nanoseconds with the same degree of utility is highly unlikely.

On checking this competitor’s data sheet for this product we found that they have documented 1.82 microseconds for a UDP 1/2RT using 64-byte packets. Compare this to the 1.44 microseconds they claimed verbally for a full round trip, and one could see that they’ve significantly stretched the truth. If it sounds too good to be true, it probably is…

Next Generation Network (NGN) Forensics

scottcschweitzer Capture, networking, Security

In two weeks I’ll be hosting two events: one near the NSA at the Courtyard Fort Meade on the 27th, and the second a few miles from the CIA at the Hilton Mclean Tyson Corner. Both will start at 4 PM, have a considerably strong Network Forensics agenda, and then wrap up by 7 PM. Both events are identical, they’re free to attend, and we’ll be providing beer, wine & soda along with appetizers.

Below is the current agenda, but it’s still somewhat fluid at this point as it’s the first NGN we’re doing that is focused on Network Forensics, and is setup for the Intelligence Community. The dozen or so prior NGNs, and another later that week have been centered on the Financial Trading market where our leadership in low latency is well known. Here is the current agenda:

  • 3:30 – 4:00  Registration, drinks and snacks
  • 4:00 – 4:05  Welcome & Introductions – Paul Bravo, Solarflare VP US/OEM Sales
  • 4:05 – 4:35  Keynote Speaker – David Riddoch, Founder, Solarflare
  • 4:35 – 4:50  Dell – Cameron Chehreh, CTO Federal
  • 4:50 – 5:35  Panel: Deep Packet Inspection (Snort, Suricata & Bro)
  • 5:35 – 5:45  Reservoir Labs – Alison Ryan, VP Business Development
  • 5:45 – 6:15  Advances in Capture – Ron Miller, Director Technical Marketing, Solarflare
  • 6:15 – 6:45  Open Q&A of all speakers
  • 6:45 – 7:30  Reception & Demonstrations

I’ll moderate the panel on Deep Packet Inspection which will include:  Eileen Donnelly, US Government, Erik Mogus Director Sales Engineering for Reservoir Labs, and someone to be named shortly. The purpose of the panel is to compare and contrast the different approaches to packet inspection so someone familiar with one can see different perspectives on the other two.  The audience will also be encouraged to ask questions of the panelists.

To sign up for either event please use one of the links below:

Tuesday, October 27th l Time: 3:30-7:30 PM
Courtyard Marriott Fort Meade, 2700 Hercules Rd, Annapolis Junction, MD
Click to attend
Wednesday, October 28th l Time: 3:30-7:30 PM
Hilton McLean Tysons Corner, 7920 Jones Branch Drive, McLean, VA
Click to attend

 

Stock Trading in 300ns: Low Latency Redefined

scottcschweitzer HFT, networking

Henry Ford is quoted as having once said “If I had asked people what they wanted, they would have said faster horses.” Not all innovations are as ground breaking as the automobile, but when one approaches an old problem with both a new strategy and improved technology great things can happen. In June Solarflare released an update to OpenOnload (OOL) that introduced TCP Delegated Send, and in late September they will begin shipping the latest version of their Application Onload Engine (AOE) with an advanced FPGA. The combination of these two will result in the capability of turning around a TCP Send in 300ns, compared to roughly 1,700ns today. In latency focused applications a savings of 1,400ns, an 82% improvement, is game changing. To understand how Solarflare pulls this off let’s look at a much simpler example.

My son uses an online exchange to trade Magic cards, and traders are rated on how fast they fill orders. Not much different than NASDAQ processing an order for Apple stock. When my son started he would receive an order on his computer, and search through a heap of cards to find the one necessary to fill that order. He would then go down several flights of stairs to my office to fetch an envelope, and stamp then goes back up to his computer. Next, he would address the envelope, apply the stamp, run back down the stairs and walk the completed trade to the mailbox. Today he has a cache of pre-stamped envelopes with the return addresses pre-written out sitting beside his computer. All his cards are in a binder with an updated index. Filling a trade is a trivial matter. He simply checks the index, pulls the required card from the binder, updates the index, stuffs the card in an envelope, writes the final address on the front, and runs it out to the mailbox. Essentially, this is a Delegated Send. Everything that can be preprocessed in advance of the actual trade is prefetched & prepackaged.

When it comes to TCP and Delegated Send, at the start of the trading day the trading application, through OOL, establishes a TCP connection with the exchange. The trading application then calls a routine in OOL to take over control of the socket’s send path, and to obtain the Ethernet, IP and TCP headers for the connection.  The application adds to these a message template and passes the resulting packet template to the FPGA where it remains cached, much like my son’s stack of pre-stamped envelopes. In response to incoming packets arriving at the FPGA causing the RTL trading code to trigger a trade, the trade is then inserted into the pre-formatted packet, the checksum computed, and packet transferred to the exchange. The whole process takes approximately 300ns. When the ACK arrives from the exchange it is then passed transparently back to the trading application through OOL. Now some will point out that other FPGA solutions exist today that enables you to possibly trade at these speeds, but do any of these solutions make it this simple? With some minor modifications to your existing trading application you can quickly take advantage of Delegated Send with the AOE, no other FPGA solution even comes close!

So if latency in trading is important to you, and you’d like your orders moving along 1,000ns faster then perhaps it’s time to take a serious look at Delegated Send on Solarflare’s AOE. To learn more please consider checking out this whitepaper.

For those already familiar with Solarflare’s AOE, this new version of the product has several very substantial improvements:

  • It leverages the latest Solarflare high-performance ASIC with a PCIe Gen3 interface.
  • Flexible, open choice of FPGA PCS/MAC.
  • All FDK modules, and examples delivered with full source code.
  • Sample implementations of 10GbE & 40GbE pass-through (requires PCS/MAC).
  • Sample implementations of all four 1600MHz DDR3 AOE memory channels.
For additional information please send me an email.

Cookbook: Installing & Updating SolarCapture on CentOS

scottcschweitzer Capture, networking

By: Justin Borland, Equifax

Below is a document describing how to update/install Solarflare’s SolarCapture drivers on the SolarFlare 71xx NICs for a CentOS system. Note this document was user created and has not yet been reviewed by Solarflare’s Technical Support.

Requirements:
First, acquire the following software from Solarflare support:

Implementation steps:
SolarFlare SDK install:

# unzip solar_capture_sdk-.zip
# rpm –Uvh sfutils-*.rpm

OpenOnload build:

# tar –xf openonload-.tgz
# cd openonload-
# onload_uninstall && ./scripts/onload_install

Note that if any capture processes are already running (i.e. Moloch, Bro, Snort or Suricata) kill them before reloading, and keep them off until the end of this install.

# onload_tool reload
# cd ..

SolarCapture Core installation:

# rpm –Uvh solar_capture-core-.rpm

SolarCapture Python install:

# yum install rpm-build
# rpmbuild –rebuild solar_capture-python-.src.rpm
# rpm –Uvh /root/rpmbuild/RPMS/x86_64/solar_capture-python-.rpm

SolarCapture Pro installation:

# rpm –Uvh  –replacepkgs solar_capture-live-.rpm solar_capture-pro-.rpm

Update the NIC firmware

# sfupdate –write

Note that you should answer ‘Y’ for each NIC.

Restart capture services

# /wherever/your/capture/script/lives_start.sh


Verify Install:
Check installed packages

# rpm –qa | grep capture

You should now see this:

solar_capture-core-1.3.1.11-0.x86_64
solar_capture-pro-1.3.1.11-0.x86_64
solar_capture-python-1.3.1.11-0.x86_64
solar_capture-live-1.3.1.11-0.x86_64

Check NIC licensing

# sfkey

Which should then look like this:

2-interface adapter: eth4, eth6
Product name:        Solarflare SFN7122F SFP+ Flareon Ultra Server Adapter
Part number:         SFN7122F
Serial number:       712200210071141137100559
MAC addresses:       00-0F-53-xx-xx-xx, 00-0F-53-xx-xx-xx
Installed keys:      Onload
Active keys:         Onload
Blacklisted keys:    0
Invalid keys:        0
Unverifiable keys:   0
Inapplicable keys:   0

One final note, Solarflare Support has not yet validated the above steps so please use at your own risk.

25/50/100GbE Facts

scottcschweitzer Uncategorized

Several years ago the mega data center and Web 2.0 companies started looking for an alternative to the approved 40GbE standard which is the link speed offered between 10GbE, and 100GbE. They viewed the IEEE approved implementations of both 40G and 100G, which were simply multiple 10G lanes, as very cumbersome. These mega data centers seek to leverage High-Performance Computing (HPC) concepts and desire that their exascale networks utilize fabrics that scale in even multiples. Installing 25/50GbE at the servers, and (2x25G) 50GbE, or (4x25G) 100GbE for the switch to switch links is much more efficient. It turns out two groups formed in parallel: The 25 Gigabit Ethernet Consortium and the 2550100 Alliance (that’s 25, 50, 100 for those that didn’t see it) to develop & promote a single lane 25GbE specification as the next Ethernet. This approach would then be extended to two 25G lanes for 50G, and four 25G lanes for 100G. It should be noted that today the IEEE, the industry standards body, has not yet ratified a 25GbE standard (when it does it will be referred to as IEEE 802.3by). Once approved this standard will be used to create Ethernet Controller NIC silicon and compatible switch silicon. This work is underway, but won’t be completed until sometime in 2016.

The 25 Gigabit Ethernet Consortium was founded by Arista, Broadcom, Google, Mellanox & Microsoft. While the 2550100 Alliance is about fifty companies, and the ten most notable in this alliance are: Accton, Acer, Cavium, Finisar, Hitachi Data Systems, Huawei, Lenovo, NEC, Qlogic, and Xilinx. Interestingly absent from the above lists are key Ethernet product companies: Chelsio, Cisco, Emulex, HP, Intel, and Solarflare. The focus of this piece will be on the Ethernet NIC controller silicon, because if you can’t connect the server then the whole discussion is just switch to switch interconnects, which is a another class of problem for discussion in a different forum. Today there appears to be only two general purpose Ethernet controller NIC chips that support a version of 25/50/100GbE and they are Mellanox’s ConnectX-4 and QLogic’s cLOM8514. For NIC silicon to actually be useful though it must be delivered on a PCI Express adapter. At this point QLogic has only demonstrated their 25GbE publicly, but has not announced a date to ship a PCIe adapter. This means that Mellanox is the solitary vendor shipping a production 25/50/100GbE adapter today. QLogic has not formally stated when it will be producing adapters with the cLOM8514.

In the home Wifi networking market hardware vendors typically race ahead of IEEE standards, and produce products to secure “first mover advantage”, otherwise known to end users as the bleeding edge, but they can only do this because their products are for the most part stand-a-lone. Enterprise and data center markets are highly interconnected and shipping a product ahead of the approved IEEE specification is inviting an avalanche of support calls. Today there still remain significant open technical issues around 25/50/100GbE such as auto negotiation, link training, and forward error correction. The IEEE has yet to resolve these, but they are being discussed. At the end user level, interoperability is the key issue. If a company were to produce a stand-a-lone NIC product without an accompanying cable & switch ecosystem they would be flooded with support requests. The converse is also true, if a company were to build a switch around the Broadcom silicon without offering a bundled in server NIC it would quickly also become an interoperability situation. Those on the bleeding edge, would surely now understand the true meaning of the phrase.

So why haven’t the more traditional 10GbE NIC vendors jumped on the 25/50/100GbE band wagon? Simple, without an approved IEEE standard the likelihood of profiting from your investment in 25/50/100GbE is fairly low. Today, exclusive of R&D, producing an Ethernet controller NIC chip is a multi-million dollar exercise. So to justify spinning a 25/50/100GbE NIC chip in early 2015 for a “first mover advantage” one would require a plan that it would produce well into the tens of millions in revenue. Couple this with the interoperability support nightmare of getting one vendor’s NIC working with a second vendors cable, and third vendors switch, and any profit that might exist could quickly be consumed.

Enterprise customers want choice, which by definition implies multivendor interoperability based on mature standards. Once the IEEE 802.3by standard (25/50/100GbE) is ratified next year it is expected that all the NIC vendors will begin shipping 25/50/100GbE NIC products.

7/27 Update: Broadcom announced a 25/50GbE NIC controller today.

A Path from GbE to 10GbE

scottcschweitzer Uncategorized

Recently folks have asked how they could squeeze more out of their Gigabit Ethernet (GbE) infrastructures while they work to secure funding for a 10GbE upgrade in the future. I’ve been selling 10GbE NICs for 10 years and blogging the past six. What I’ve learned as a former system architect, IT manager, server sales, and now network sales is that the least painful method for making the transition is to demonstrate the payback to your management in stages. First I’d upgrade my existing servers to 10GbE adapters, but running on my existing GbE network to demonstrate that I was pushing that infrastructure to its full potential. It’s very likely that your existing multi-core servers are sometimes more CPU bound than bandwidth. Also, it is possible you may have some extra top of rack switch ports you can leverage. There are several interesting tricks worth considering. The first is to move to a current 10GbE controller, one that supports GbE (1000Base-T is the formal name for the RJ-45 telephone style modular connector). If this still doesn’t give you the performance bang you’re seeking then you can consider testing an operating system bypass (OS Bypass) network driver.

Upgrading from the generic GbE port mounted on your server’s mother board to a PCI Express option card with dual 10G ports means you’re moving from GbE chip technology designed 15 years ago to very possibly a state of the art 10G Ethernet controller designed in the past year or two. As mentioned in other posts like “Why 10G” and “Four Reasons Why 10GbE NIC Design Matters” some of today’s 10GbE chips internally offer 1,000s of virtual NIC interfaces, highly intelligent network traffic steering to CPU cores, and a number of advanced stateless network packet processing offloads (meaning that more work is done on the NIC that would otherwise normally have to be done by your Intel server CPUs).  Much of this didn’t exist when your server’s GbE chip was initially designed back in 2000. So what is the best way to make the jump?

There are two methods to plug your existing RJ45 terminated networking cables into new 10GbE server class NICs. The first, and easiest, is to use a native dual port 10GBase-T card that supports GbE like Solarflare’s SFN5161T which runs roughly $420. The second approach, which provides a much better path to 10GbE, is to use a dual port SFP+ card like Solarflare’s SFN7002F with a pair of 1000Base-T modules. In this case, the adapter is $395, and each module is roughly $40 (be careful here because there are numerous Cisco products offered that are often just “compatible”). When you get around to migrating to 10GbE both approaches will require new switches and very likely new network wiring. The 10Gbase-T standard, which uses the familiar RJ45 networking connector, will require that you move to the more expensive Cat6 cabling, and often these switches are more expensive and draw more power. If you have to rewire with Cat6, then you should seriously consider using passive DirectAttach DA cables with bonded SFP+ connectors that start at $20-$25 for 0.5-2M long cables. By the time your network admin custom makes the Cat6 cables for your rack it’ll likely be a break even expense cost (especially when you have to spend time diagnosing bad/failing cables). DA cables should be considerably more trouble free over time, frankly, 10GBase-T really pushes the limits of both the Cat6 cables and RJ-45 connectors.

Another thing to consider is leveraging an OS Bypass layer like Solarflare’s OpenOnload (OOL) for network intense applications like Nginx, Memcached, and HAProxy. We saw that OOL delivered a 3X performance gain over running packets through the Linux OS, which was documented in this whitepaper. In the testing for this whitepaper, we found that for Nginx content served from memory would typically take six cores to respond to a full 10G link. Running OOL it only required two cores. Turning this around a bit, with OOL on a dual port 10G card you should only need roughly four cores to serve static in-memory content at wire-rate 10G to both ports. So suppose you have an eight core server today with a pair of GbE links, and during peak times it’s typically running near capacity. By upgrading to a Solarflare adapter with OOL, still just utilizing both 10G ports as GbE ports, you could easily be buying yourself back some significant Intel CPU cycles. The above requires a very serious your mileage may vary type statement, but if you’re interested in giving it a try in your lab Solarflare will work with you on the Proof of Concept (POC). It should be noted that adding OOL to a SFN7002F adapter will roughly double the price of the adapter, but compare that additional few hundred dollars in a 10G software expense to the cost of replacing your server with a whole new one, installing all new software, perhaps additional new software licenses, configuration, testing, etc… Replacing the NIC, and adding an OS Bypass layer like OOL is actually pretty quick, easy & painless.

If you’re interested in kicking off a GbE to 10GbE POC please send us a brief email.

Why 10G?

scottcschweitzer networking

Today servers are typically installed with more than 16 logical CPUs, and yet many still leverage the same Gigabit Ethernet interface for production which was common when we had single core CPUs. There are three simple reasons you should consider 10GbE in your next server deployment: price, performance, and security.

Price is the easiest attribute to grasp. Dual port 10GbE adapters start at $395, and often because of the advanced silicon they deliver more than 10X the throughput of GbE. In fact, in some operational modes that we’ll talk about below 10GbE adapters with additional software can actually reduce host CPU load 70% by simply off-loading production networking traffic from having to traverse the Linux kernel.

Last October in the “Four Reasons Why 10GbE NIC Design Matters” we talked about Ethernet controller advances like vNICs, MSI-X,  RSS, and virtualization support. Two additional articles cover some very simple methods for extracting the best possible performance from your 10GbE NICs and they are: “Network Performance Tuning for Bandwidth on Linux” and “How to achieve low latency with 10Gbps Ethernet.” When it comes to improving performance specific to applications like Nginx and Memcached we’ve found that OS Bypass can often yield anywhere from a 130-300% performance gain. The links attached to these two applications above will take you to articles covering this in more detail.  Back in March, we discussed “10G/40G NIC Partitioning Using SR-IOV or PF-IOV Modes” to explain how today’s sophisticated Ethernet controller silicon allows you to leverage 1,024 vNICs per physical 10GbE port. Finally, in “Density Comes to 10GbE” we reviewed the latest entry into the quad-port 10GbE adapter market Solarflare’s SFN7004F which sells for $645.

The third and last point is security. Only one 10GbE company “Enables Servers to Defend Against a DDoS Attack.” Solarflare adapters support a low-level kernel device driver called SolarSecure Filter Engine which can fend off a Distributed Denial of Service attack, rate limit traffic by IP address, and leverage both white & black list filtering (by address, and port) to protect the server.  In the near future Solarflare will also deliver a Docker container that will dynamically detect attacks on a server, and in real time create new filter rules and load them into the kernel device driver. Early next year those rules will be loaded directly into the server adapter itself.

To learn more please send us a brief email.