networking

Arm photon Torpedoes. Prepare to fire on the Klingon Warbirds.

scottcschweitzer networking

Is an unforgettable quote from Captain James T. Kirk of the starship Enterprise that still sends chills down my spine. Unfortunately, we need to take a break this week from Machine Vision Acceleration (MVA), to let the ink dry on a few things. So instead we’re going to explore tools by answering one simple question, how strong is your optical signal? From mythical Photon Torpedoes to real life photonic communication mankind defines itself by its tools. How good are the 10GbE tools your NIC vendor provides to configure & diagnose the status of your adapter and optics, and are they free? 

Yesterday I had the pleasure of visiting a DoD customer of mine in DC whom I’ve been selling 10GbE NICs to since early 2006. They asked what appeared to be a very simple question: “How can we tell the optical signal strength coming into your NIC?” Their servers are installed in extremely remote collocation facilities. Clearly, this is a reasonable question, but in my nearly eight years of selling 10GbE, it’s one I’ve never been asked. I said I’m sure we have something that provides that info, perhaps “ethtool -S” might do it, I’ll look into it. It turns out we have a tool called “mdio_rw” that we provide with our free Myri-10G Adapter Tool Kit. Today I had a chance to install the toolkit and test it out. There are over 20 tools in this digital bucket, most are diagnostic or for extreme setup, meaning they aren’t for the faint of heart. Here is how you install the toolkit on Linux if you’re a windows user visit the above link for those details.

# gunzip -c myri-tools-linux.tar.gz | tar xvf – 
# cd myri-tools-1.26-linux/bin
Then to see the receive power, the amount of light arriving into the adapter on the first 10GbE port, you can issue the command:

# mdio_rw -b 0 | grep RX 

RX power = 0.6199 mW

This is the actual output from one of my test systems. The command returns another dozen parameters, some pretty interesting, but I’ll leave that to you to explore. 

Many of the 20+ tools in this package can be used to change how the adapter works by altering settings, upgrading the firmware, or performing deep diagnostics, all of these should be used with great caution. Here are several other interesting ones that rookies like I can play around with to gather useful information:


myri_info – Returns a number of hardware statistics on the adapter & PCIe bus. 
myri_pcie_conf – This confirms the PCIe payload sizes of the adapter and the system chipsets.
myrige-test.sh – Performs an adapter loopback self-test.

There is more to the loopback self-test, which you can read about in this link. 
So the next time you buy a 10GbE adapter check first to see what tools your vendor provides to help answer any questions you might have, diagnose any issues that may arise or allow you to further tune or configure your system’s performance. Also are the tools free, and easy to install and use? You may be surprised at what you might learn. 

Video Streaming to Jumbo Boob Tubes (Part 1 of 3)

scottcschweitzer networking

This series of articles was also posted on 10GbE.net. 

At CES this week Netflix, announced streaming support for SuperHD and 3D.  For a select few in the back of their booth, they also showed off UltraHD (4K) streaming. Some of you may know that Hulu Plus already supports HD streaming, but NetFlix is the 800Lbs gorilla, and when he moves people pay attention. Also this month we learned that Redbox has partnered up with Verizon to deliver their new Redbox Instant streaming service for the same $8/month that NetFlix charges.  Let us not forget that Amazon Prime also offers HD streaming, so there are plenty of choices now for streaming quality commercial content. 
Now, to be honest, most of my NetFlix streaming is still on my iPad, and the content is older TV shows that are likely only stored in Standard Definition (SD) format in the “cloud”.  Also like most folks these days my primary TV is one of those jumbo sets, but unlike my college sophomore daughter who exclusively uses a Roku box, ours is still connected to a digital Time Warner set top box.  So what does all this have to do with 10Gb Ethernet?  
Streaming content comes from service providers and content delivery networks that make up the “cloud”.  As we move from SD to HD formats like SuperHD, 3D, and soon UltraHD their servers and networks will be bursting with the new demand for bandwidth.  What is the actual impact of moving to these high-resolution options?
Here are some rough numbers that I’ve pulled from Wikipedia’s page on H264 stream encoding:

Format                Bandwidth   SD Equivalent  H264 Level

SD (480i)                  5Mbps              1X                 2.1
SuperHD (1080p)   25Mbps              5X                  4
UltraHD (3840p)   300Mbps             60X                5.1

How does this impact the video servers on the other side of the network providing your content? Well if they’re still using Gigabit (GbE) links, and only serving up SD streams they can support about 150 customers per network port.  When they move to 10GbE that jumps up to 1960 streams (10GbE links are a bit more efficient) per port.  Most of the Content Delivery Networks (CDNs) have already moved to 10GbE for this initial boost.  It becomes even more affordable when they utilize dual port cards so they can provide nearly 4,000 customers service from a single server.

Now along comes HD, and these servers with dual 10GbE ports can only handle about 800 customers.  Once those customers upgrade to UltraHD that number drops to 65 customers/server.  Wow from 4,000 to 65, and those 65 are consuming the same number of bits/sec so the backend storage demand remains constant. Server managers have to balance demand, people wanting to watch videos, with the hardware capability of both their storage & network infrastructure. As we saw on Christmas morning when NetFlix’s streaming service crashed for 24 hours, it’s not hard to miscalculate maintaining this balance.
Stay tuned, and next week I’ll share what technology is available to ease the burden on these video server vendors and server managers.

Video Streaming to Jumbo Boob Tubes (Part 2 of 3)

scottcschweitzer networking

The cult classic Monty Python and the Holy Grail in standard definition (SD) from Apple’s iTunes service is 1.00GB of data, the HD (720p) download file is 2.91GB, and the SuperHD (Apple just calls it 1080p) file is 4.84GB. This is a perfect real world example of the problem because the base file is something most of us geeks are familiar with and it’s exactly 1.00GB. As such it clearly shows the expansion as we move from format to format. Also, it should be noted that this doesn’t address UltraHD which will grow files another 12X in size over SuperHD.

So with the data demand for video streaming growing exponentially as a result of consumers moving from SD to HD & SuperHD, and soon UltraHD what can be done to ease the burden on these servers? First, we need to take a quick low-level look at the problem. A movie in the “cloud” is nothing more than a file on storage somewhere that needs to be transferred through a network to a consumer. Focusing on the server side we have several areas where one can further optimize performance:

  • Tracking all the clients & their associated streams.
  • Moving the bits that make up the movie from storage to memory.
  • Then moving those bits from memory to the network adapter.
  • Finally transmitting these bits from the network adapter onto the wire so they can reach the consumer.

Tracking network connections and mapping them to streams is a collaboration between both the application, and the OS. File transfers from storage to memory is an OS function requested by the application, these have been well optimized over the years so there can only be marginal improvements here. Moving bits from memory to a network adapter is also considered an OS function, but it’s been one that the HPC (High-Performance Computing) crowd has been optimizing for well over a decade through a technique called OS Bypass. This is still more art than engineering so it’s not mainstream. The final step is often considered hardware, but it’s also the most intriguing. Normally packets arrive via the OS and have to be transmitted as is onto the Ethernet, boring. If on the other hand, they arrive via OS Bypass then some real magic can happen.

It appears that two of the highest profile web video streaming entities, NetFlix & Hulu, along with a collection of others have moved to a new web server platform called Nginx (pronounced engine-x) to address the C10K problem. C10K is the name given to a class of problems associated with keeping track of 10’s of thousands of web server clients, and the data they are connected to. Nginx addresses this through a number of sophisticated techniques, that are better explained on their website.

The next issue is that of transporting the movie from storage, as a file, to memory. Again, this falls into the realm of Nginx as it’s both the application and the OS that are responsible for all the latency around file I/O. Nginx has had extensive tuning over the years to minimize this sort of latency.

So now we’re left with what is considered both an OS and a hardware issue. This is moving the bits to the network adapter and putting them on the wire to the client. Nginx with any generic 10GbE adapter works fine, but the real magic happens when it’s coupled with an intelligent 10GbE adapter and FastStack™ VideoPump™ which utilizes HPC OS Bypass techniques to dramatically improve system performance. Next week in part 3 of this series we’ll show you what those improvements really are and give you a peek into the magic behind the curtain that makes this all possible. Stay tuned, same bat time, same bat channel…

Video Streaming to Jumbo Boob Tubes (Part 3 of 3)

scottcschweitzer networking

So how do web content delivery networks (CDN) or a video streaming service providers like NetFlix or Hulu handle such explosive growth? Well, the first thing most of them did was move to a more robust web server platform designed to handle many thousands of users at a time, in most cases as mentioned in “Part 2” that has been Nginx. As content, Video places a far greater demand on the server than traditional web pages. A normal web user requests a page, the components arrive, they are then assembled by the user’s browser, and typically read for about one minute. The user then requests another page. A video user, on the other hand, is far more demanding. For their experience, they demand a constant stream of packets is sent to them continuously for very long periods of time. These packets need to be transmitted from the content server at very regular intervals, much like the lights on California highway on ramps that meter traffic entering the highway during rush hour. Maintaining continuous streams of video packets to thousands of web connected users is something Nginx can do very well.

So we mentioned in “Part 2” of this series that magic can occur when Nginx is coupled with an “Intelligent NIC”. This “magic” is called FastStack VidePump™, and here are three main benefits:

Congestion mitigation, having a well-tuned user space communications stack and a processor based NIC enables us to be considerably smarter with regard to sending many streams through many small pipes.

Asynchronous sendfile() support, this dramatically improves overall I/O performance.
This “magic” requires a processor based network adapter that can communicate directly with user space memory. This alone dramatically offloads the host CPU from having to provide file based content delivery to web users.

FastStack VideoPump™ offered with both Emulex OneConnect and Myricom network adapters provides all of the above features. Today Nginx supports many users by typically having 1,024 worker threads all doing synchronous sendfile(). With VideoPump™ and a single worker thread we see dramatic performance improvements because Nginx isn’t flopping around between many threads trying keep the I/O pipeline full. So if all of the Nginx video streaming demand is coming from TCP sources (intelligent devices like browsers or applications running on laptops, tablets or smartphones) and the video content is being served up from files, the performance gains afforded by VideoPump™ in transparent mode (which requires no code changes to Nginx or applications) delivers a 25% reduction in host CPU utilization.

VideoPump™ can also be accessed through an Application Programming Interface. For applications designed to serve video content we’ve measured host CPU gains of up to 50%. To learn more about FastStack VideoPump™ please consider contacting your Emulex or Myricom sales representative.

How Ethernet Won the West (Part 1 of 2)

scottcschweitzer networking

This article was originally posted in December 2012 at 10GbE.net. 

Back in the early 90s an internal IBM cartoon called “How OS/2 Won the West” was being passed around on VHS it was hysterical but ultimately wrong. Superior technologies and products like Betamax, Token Ring & the Apple Newton litter the battlefield of technology. In the end, all that matters is market dominance.  It’s our culture, look at the Olympics, we all know that Gabby Douglas won gold for the Women’s Gymnastics Individual All-around, so who won the silver?  Someone from China? No, Viktoria Komova from Russia.  Our society pays very little attention to second place.  

Over its three decade rule of the networking market Ethernet has crushed: PC Network, Token Ring, AppleTalk, DECnet, Myrinet, Quadrics, and dozens of other interesting approaches.  Today only two competing technologies remain and each has tried to carve out its own niche.  Fiber Channel defined and ruled the storage market, while Infiniband has become the network of choice for the scientific community.
 
Fiber Channel is now seriously threatened by a protocol called FCoE, Fibre Channel over Ethernet. Essentially all that is Fiber Channel is wrapped in an Ethernet package and shipped over an Ethernet network.   Fiber Channel has moved through  2, 4, 8 and now 16Gbps.  During roughly that same time Ethernet has migrated from 1 to 10 and very soon 40Gbps (note 100Gbps exists for the switch to switch connections, but it is impractical today for the server to server communication).  Much of the Fiber Channel world is running with 4 & 8Gbps networks while the server market has been moving rather quickly from 1 to 10Gbps. Furthermore, Ethernet controllers, the chips in the server that connect to the actual network, have gotten far more sophisticated over the last two generations so they can now off-load the host CPU from doing many of the mundane networking chores while delivering amazing performance.  Alan Weckel, senior director at Dell’Oro, said “We believe that in 2013, most large enterprises will upgrade to 10Gb Ethernet for server access through a mix of connectivity options, ranging from blade servers, SFP+ direct attach and 10G Base-T,”  Alan then went on to say “We further anticipate that in 2014, small and medium businesses will upgrade to 10Gb Ethernet.” The bleeding edge customers will begin testing 40Gb Ethernet in 2013, with adoption really starting to ramp in 2014.  At this point several things converge:  Ethernet network performance will far exceed Fiber Channel network performance, this generation of ethernet controller silicon coupled with enhanced value added software should be mature enough to deliver all the features of Fiber Channel, and all of this will be done at a commodity price point.
 
Infiniband (IB) is a network of a whole different color.  Here the value proposition is two fold, low latency at a low price point.  One of the issues plaguing IB is the decreasing number of competitors in this market that sells switches and adapters.  Several years ago there were well over a half a dozen, today there is just Mellanox and Intel (they bought QLogic’s switch & adapter business in 2012 for $125M).  Intel’s roots in this space run deep, all the way back to 1999.  At that time Intel had co-developed something called Next Generation I/O (NGIO) with SUN and Dell.  In 1999 a consortium was formed that merged NGIO with Future I/O (a competing technology developed by IBM, HP & Compaq) to become Infiniband.  The division of Intel that acquired IB from QLogic is the same division that drives Intel’s very robust 10Gb Ethernet business.  In 2011 they acquired Fulcrum to fill out their networking line in order to compete head to head with Broadcom.  This division of Intel did just over $10B in 2011 with expectations of ramping it to $20B over the next five years. Intel, like Broadcom, is very good at selling chips, and pretty good at selling adapters.  I doubt we’ll see any new Intel developed IB switches anytime soon so Mellanox will likely be walking this path alone. It will be interesting over the next year to see if Intel fuels IB or pours water on it.
 
Next week – Part II, “Where is Ethernet Headed & How Network Companies, Other Than Intel & Broadcom, Will Survive & Prosper”

How Ethernet Won the West (Part 2 of 2)

scottcschweitzer networking

This article was originally published in January 2013 at 10GbE.net. 

This could also be titled where is Ethernet headed & how network companies, other than Intel & Broadcom, can survive & prosper.

Last month we saw two interesting announcements that may have foretold the future of Ethernet.  The first was by Emulex, and it was that they were going to be acquiring Endace, an Ethernet analytics technology company and that it would close this acquisition in the first quarter of 2013.  The second came on“Cramer’s Mad Money” when Cramer said sell Mellanox, don’t touch the stock.  It has dropped from a high of $120 in September to $50 this week, a nearly 60% drop in three months.
 
In May of 2012, Emulex partnered up with my employer, Myricom, to license both our hardware and software technology for adding value to Ethernet.  Two years ago Emulex acquired ServerEngines, a company that had commodity grade 10Gb Ethernet adapter silicon.  So why did a leader in Fiber Channel storage networking buying a 10Gb Ethernet commodity silicon company, partner with a value added Ethernet company and announce an intent to acquire a third Ethernet analytics company?  Simple Emulex understands that Fiber Channel has the same future that Token Ring, AppleTalk, and  Myrinet have all shared, and that’s Ethernet.  Ethernet isn’t the best architecture, but it is by far the most pervasive.  The IEEE, the standards body behind Ethernet has remained active so the standards continue to evolve and keep pace with processor technology. If you recall from your days back in Computer Architecture, there are four performance curves: CPU, memory, disk & network.  When we moved from Gigabit Ethernet to 10Gb Ethernet the network curve jumped in front of the disk curve.  Now even disks spinning at 15K RPMs can only provide data at about 1.2Gbps, and the newest FLASH drives are typically only 4X better, still well under 10Gbps. Furthermore, 40Gb Ethernet is just around the corner for server to server communications.  Add to that enhancements like: Data Center Bridging (DCB) which includes key concepts from Converged Enhanced Ethernet (CEE) and Data Center Ethernet (DCE),  RDMA over Converged Ethernet (RoCE, pronounced Rocky), and the Common Communication Interface (CCI) and you’ll see that Ethernet is continuing to expand to meet the demands of the bleeding edge. Companies like Emulex understand that the future of networking is Ethernet, and that profit will only come by adding value on top of this industry standard.
 
With only two companies selling Infiniband the future of this interconnect is in question.  On August 20th Cramer’s Mad Money was Obsessed with Mellanox at $113, it soon hit $120.  At the time the implications of Intel’s acquisition of QLogic’s Infiniband assets weren’t totally understood.  Intel is a whale in the silicon market, and it can swish its tail one way and buy Fulcrum (Ethernet switching silicon), then the other and buy up the second to last Infiniband provider.  The question is will they continue to support both?  Now four months later Cramer’s calling Mellanox “the disaster du jour” as it struggles in the $50s.  You can bet Cramer’s recent remarks aren’t going to help that stock move up anytime soon. Efforts like CCI, mentioned above, will remove most if not all of the technical advantages Infiniband might have, and this will force Mellanox into committing 100% to becoming an Ethernet technology provider. At that point on the server side, they’ll need to compete head to head with Emulex, and Qlogic who are both very well entrenched, and on the switch side with a host of deep pocketed competitors like Cisco, Brocade, Foundry, Force10, Extreme, etc… Competing on two fronts against well-established companies as one orphan their primary technology differentiation is a challenge very few companies ever have to face, let alone survive.
 
So what does all this mean?  If you are a networking company it’s rather simple, embrace Ethernet, and find a way to add value on top of it or die.  I’m sure other networking technologies will pop up in the future, like say quantum state photon based approaches, which will spawn new companies, and network products. Eventually, though I’m sure we’ll figure out how to map Ethernet into the quantum world…
 
Happy New Year.
 

Turkey Time, a Watch, and Accuracy

scottcschweitzer networking, Time

This article was originally posted in November of 2012 on 10GbE.net. 

While waiting on the turkey yesterday I was flipping through the latest issue of Wired and stumbled across the new Seiko Astron watch, and my inner nerd started to swoon.  Now for the few of you out there who don’t get Wired, especially the December issue, think of it as the geek version of the old Sears Wishbook. This time of year every tenth ad in the magazine is a high-end watch, its geek meets chic.  Among all the fancy watch ads, here was both an article and an ad for a Seiko, the brand my dad wore.  Dad worked outdoors every day of his life, never used a computer & swore by his Seiko, he considered it the working man’s watch. So it was kinda funny seeing Seiko among ads for all the other high-end brands from Rolex on down.

To be sure we’re all on the same page let’s first take a moment, and define accuracy.  Simply put accuracy, when talking about time, is the average deviation from the reference time.  Today for high precision instruments accuracy is often measured in nanoseconds (1x10E-9 or billionths of a second) lost, or gained each second or day. With 86,400 seconds in a day sometimes it’s easier to use a day when dealing with really small numbers. National and international reference clocks use the excitation of Cesium atoms by microwaves then they measuring the frequency of the resulting emitted photons as the electrons jump energy states.  This process is so repeatable, that it was made the international standard for time keeping over 50 years ago.
 
So what about this Seiko arose the geek inside me? Active GPS synchronization to the local time zone, and an understanding of all 39 time zones world wide.  This watch figures out where you are on the planet then selects the appropriate time zone and resets itself to local time, all for only $2,300.  My smartphone has been doing the same thing since they first arrived, but that’s a different story.  Seiko also claims the Astron has an accuracy of 1 second every 100,000 years, or 27 nanoseconds/day. By watch standards, this is very accurate.
 
So how does the Astron stack up to some real world high precision clocks?  Clock systems used in electronic financial markets typically use highly accurate clocks (1 picosecond/day internally) that are even more precise than Cesium clocks. To be in-step with the rest of the world though they must rely on our less accurate GPS system (10 nanoseconds/second), and often a pulse per second distribution mechanism which reduces this further to 25 nanoseconds/second (2 milliseconds/day).  As mentioned above commonly used Cesium clocks are accurate to 1 second every 1,400,000 years or 2 nanoseconds/day. A new proposed standard clock would excite neutrons instead of electrons and thus be even more accurate, 1/20th of second every 14,000,000,000 years. Well, there’s the oven timer, it’s accurate to a minute every six months when I reset it, the turkey’s done. Happy belated Thanksgiving everyone…

Who Spilt the Milk? The Importance of Attribution

scottcschweitzer networking, Security

This article was originally posted in November of 2012 at 10GbE.net. 

As a parent the moment we hear a glass of milk bounce off the floor we’re conditioned to assign blame, and if necessary meter out punishment. In the real world attribution for a kinetic attack has become much easier given our satellite, ground & ocean sensor networks, and forensic sciences. Today determining who launched an air strike, fired off a missile or attacked an embassy can often be resolved in hours, or worst case days. In the world of cyberspace, the exact opposite is true. Furthermore, even if a nation could unequivocally attribute an attack, international case law for cyber warfare doesn’t exist so justifying a counter strike, cyber or kinetic, is uncharted territory. 

Many techniques and tools exist to intentionally obfuscate the source of an attack. Furthermore, the most sophisticated assaults are designed to morph with each stage of the deployment so the strategy itself is also hidden. This can be done many ways, but the intent is always the same, to further mask the original target and the method of the offensive attack. That way when the forensic computer scientists come knocking all they will find are empty cyber shell casings, of the most generic type, devoid of any usable digital fingerprints. If there are clues, it’s very likely those were planted to frame a believable third party so that retribution will result in a secondary attack perhaps on the actual intended target. Suppose you whack the biggest bully in school on the back of the head, and quickly your friend assigns blame to your enemy, who also just happens to be standing there. Now you know the bully will deliver a far more destructive attack than you ever could so why wouldn’t you leverage such a strategy. If it’s executed properly only you and your buddy will ever really know what happened.
 
Why is attribution important? Because the US, and other countries, have publicly stated that given the void of international cyber warfare case law a significant attack in cyberspace will be met with a declaration of war. Attacking nations of the world have been put on notice to expect both a kinetic and a cyber response. Not responding will be seen as a sign of weakness. So what does a country do though when attribution is murky or impossible? Are they justified in launching their own covert stealthy cyber attack?

Trick or Treating and Why Failing Open is Failing Safe

scottcschweitzer Capture, networking

This article was originally purchased in October of 2012 on 10GbE.net.
Revised on 11/1 with additional technical details found at the end.

Tomorrow children across the US will be going door to door begging for candy, my teenager will be one of them.  Suppose you weren’t home, would you appreciate it if they let themselves in, and rummaged through your home looking for treats?  How about if they also stole your extra front door key, and used your home to party whenever you weren’t home. This is what botnets do. Some years ago it was demonstrated that a new “out of the box Windows PC” without updates survived exactly 20 minutes when attached directly to the Internet.
 
Some network hardware vendors consider this approach acceptable.  They’ve gone to great lengths to design hardware so that when the server dies the network “fails-closed” (in the electrical sense) meaning that traffic continues to flow through the network unchecked.  Now if this approach was used to enhance the networking experience, perhaps traffic shaping or load balancing, I could see how this might be a selling point. What we need to avoid is having engineers design security appliances that “fail-closed”. Security adapters should always “fail-open” or “fail-safe”. Today unknowingly our firewalls, and systems are probed & assaulted hourly as pawns in a cyber war waged by individuals, corporations & nation states. I bring this up because we recently came across a line of Intrusion Prevention Systems (IPS) that offered this as a feature, seriously.
 
Imagine you hire a doorman to increase the safety of your home, and early on Halloween evening he dozes and falls off the porch into the bushes sound asleep. This is “failing-closed”. I’d rather if my doorman was to fall asleep that he slumped against the door totally blocking entry “failing-open” or “failing-safe”. How does your 10GbE IPS or IDS fail?
 
Additional Technical Details
I’ve been asked how “failing-closed” actually works. The simple analogy is a single lens reflex camera. These are the big bulky cameras that professional photographers have used for decades. How they work is simple, light enters through some very expensive lenses on the front of the camera. A mirror reflects that light up to an eye piece so you can compose your picture. When the camera is on, and you depress the shutter release that mirror quickly flips out of the way to expose your film or the CCD in your digital SRL, then the mirror flips back much like the shutter in even older style cameras.
 
With network adapters that fail open consider the first network port as the lens and the second network port as the eye piece. When the card is powered off it is in bypass mode, “failing-closed”, a mirror (actually an optical MEMS) takes the light entering the first port and sends it to the second and vice-versa. When the network adapter is fully powered on, and the drivers have all successfully loaded the mirror can be flipped open and the 10GbE ASIC chip sees all the traffic and can selectively pass traffic back and forth between the ports. It can also send traffic up to the server, and the server can then send traffic out one or both ports. It’s a very flexible approach, but the use case is not very common and the cards are often priced five times higher than competing dual port cards.
 
Thanks to Gary Archer from Emulex for suggesting this topic.

Creating a 160 Gbps Ethernet Server Port

scottcschweitzer networking

This article was originally published in October 2012 at 10GbE.net.

In Ethernet five terms: link aggregation, port trunking, link bundling, teaming, and channel bonding, are all used to describe coupling multiple network ports together within a system or switch, to form an even larger logical network pipe.

Typically Linux, OSX, and Unix systems offer this as a standard OS feature. All versions of Windows before “Server 2012” needed a special driver to handle this, but with Myricom’s standard 10GbE windows driver, this feature is included.  In fact, Myricom’s Windows driver will team up to 16 ports of 10GbE together creating a single logical 160GbE port.  Today 5U servers exist with up to 10 available PCIe slots, an example might be the SuperMicro 5086B-TRF.  If one were to install 8 dual port Myricom 10GbE adapters running under Windows and “Team” them all together they could create the network equivalent of a firehouse. Now honestly you wouldn’t get all 160Gbps in each direction, likely at best only 70% of that. Various issues ranging from the method used for teaming to how your switch is configured and how it views a teamed connection will likely establish your limit.
 
Several years ago a customer reported to me that he’d teamed five 10GbE ports together and was seeing 62Gbps one way and 72Gbps the other, this was a single socket Core i7 board and we both believed was very likely hitting the limit of the PCIe chipset he was using.
 
Why would one want to do this?  Well, suppose you need to serve up an enormous library of content to a vast array of computers.  By “teaming” up several adapters you can quickly, easily & dramatically improve your network performance because your software won’t have to manage multiple network connections.  Furthermore, the teaming driver will handle network port outages and mask those from your application. To learn more about teaming (link aggregation) consider reading this wiki page.
 
Thanks to Mike Fahey from Emulex for suggesting the topic.