Author: scottcschweitzer

Performance Beyond 10GbE

scottcschweitzer 25GbE, networking

Today performance servers leverage 10 Gigabit Ethernet (10GbE) to fully utilize all the compute resources at their disposal. As Intel’s latest release of the 18 core Haswell server chips hits the market system architects are beginning to consider network fabrics above 10GbE. Four potential choices exist, and they are 20GbE, 25GbE, 40GbE & 100GbE.

Before diving into each of these options we should set some groundwork. Most performance I/O adapters these days are inserted into the motherboard in a third generation PCI Express (PCIe Gen3) slot that is 8 lanes wide. The theoretical performance of this slot is 64 gigabits/second (Gbps), but after encoding & overhead the effective data rate is more like 52Gbps. Also, it should be noted that on Intel systems PCIe slots have a preference to CPU sockets. So data coming from a PCIe slot that is “wired” to “Socket 0” but is destined for a core on the CPU in “Socket 1” will see a measurable degradation in performance. Most applications will likely not care, but if performance is your specialty you should look into this. You see those bits have to travel a much longer path to reach that distant core. If you’re really interested in achieving the optimum performance you should evenly split your I/O across slots mapped to each CPU socket.

Beyond 10GbE the two currently approved standards which are 40GbE and 100GbE. Many of the NIC companies are already shipping products that support 40GbE, and most of the performance switch vendors support both 40GbE & 100GbE connections. The reluctance of the NIC companies to go beyond 40GbE is bound to the common 8 lane PCIe Gen3 slots that most NIC cards are installed into. As mentioned above the slots these cards go into supports roughly 52Gbps in each direction. So while a dual port 40G NIC can deliver up to 80Gbps by definition, the card can only bring data into the motherboard at 52Gbps so the card by definition is roughly 35% over subscribed. This is why we’re not going to see any 100GbE NICs in existing servers. For 100GbE NIC companies will require a 16-lane PCIe Gen3 slot or a future 8-lane PCIe Gen4 slot, as both should sustain roughly 104Gbps. So you’ll have to wait for Intel’s next tock ( a major step forward) and the delivery of Skylake, the successor to Broadwell, for real 100GbE NIC systems to appear.

So what about 20GbE, is this something to consider? Well, 20GbE is something HP cooked up working with QLogic that they’d delivered as a product for their blade system. It never really gained any traction outside of that platform. Normally 20Gbps is simply achieved by bonding both ports on a dual port 10GbE adapter together. This can be done several ways and is very common place. This will likely go no further as a hardware option.

Now 25GbE is a horse of a different color, and it is seeing some adoption, but mostly at the top of rack switch level. To better understand this 100GbE is actually four 25GbE lanes, so fracturing this into 25GbE is actually somewhat logical. Arista Networks, Google, Microsoft, Broadcom & Mellanox are all working the switch side of this. In September of 2014, Broadcom announced their StrataXGS Tomahawk chip, which supports 128 ports of 25GbE, 50 ports of 50GbE or 32 ports of 100GbE. So these switches are really close, and we may even see them at SC14 this week. In October Emulex joined the 25GbE Consortium so clearly, there will soon be some NICs in this space. At this time no vendors have announced 25GbE NICs.

Beyond Gigabit Ethernet

scottcschweitzer Uncategorized

Where wired connections can be made, they will always provide superior performance to that of wireless techniques. Since the commercialization of the telegraph over 175 years ago mankind has been looking for ever faster ways to encode & transfer information. The wired standard we’re all most familiar with today is Gigabit Ethernet (GbE). It runs throughout your office to your desktop, phone, printers, copiers, and wireless access points. It is the most pervasive method in the enterprise for reliably linking devices. So what’s next?

Two weeks ago if you’d have asked most technology professionals they would have answered 10 Gigabit Ethernet (10GbE). That was the commonly accepted plan. Then Cisco, Aquantia, Freescale & Xilinx announced an alliance to further develop & promote a proposed Next Generation (NBase-T) wired standard supporting 2.5GbE & 5GbE speeds over existing installed wires (Category 5a & 6) cables. We all know Cisco, and that’s enough to get pretty much everyone’s attention, but who are the other three? Aquantia is one of the leaders in producing the physical interface (PHY) chips that exist at both ends of the wire. Switch companies like Cisco use Aquantia, as do network interface card companies like Solarflare, Intel, and Chelsio. Aquantia has figured out how to take digital information and encode it into electrical signals designed to travel at very high speeds through very noisy wires. Then on the other end, their chips have the smarts to find the signal within the vast amounts of noise created by the wires themselves. Freescale & Xilinx are a bit further up the food chain, they make more programmable chips that can be positioned between Aquantia & Cisco’s switch logic, or the Intel processor in your computer.

So why did Cisco push to form the NBase-T Alliance, what do they gain from this investment? It turns out that improvements in Wireless networking are behind this, and Cisco has a large wireless business. In commercial environments, wireless access points now use a wider range of frequencies in parallel so they can service more of our wireless devices. These access points are pushing the limits on the back end with what GbE is capable of. Since most enterprises are already wired with Cat5a or Cat6 rewiring to support 10GbE would be very expensive. Hence the drive towards NBase-T.

The question though is how about performance desktop users? Folks doing video editing, simulation, or anything that is data intensive could easily push well beyond GbE. We’re now starting to see Apple & others ship 4K resolution desktop computers and displays. These devices can be huge data consumers. What’s the plan for supporting them beyond GbE? The answer still appears to be 10GbE, but time will tell.

Four Reasons Why 10GbE NIC Design Matters

scottcschweitzer networking

Apple, Ferrari, & Dyson have all proven the immense power artful design has over influencing our purchasing decisions. Does this really apply to those products that we don’t see, is design just as critical? Yes, and just last week a master mechanic who’d just rebuilt my friend’s Transmission explained why. He then handed me the failed Ford part, which was made out of pressed steel, and explained that it typically lasted 200K miles. He then showed me the equivalent Chevy part which was cast, and considerably heavier, and rarely failed. So why doesn’t Ford use a cast metal part? Simple, one of Ford’s key design criteria is weight. This enables Ford to more easily meet their Corporate Average Fuel Economy (CAFE) regulations that establish the ratio of high MPG to low MPG vehicles a company can sell. So to save two pounds on this part, perhaps a 1/100 of a MPG, Ford intentionally is designing their truck transmission to fail at 200K miles.

Well what about 10Gb NIC design, perhaps the third most important subsystem in your server behind the CPU & memory, how is this important? For this discussion, we’ll compare Solarflare’s SFN7002F to Intel’s X520-DA2 (which uses Intel’s 82599 controller). Solarflare’s key design criteria is low latency, a fancy way of saying reducing the time it takes to get your data into & out of main memory. Contrast this to Intel whose focus is on producing a general purpose commodity product that is designed to meets the widest range of requirements. We’re going to look at four areas that highlight significant performance differences between these two approaches: transmit & receive queues, MSI-X interrupt vectors, Receive Side Scaling (RSS) queues, and physical/virtual functions (a foundational approach to supporting virtualized computing).

First we need to look at what a network interface card (NIC) really does. A NIC receives information from an external network, and places it into your computer’s memory. It also takes information provided to it, and places that on the network. Solarflare’s approach to networking has been honed over the past five years by servicing the financial markets of the world, the folks who dollarize every nanosecond of the trading day. As such Solarflare has 1,024 transmit & receive queues, or Virtual NICs (vNICs) connected to each 10GbE port. On the Ethernet controller chip Solarflare has also placed a layer-2 network switch in front of those 1,024 vNICs. This network switch can use the packet’s VLAN tag to intelligently steer packets to the proper vNIC assigned to a given VLAN. While Intel on the other hand has only 128 receive/transmit queues attached to each port, or 1/8th of what Solarflare has committed.

Message Signaled Interrupts (MSI-X), is a very common way to inform the processor that data is waiting at an I/O device to be picked up. With PCI Express, we no longer have dedicated hardware interrupt request lines, so I/O devices have to use a shared messaging interface to inform the processor data is waiting. Solarflare supports 1,024 MSI-X interrupts compared to Intel’s 128. Again, this is 1/8th the underlying infrastructure necessary for passing high performance data to the host CPU complex. All of these numbers are per port.

As computers moved from one processor chip to two, then from single core chips to now 18 cores/chip (Intel) the challenge has always been mapping pathways from I/O devices directly to these processor cores. One of the most efficient mechanisms for linking cores to ethernet receive queues is a process known as Receive Side Scaling (RSS). On Intel servers PCI Express slots have an affinity for a specific CPU socket. So for optimal performance you align your 10G ethernet NICs to utilize specific CPUs by the PCI Express slot you install them into. Suppose for example you have a state-of-the-art Haswell dual socket server, and each socket has an 18 core processor. For optimal performance you might install two dual port 10GbE adapters, one in a slot that maps to CPU socket 0, and a second in a slot that maps to CPU socket 1. With this approach you can then achieve peak network performance on your server. There’s a problem though, Intel’s 82599 controller only supports 16 RSS queues per port so two cores on each of your sockets will receive less then optimal performance, as their traffic has to be routed through other cores. Solarflare’s controller on the other hand has 64 RSS queues per port, and can easily spread traffic over multiple paths to every core in your server.

Today many computers rely on virtualization to fully utilize all the resources of the server. To do this network adapters support what are called Physical & Logical functions. Physical Functions (PFs) are a method for exposing to the Virtual Machine’s Hypervisor to what is essential a fully complete physical instances of the network adapters. Solarflare supports 16 Physical Functions while Intel only supports two. Virtual Functions (VFs), are a method for creating full virtualized NICs, Solarflare supports 240 while Intel only has 128. In testing with 32 VMs running Solarflare has demonstrated that it delivers 18% greater overall performance. Note that these PF & VF numbers are per adapter.

Earlier we mentioned that Solarflare’s NICs were designed around latency, yet we’ve not covered latency. Solarflare’s generic kernel device driver that runs on their commodity SFN7002F adapter delivers sub 4 microsecond latency for a 1/2 round trip. Note a half round trip is a single send & receive combined. Intel with their generic driver is more than double this. Furthermore, Solarflare also sells an optional driver called Open Onload that further reduces latency to under 1.7 microseconds!  When it comes to overall server performance, network adapter latency really does matter.

So next time you’re selecting components for a server deployment please consider what you’ve learned above about 10GbE NICs, and “Choose Wisely.”

Linux Network Bandwidth Performance Tuning

scottcschweitzer networking

Solarflare provides a whole Adapter User’s Guide on how to install & configure their 10GbE & 40GbE adapters, but let’s face it who reads manuals?  We all do a Google search for what we need. Anyway, for those seeking a quick recipe to improve their bandwidth here are some suggestions.

First, before you start tuning your configuration you should become familiar with Netperf or Iperf, two of the most well known tools for testing network performance. Pick one of these, install it, and get some benchmarks for your system. Then as you make each change, retest your system performance to see if things improved or got worse.

I’ll assume that you’ve installed and wired up the adapter successfully. Next I’d suggest you confirm that the interface is recognized by the system through the use of this command:

“dmesg /var/log/messages | grep eth”

Then you should validate that the adapter is in the proper PCIe slot with the right generation bus, note it should show a width of x8 or greater. Solarflare 5xxx & 6xxx series cards are PCIe Gen2 (Link: Speed 2.5Gb/s) & 7xxx cards are Gen3 (Link: Speed 5.0Gb/s). This command will display how Linux views the adapter:

“lspci -d 1924: -vv”

If your system supports “Intel QuickData”, chipset based data movement designed to offload copies from the host CPU, you may want to enable these drivers with:

“modprobe ioatdma”

Next, since we’re focused on bandwidth MTU size is important. Check with your network admin to see if the switches you’ll be using have been configured for jumbo frames. If so you can change the MTU size with:

“/sbin/ifconfig mtu “

Where X is the ethernet interface your testing on and is the largest packet size supported on your network. So for example:

“/sbin/ifconfig eth3 mtu 9000”

Next try:

“/sbin/service irqbalance stop”

Consider adding the following lines to your “/etc/sysctl.conf” file:

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 250000

Then execute this command to enable this new settings to take effect:

“sysctl -p /etc/sysctl.conf”

Next consider turning off interrupt coalescing with:

“ethtool -C adaptive-rx off”

Where is the interface you’re testing with.

Hopefully these suggestions have helped improve your overall 10GbE & 40GbE network bandwidth.

Building a Mobile 10G Capture Platform

scottcschweitzer Capture, networking

Someone asked Tuesday if there exists a mobile platform for capturing traffic on a 10GbE link for analysis in real time. Although I’d not heard of anyone selling such a device or bundle I thought it might be an interesting exercise to outline the parts required, estimate the cost, and review what one might need to install.  Note this is all theoretical as I’ve not the parts or the lab at this time to test this out. Also, and this is important, everything below could also be used to build an inexpensive packet generator for network testing. Here are further instructions for doing this activity as well.

With regard to equipment, the first criteria would be a laptop with a high-performance Core i7 processor with plenty L3 cache, considerable main memory, and sporting at least one Thunderbolt port. I’m partial to Apple platforms, but recently HP and others have jumped on the Thunderbolt bandwagon.  Here are several options:

  1. Apple Macbook Pro 15″ with Retina Display, 2.8Ghz, Quad-core Intel Core i7, Turbo boost up to 4Ghz, 16GB memory, 1TB PCIE Flash disk. $3,199 from Apple.
  2. HP ZBook 15″ with Quad-core Intel Core i7-4800MQ, 32GB memory, 256GB SATA Flash Drive. $3,810 from CDW.
  3. Lenovo Thinkpad W540, 15.6″ display, Intel Core-i7-4700MQ, 8GB of memory, 500GB of disk, from Lenovo for $1,119.
While the Apple platform has only half the memory it has four times the disk, with a high-speed PCIe flash interface that is considerably faster than the SATA interface used by HP, so all things should balance out. I added the Lenovo as it too has a Thunderbolt interface, but with only 8GB of memory, and a spinning disk the overall system performance with regard to capture will be impacted, consider this the solution on a budget. After some additional checking the Thinkpad W540 can be easily be upgraded to 16GB for $166. Since this system has four memory sockets you can actually buy two of these kits and run it up to 32GB, which for what would then be a $1,500 laptop would be pretty sweet.
Next, we need an enclosure with an internal PCIe interface to house a 10GbE capture card.  My favorite is the mLogic mLink which sells for $399, and can be purchased from several resellers or directly from Apple.  This enclosure has a PCIe 16 lane socket inside, but only 4 lanes of PCIe Gen2 are actually wired up, which is fine. Thunderbolt is a 10Gbps connection and 4 lanes of PCIe Gen2 is theoretically 16Gbps, but after overhead is more like 12Gbps.
For a capture card, I’d use the Solarflare SFN7122F which can be purchased from CDW for $1,055. This is a dual port 10G card that includes the necessary Open Unload license so you can also run Solarflare’s SolarCapture Pro capture driver (SFS-SCP) which is also available from CDW for $233. Finally, if you want to leverage accurate time stamping of packets via Precision Time Protocol you should buy a PTP (SFS-PTP) license also from CDW for $194.
Finally, every Boy Scout knows you should always be prepared & carry you own two meter 10GbE Direct Attach cable, also $82 from CDW. Ok, now for setup…
First, install Linux on the laptop, these are the supported versions of the capture driver: RHEL 5 & 6, SLES 10 & 11. I’d suggest something with a 2.6.32 or newer kernel.
After making yourself comfortable with the system, installing optional stuff, customizing, updating everything, etc…  You’ll need to visit the SolarCapture support page at Solarflare to install the capture driver & supporting code. First, you need to install Open Onload 201310-u1 or newer. I’d suggest at least 201405-u1. For good measure, I’d also install the Linux Utilities RPM on this page (version 4.1.2.1009 or newer).  Finally, there is the SolarCapture SDK (version 1.3.0.31).  All this covered in the SolarCapture Pro User’s Guide (SF-108469-CD) which can also be found on the same webpage. That includes setting up and configuring the software.
So for roughly $5K you can build a pretty robust mobile workstation that can record 10G traffic at wire-rate…

Your Server as the Last Line of Cyber Defense

scottcschweitzer Uncategorized

Here is an excerpt from an article I wrote for Cyber Defense Magazine that was published earlier today:

Since the days of medieval castle design, architects have cleverly engineered concentric defensive layers along with traps, to thwart attackers, and protect the strong hold. Today many people still believe that the moat was a water obstacle designed to protect the outer wall, when in fact it was often inside the outer wall and structured as a reservoir to flood any attempt at tunneling in. Much like these kingdoms of old, today companies are leveraging similar design strategies to protect themselves from Internet attackers.

The last line of defense is always the structure of the wall, and guards of the castle keep itself. Today the keep is your network server that provides customers with web content, partners with business data, and employee’s remote access. All traffic that enters your servers comes in through a network interface card (NIC). The NIC represents both the wall and the guards for the castle keep.  Your NIC should support a stateless packet filtering firewall application that is authorized to drop all unacceptable packets. By operating within both the NIC, and the kernel driver, this software application can drop packets from known Internet marauders, rate limit all inbound traffic, filter off SYN floods, and only pass traffic on acceptable ports. By applying all these techniques your server can be far more available for your customers, partners, and employees.

For the rest of the article, with several cool sections of code that explain how to protect your server, please visit Cyber Defense Magazine.

A First Generation Firewall in Your NIC

scottcschweitzer networking, Security

Earlier this year Solarflare released a software driver for their line of Flareon adapters called SolarSecure. SolarSecure is a stateless packet filtering engine with a rich set of features that maps almost perfectly into what Wikipedia has defined as a first generation packet filtering firewall. This is meant to separate it from the more common second generation state-full firewall & third generation application level firewalls that have become the more commonly accepted definition of a firewall today.

Earlier this morning someone asked me how SolarSecure might help a huge cloud service provider, say one embroiled in a celebrity scandal.  To fully leverage SolarSecure to make production Internet servers more secure there are at least five potential use cases, and they are:

1. SolarSecure could be used to augment existing firewalls by providing a high bandwidth front end method for rate limiting inbound traffic from chatty address, ex. no more than 10 packets in 10 milliseconds from any specific source (anything over that will be dropped) while also filtering by TCP address. Here is a sample configuration file that one could use to do just this.

2. On all Internet facing interfaces, one should also check the TCP flags to thwart a SYN flood attack. SYN floods make up just over 30% of the attack traffic every day. Another configuration example can be found here that addresses this use case.

3. Also for all Internet facing ethernet interfaces, one should use SolarSecure in conjunction with Norse’s Darklist, or a similar provider, to do blacklist filtering to ensure that known bad actors on the Internet can’t impact my systems at all by dropping their traffic immediately. Again here is some code demonstrating IP address filtering.

4. As a method within the DMZ to tightly control what every server port can be used for.  For example, Internet facing ports should only pass traffic for permitted UDP/TCP ports that are specific to the service that system offers, if it is a web server then white list the system to deny (drop all traffic) on that port that is NOT port 80.  Also for backend service interfaces on that same system white list both the addresses & ports permitted on each & every ethernet interface to ensure that sideways attacks from other DMZ systems that might have been compromised is locked out. Here is an example of how to drop SSH requests to an interface.

5. Finally, HTTP request filtering. Looking at the actual requests coming into the server & only accepting ones that meet the objectives of the server itself.  Hackers will attempt to get in via the web server interface, if SolarSecure is only passing the web server specific types of pre-approved requests then the web server is much less exposed. One final example is here.

If you have any additional questions or comments, don’t hesitate to reach out to me.

A 10G Bouncer for your Network Server Ports

scottcschweitzer networking, Security

We’ve all been to an exclusive club where we may have been denied entry because we weren’t on “the list.”  Does your server have a “list?” It should.

Friday the Department of Homeland Security informed us that over 1,000 US business have been hit by a new hacker tool called “Backoff.” UPS it seems got the most negative press on this one, and “Backoff” had ONLY compromised 1% of their US stores, 51 in total. Reporting the cyber break in made nationwide news on Friday.  So this week UPS is the new face of cyber victims, for the time being replacing Target who suffered an estimated $2.5B loss due to a similar incident last winter.

Unlike Target, which started with weak security on an HVAC system, UPS was hit by a very sophisticated piece of software called “Backoff.” This hacker tool contains a key logger, memory scraping (yes it crawls through memory looking for userids, passwords, system names, credit card info, etc…) and it plays several cute tricks to immunize itself from removal. So how do you minimize & possibly eliminate your servers from being hit by “Backoff?” One method would be to install a white list based security firewall in your server, essentially putting a bouncer on your server’s network port.

This network server bouncer would then only let systems enter your server that come in from places on the list, and they would only be allowed in via communications ports they were pre-approved for. In other words, say you have a web server on an internal address of 10.0.0.77 and you’ve configured it to talk to your database server on port 7940.  Your database server also has this network server bouncer running, and he’s using a whitelist that says: allow 10.0.0.77:7940.  Now suppose your web server was hacked the only possible way the attacker could jump from the web server to the database server would be through port 7940.  All the other ports: telnet, ssh, ftp, etc… from 10.0.0.77 don’t exist, those paths from the web server to the database server are unavailable to the attacker because they aren’t on the whitelist. To actually get into the database server the attacker would have to communicate as if he were a database server himself because that is all the database server will permit on port 7940.  Also remember that HVAC server that brought down Target, it wouldn’t be on the list at all to access any of your core business servers so the sideways attack Target experienced would be impossible.  Your data, on these servers at least, would be considerably more secure.

Taking this one step further, as the admin you could then use this whitelist capability on each ethernet server port to logically wire together your infrastructure such that only systems, and admin workstations that should talk with each other on predetermined, acceptable, ports can. A typical web server might have a whitelist that contains a pool of database servers, on a fixed set of ports, a management server or two, also talking on a different set of permitted ports, and a management workstation or few that also can only communicate over a very narrow band of ports. This would all be enforced in the network server adapter’s hardware that was executing the white list, being the bouncer, so ANY attempted access is stopped before it reaches the operating system.

Does such a magical network server bouncer with a white list on his clipboard exist today, yes.  It’s SolarSecure running on Solarflare’s Flareon network server adapters. With it you can specify a list of acceptable individual IP addresses or ranges of IP addresses that can communicate with a server on a given ethernet interface.  You can then filter off all communications on unapproved ports, in a second layer of filtering.  To learn more post something here, or drop me an email.

Building an Inexpensive Performance Packet Generator

scottcschweitzer Uncategorized

You know that nice feeling you get when someone surprises you with a feature you weren’t expecting, but that totally change the way you use something. Like when your son told you about that free HBO app so you can now watch HBO on your iPad.” Well, recently Solarflare released an update to SolarCapture Pro (SCP V1.3) with just such a feature, replay.

On the surface replay sounds rather hum drum, you can replay libpcap files out to an Ethernet interface. So for the Uber nerds out there yes, you can plug this into Ostenato and make a poor man’s high-performance Ixia on the cheap under $2,250 (you’ll need an SFN7122F & the software SFS-SCP) plus the cost of your server.

If one considers that Solarflare provides the highest performance network adapters currently available for both 10GbE & 40GbE.  This replay feature could be extremely powerful. For example, someone could load a server up with memory, spin up one or more very large libpcap files then use the following command to blast them into their network at wire-rate.

solar_replay pps=1.5e6 prebuffer repeat=512 eth2=play1.pcap eth3=play1.pcap

In this example replay will sustain 1.5 million packets per second (Mpps), note this rate can be as high as 14.8Mpps if your pcap file is all small packets. Next before the replay actually starts play1.pcap will be “prebuffered” meaning that it will be loaded into memory before the replay begins so that disk performance won’t be a factor in the playback. Next, the replay will loop 512 times.  Finally, it will replay the same buffer out both 10GbE or 40GbE ports on the adapter, eth2 & eth3.

So what will this look like? Simple a storm of packets on two interfaces that are hopefully attached to different switches in your infrastructure. Note that the packet rate is actually limited by the size of the packets.

Additionally, you can pin the replay to specific cores, increase the number of buffers, adjust port & time ranges of what you want to replay from the pcap files, and throttle the rate to a multiple of the initial capture speed.

This is by far the most advanced replay capability available today on an ASIC based network adapter. SolarCapture is extremely powerful, and this sample just scratches the surface of what it is capable of.

If you’re interested in taking SolarCapture out for a test drive, or just want to learn more feel free to contact me, or reach out directly to Solarflare.

Enabling Servers to Defend Against a DDoS Attack

scottcschweitzer networking, Security

When was the last time you went into a travel agent or actually bought a ticket at the airport? If AA.com goes down they start canceling flights. In one catastrophic single day server outage, last April American Airlines canceled 970 flights and delayed 1,078 more.  Internet server availability is just as critical to your business.

An industry report published last week stated that over the past 12 months the bandwidth consumed by Distributed Denial of Service (DDoS) attacks worldwide had grown by 72%.  How is this even possible? Simple, the percentage of total cloud & infrastructure servers that have been compromised and turned into bot nets has soared. As recently as last week Amazon’s cloud service had been comprised.  So what chance do any of us have in protecting the servers we use to run our business?

In addition to the traditional defense in depth technologies, one can add another layer of protection by placing a hardware firewall on the server itself. This is not a new concept, back in 2002 3Com sold the 3CR990 which offered an optional firewall on their current 10/100Mbps ethernet controller. Few companies employ this technique because the hardware is rare, and providing a quality firewall at 10GbE & 40GbE speeds if often believed to be expensive. Today only two companies offer this capability in their 10/40GbE server network adapters in the form of a packet filtering engine, and Solarflare is the leader.  Both take a slightly different approach to solving the problem, but the ultimate objective is the same:  providing a server with the last line of defense. So how do these systems work?

Solarflare leverages specialized packet filtering circuits designed into their Ethernet Controller ASIC on their network server adapter. They then crafted a complete distributed solution called SolarSecure that leverages both their ASIC & their user space driver. Through examples, we’re now going to take a brief tour of what Solarflare offers with SolarSecure, and how it can be used to protect your servers.

SolarSecure relies on a pseudo-microcode to define filters and load them into the distributed solution, some filtering is done in the network adapter while other filtering is handled in the user space driver. To learn how SolarsSecure works we’ll explore the following three areas: rate limiting, white/black listing, and SYN cookie support.  There are a number of other interesting options on which one can filter, but these are the most commonly requested, for more examples you’re welcome to check out chapter 7 of the User’s Guide.

Here is a complete example of a configuration file for rate limiting written using this pseudo-microcode, the User’s Guide explains each line of this program, its shown here though to provide a complete example:

set_max_channels 4
set_default_action accept
set_max_objects 30
set_max_minaddrs 30

start_code
accept:
load 1 r0
stop
reject:
load 0 r0
stop
start_rate_limit:
inline_state p1 rate_limit –
test_rate_le p1 0 10pkts 10ms
jmp_if_not reject
jmp accept
end_code

That the majority of the magic for this configuration file happens in this single line:

test_rate_le p1 0 10pkts 10ms

In this example every sender “p1” is limited to transmitting up to 1000 packets/second, anything over that in any given second will be dropped.  To be exact this really means 10 pkts ever 1/100th of a second. This will dramatically reduce the impact of unknown bad actors on your infrastructure.

For white or black lists you can simply define a lookup table with the following structure, note this is a line from a typical black list:

ip4tbl_insert badnets172.16.128.0/21 reject

Then when this line of pseudo-code is executed:

lookup bad_nets pkey

If the inbound IP address and port “pkey” matches any entry in the “bad_nets” table, say 172.16.128.0/21 the packet will be dropped.

If you’re only white listing a port on the server you might use:

ip4tbl_insert ourservers 10.1.1.200/22 accept

In the report mentioned above, over 25% of the attacks that occur are SYN based.  To thwart this you can define a configuration file that will easily drop attackers using this attack vector.  Here are several lines of pseudo-code that demonstrate this point:

load_tcp_flags r2
test_mask_match r2 syn syn
jmp_if_not accept
jmp reject

Note that “test_mask_match” is used to see if the “syn” flag is set for this packet, and if it is, then register “r1” is set to true. Then the conditional “jmp_if_not” uses register “r1” to make the jump.

The SolarSecure filter engine is extremely powerful, and these samples just scratch the surface of what it is capable of. If you find this really interesting you should seriously consider looking reading the User’s Guide.

If you’re interested in taking SolarSecure out for a test drive, or just want to learn more feel free to contact me, or reach out directly to Solarflare.