Solarflare provides a whole Adapter User’s Guide on how to install & configure their 10GbE & 40GbE adapters, but let’s face it who reads manuals?  We all do a Google search for what we need. Anyway, for those seeking a quick recipe to improve their bandwidth here are some suggestions.

First, before you start tuning your configuration you should become familiar with Netperf or Iperf, two of the most well known tools for testing network performance. Pick one of these, install it, and get some benchmarks for your system. Then as you make each change, retest your system performance to see if things improved or got worse.

I’ll assume that you’ve installed and wired up the adapter successfully. Next I’d suggest you confirm that the interface is recognized by the system through the use of this command:

“dmesg /var/log/messages | grep eth”

Then you should validate that the adapter is in the proper PCIe slot with the right generation bus, note it should show a width of x8 or greater. Solarflare 5xxx & 6xxx series cards are PCIe Gen2 (Link: Speed 2.5Gb/s) & 7xxx cards are Gen3 (Link: Speed 5.0Gb/s). This command will display how Linux views the adapter:

“lspci -d 1924: -vv”

If your system supports “Intel QuickData”, chipset based data movement designed to offload copies from the host CPU, you may want to enable these drivers with:

“modprobe ioatdma”

Next, since we’re focused on bandwidth MTU size is important. Check with your network admin to see if the switches you’ll be using have been configured for jumbo frames. If so you can change the MTU size with:

“/sbin/ifconfig mtu “

Where X is the ethernet interface your testing on and is the largest packet size supported on your network. So for example:

“/sbin/ifconfig eth3 mtu 9000”

Next try:

“/sbin/service irqbalance stop”

Consider adding the following lines to your “/etc/sysctl.conf” file:

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 250000

Then execute this command to enable this new settings to take effect:

“sysctl -p /etc/sysctl.conf”

Next consider turning off interrupt coalescing with:

“ethtool -C adaptive-rx off”

Where is the interface you’re testing with.

Hopefully these suggestions have helped improve your overall 10GbE & 40GbE network bandwidth.