A customer recently requested some 40GbE performance tuning advice, and this is what we shared with them.
Solarflare provides some initial tuning advice in our “Solarflare Server Adapter User’s Guide” Section 3.24 “Performance Tuning on Linux”. The User Guide can be downloaded from https://support.solarflare.com/ -> Downloads as part number SF-103837-CD. This is frankly the best place to start.
The biggest gain for throughput will come from reducing the CPU load when transferring data and this done in two ways. The first is to reduce the number of packets being sent/received by increasing the size of the MTU for the interface. This does require all the network devices to be configured so as to support this larger size, it will automatically negotiate down if necessary. You can set the larger size using the following command, where is the interface name:
# ifconfig mtu 9000
We support up to 9216 with our adapters but the 9000 value is the most commonly used by network equipment.
The second improvement can be done by increasing the number of packets processed each time the system processor is interrupted. This is done using ‘ethtool’ to increase the interrupt moderation time and will increase latency but reduce CPU utilization. You can do this with the following command:
# ethtool -C rx-usecs 100
The default value is 60 (or 56 on more recent firmware versions) and values up to 200 may be used. The benefits from going beyond this value experience diminishing returns.
With this second method, we also recommend disabling interrupt moderation where the kernel will tune the value used to somewhere between 1 and the value specified. In theory, this can be useful if you are doing a mixture of large and some small, latency sensitive transfers but the algorithm isn’t great and can get stuck at either end of the range. To disable this, you can use the following command:
# ethtool -C adaptive-rx off
This can be combined with the previous ‘ethtool’ command into a single one if desired:
# ethtool -C <if_name adaptive-rx off rx-usecs 100
These two changes should give increased throughput, most probably limited by the CPU processing capability of the incoming packets. It should be noted that if you use multiple streams for TCP or different hosts for UDP/TCP you can get better performance because the processing will be done by more cores.
— Richard Drinkwater, Solarflare