Security

Enabling Servers to Defend Against a DDoS Attack

scottcschweitzer networking, Security

When was the last time you went into a travel agent or actually bought a ticket at the airport? If AA.com goes down they start canceling flights. In one catastrophic single day server outage, last April American Airlines canceled 970 flights and delayed 1,078 more.  Internet server availability is just as critical to your business.

An industry report published last week stated that over the past 12 months the bandwidth consumed by Distributed Denial of Service (DDoS) attacks worldwide had grown by 72%.  How is this even possible? Simple, the percentage of total cloud & infrastructure servers that have been compromised and turned into bot nets has soared. As recently as last week Amazon’s cloud service had been comprised.  So what chance do any of us have in protecting the servers we use to run our business?

In addition to the traditional defense in depth technologies, one can add another layer of protection by placing a hardware firewall on the server itself. This is not a new concept, back in 2002 3Com sold the 3CR990 which offered an optional firewall on their current 10/100Mbps ethernet controller. Few companies employ this technique because the hardware is rare, and providing a quality firewall at 10GbE & 40GbE speeds if often believed to be expensive. Today only two companies offer this capability in their 10/40GbE server network adapters in the form of a packet filtering engine, and Solarflare is the leader.  Both take a slightly different approach to solving the problem, but the ultimate objective is the same:  providing a server with the last line of defense. So how do these systems work?

Solarflare leverages specialized packet filtering circuits designed into their Ethernet Controller ASIC on their network server adapter. They then crafted a complete distributed solution called SolarSecure that leverages both their ASIC & their user space driver. Through examples, we’re now going to take a brief tour of what Solarflare offers with SolarSecure, and how it can be used to protect your servers.

SolarSecure relies on a pseudo-microcode to define filters and load them into the distributed solution, some filtering is done in the network adapter while other filtering is handled in the user space driver. To learn how SolarsSecure works we’ll explore the following three areas: rate limiting, white/black listing, and SYN cookie support.  There are a number of other interesting options on which one can filter, but these are the most commonly requested, for more examples you’re welcome to check out chapter 7 of the User’s Guide.

Here is a complete example of a configuration file for rate limiting written using this pseudo-microcode, the User’s Guide explains each line of this program, its shown here though to provide a complete example:

set_max_channels 4
set_default_action accept
set_max_objects 30
set_max_minaddrs 30

start_code
accept:
load 1 r0
stop
reject:
load 0 r0
stop
start_rate_limit:
inline_state p1 rate_limit –
test_rate_le p1 0 10pkts 10ms
jmp_if_not reject
jmp accept
end_code

That the majority of the magic for this configuration file happens in this single line:

test_rate_le p1 0 10pkts 10ms

In this example every sender “p1” is limited to transmitting up to 1000 packets/second, anything over that in any given second will be dropped.  To be exact this really means 10 pkts ever 1/100th of a second. This will dramatically reduce the impact of unknown bad actors on your infrastructure.

For white or black lists you can simply define a lookup table with the following structure, note this is a line from a typical black list:

ip4tbl_insert badnets172.16.128.0/21 reject

Then when this line of pseudo-code is executed:

lookup bad_nets pkey

If the inbound IP address and port “pkey” matches any entry in the “bad_nets” table, say 172.16.128.0/21 the packet will be dropped.

If you’re only white listing a port on the server you might use:

ip4tbl_insert ourservers 10.1.1.200/22 accept

In the report mentioned above, over 25% of the attacks that occur are SYN based.  To thwart this you can define a configuration file that will easily drop attackers using this attack vector.  Here are several lines of pseudo-code that demonstrate this point:

load_tcp_flags r2
test_mask_match r2 syn syn
jmp_if_not accept
jmp reject

Note that “test_mask_match” is used to see if the “syn” flag is set for this packet, and if it is, then register “r1” is set to true. Then the conditional “jmp_if_not” uses register “r1” to make the jump.

The SolarSecure filter engine is extremely powerful, and these samples just scratch the surface of what it is capable of. If you find this really interesting you should seriously consider looking reading the User’s Guide.

If you’re interested in taking SolarSecure out for a test drive, or just want to learn more feel free to contact me, or reach out directly to Solarflare.

A Fire Sale, Napolitano, and Cyber 9/11 is it Probable or Just Possible?

scottcschweitzer Security

This article was originally published in January of 2013 on 10GbE.net. 

We’re going to zero in on a low profile Cyber Warfare story that I’m surprised saw such little media coverage. Thursday U.S. Secretary of Homeland SecurityJanet Napolitano talked publicly at the Wilson Center (a DC Think Tank) on the looming threat of a “Cyber 9/11” with the potential for a hurricane Sandy like outcome. Wow, now that’s a current image that evokes fear. When I first learned of the story Friday morning my initial impression was that perhaps earlier in the week Secretary Napolitano had picked up Bruce Willis’s 2007 movie “Live Free or Die Hard” in the discount bin at the Walmart and watched it a bit too late one night. The skeptic who brought this story to my attention pointed out that it was perhaps a veiled attempt to grab more budget and create even more regulation & government oversight. 

Now I’m not just some random blogger with an opinion, and no street cred to back it up. I started hacking back in 1983 as a hobby with my TRS-80 Model III when it was still cool, and legal. “War Games” had just been released, and ethics were the only rules. Later that year I was hired by IBM Research, a company I remained with for 16 more years. During that time I’d tied many different systems together. At one point I worked side by side writing code for Davis Foulger to link Prodigy, Compuserve, AOL, and IBMPC (the IBM internal social network) together so IBMers could answer customer questions from anywhere using a single internal system. In the late ’90s I designed and created a solution for the company that linked many different systems together and in 1999 it collected $2B in revenue. This effort earned me an IBM Outstanding Technical Achievement Award, but to pull this off we also created a state of the art hacking tool for Lotus Domino to certify that our system was secure, which almost landed us in jail. So I know a bit about cyber security, and linking together different systems.
 
So is a “Fire Sale“, a “Cyber 9/11”, probable or Just possible? Possible, definitely. Probably, within a reasonably defined scope, say a single city, and it would require substantial capital to motivate and assemble a team with the proper skills. The team would need to touch every power grid, water, sewage treatment, mass transit, oil refinery, financial trading, etc. system within the scope. These systems don’t need to be Internet connected, people are always required to service, and inspect them, and internally they are often networked together. One would simply have to crack the case off a $100 Verizon 4G USB Cellular modem, then 3D print a new case branded to match the software product that industry runs so it would be camouflaged as a USB software license key. It could go unnoticed forever unless they did an active RF sweep. Prior to inserting this USB stick another containing a modified version of Operation Olympic Games (Stuxnet) could be used to open the security hole and inject the software, then the 4G connectivity would be used to continuously report back, and eventually carry the trigger message to bring the system down. The right mix of social hacking, posing as contractors doing repairs, or government regulators conducting inspections, even OEM system engineers doing required service could easily facilitate this objective.
 
George W. Bush authorized Operation Olympic Games in 2006, during Homeland Security Secretary Michael Chertoff’s time in office, why wasn’t this considered and addressed seven years ago? Secretary Napolitano has been in office three years now, and all of the sudden it’s a priority. Our government could form its own cyber security Tiger Team. This team could then covertly attack key industries, and report back to these firms their findings, and the appropriate counter measures. These attacks would simply demonstrate vulnerabilities, not bring down systems. Since the early ’90s, IBM has had it’s own Security Tiger team, “Global Security Analysis Lab“, that regularly trained key three letter agencies. I met with their leader in the late ’90s to provide him with a copy of our IBM Internal use only Lotus Domino ethical hacking tool.
 
A “Cyber 9/11” can be prevented, but regulations aren’t the answer, that process is far too slow and rigid. An elite group of patriotic, well compensated, cyber professions could do in two years what the administration has ignored in the past seven years. Only time will tell.

Zombies vs. 10GbE ICBMs as Internet Arsenals

scottcschweitzer Security

This article was originally published in December of 2012 at 10GbE.net.

The image of hundreds of thousands of zombies attacking is scary, but then so is the image of a large city being vaporized by a purpose built intercontinental nuclear warhead. When I say 10GbE ICBMs I’m loosely defining a new class of cyber weapon, crafted from COTS parts that are purpose built to deliver massive digital payloads across the web from a tightly managed, yet geographically disperse, a collection of launch platforms, an Internet Converged Binary Munition.  The cyber warfare equivalent of our current arsenal of 450 Minuteman-III ICBMs spread across Wyoming, Montana & North Dakota.  Yes, the US still has hundreds of ICBMs fueled up and ready to fly.  Conversely, China is a big proponent of leveraging zombie-bot networks, co-opted personal computers in homes universities and businesses around the world, as their primary offensive weapon for industrial espionage, and their cyber Cold War with the rest of the world.  

The moral attitude of the United States, our culture, and a constitutional amendment prohibit us from adopting the zombie approach.  Anyone familiar with battle understands that you rarely win because you have more soldiers, but that you win more often when the logistics are in place to support them.  Individually zombie bots have erratic performance, and as an army, they are unpredictable because control is limited, and access can easily be terminated by a third party (the owner, ISP or law enforcement).  Also, the foundation of your zombie army is based on the theft of processing cycles, electricity, and network bandwidth.  Eventually, most thefts are detected, prevented and reported.  Furthermore, the third amendment to our Constitution bars the government from quartering soldiers, including digital ones and even remote physical ones, on private property.  Personally, I find the whole concept of zombies bots amateurish and offensive.
In contrast, the US is exploring purpose built cyber weapons platforms utilizing high-performance 10Gb Ethernet technology that can be well over 10,000 times more efficient, per link than even the best zombies.  Today millions of zombies are inhabiting people’s home computers all around the world.  As an example, in July the third largest zombie bot network, Grum, was taken off-line and it contained 120,000 unique network addresses.  This year the average US home internet speed is 7Mbps. ISP’s as a rule throttle upload bandwidth to 1/10 that of download so the average home has at best 0.75Mbps of outbound capacity available to a zombie.  A single server using several multi-port 10GbE adapters could easily be built today using COTS (Commodity Off The Shelf) parts which could deliver a payload equivalent to 250,000 zombies.  That’s twice the size of the botnet mentioned above, all in a single tightly managed server, a logistical battle planner’s dream weapon.  Now imagine a few of these servers per rack, a few racks per ISP, and a dozen or so major ISP locations spread around the world, all for less than the purchase price of a single Joint Strike Fighter (cost $207M/each, and the US is buying 2,443).  Cyber war is here, and DARPA is doing the R&D to ensure the US is prepared to win.

Who Spilt the Milk? The Importance of Attribution

scottcschweitzer networking, Security

This article was originally posted in November of 2012 at 10GbE.net. 

As a parent the moment we hear a glass of milk bounce off the floor we’re conditioned to assign blame, and if necessary meter out punishment. In the real world attribution for a kinetic attack has become much easier given our satellite, ground & ocean sensor networks, and forensic sciences. Today determining who launched an air strike, fired off a missile or attacked an embassy can often be resolved in hours, or worst case days. In the world of cyberspace, the exact opposite is true. Furthermore, even if a nation could unequivocally attribute an attack, international case law for cyber warfare doesn’t exist so justifying a counter strike, cyber or kinetic, is uncharted territory. 

Many techniques and tools exist to intentionally obfuscate the source of an attack. Furthermore, the most sophisticated assaults are designed to morph with each stage of the deployment so the strategy itself is also hidden. This can be done many ways, but the intent is always the same, to further mask the original target and the method of the offensive attack. That way when the forensic computer scientists come knocking all they will find are empty cyber shell casings, of the most generic type, devoid of any usable digital fingerprints. If there are clues, it’s very likely those were planted to frame a believable third party so that retribution will result in a secondary attack perhaps on the actual intended target. Suppose you whack the biggest bully in school on the back of the head, and quickly your friend assigns blame to your enemy, who also just happens to be standing there. Now you know the bully will deliver a far more destructive attack than you ever could so why wouldn’t you leverage such a strategy. If it’s executed properly only you and your buddy will ever really know what happened.
 
Why is attribution important? Because the US, and other countries, have publicly stated that given the void of international cyber warfare case law a significant attack in cyberspace will be met with a declaration of war. Attacking nations of the world have been put on notice to expect both a kinetic and a cyber response. Not responding will be seen as a sign of weakness. So what does a country do though when attribution is murky or impossible? Are they justified in launching their own covert stealthy cyber attack?

Cyber Warfare – The Next Big Networking Market

scottcschweitzer Security

This article was originally published in October of 2012 at 10GbE.net. 

As someone who hacked systems using a 300 Baud modem several decades ago, and who now sells extreme networking technology I find recent Cyber Warfare announcements exciting.  Last month DARPA, the bleeding edge research arm of the defense department, announced PlanX, and it’s funding target of $100M over four years.  Last week US Secretary of Defense Leon Panetta confirmed the US’s desire to lead the world in Cyber Warfare preparedness.  He also stated that Iran has set aside $1 Billion to dramatically improve its position given recent difficulties with Stuxnet and its offspring.  We’ve also seen stories out of China, Israel, and North Korea in the past two months about these states are officially assembling troops of cyber warriors.

So what does this mean for networking vendors?  A whole new market for leading edge products that enable defensive and offensive cyber weapons.  For example, Myricom’s processor based 10Gb Ethernet network adapters have an optionally available software product called Sniffer10G.  This software bundle contains both firmware for the processor on the adapter and a high-performance user space device driver. This package enables lossless wire-rate packet capture and injection.  In the world of kinetic weapons, Sniffer10G is the Kevlar (capture technology) in Dragon Skin Body Armor and the repeating/propulsion mechanism (wire-rate 10GbE injection) behind high caliper rail guns.
 
Sniffer10G comes with some foundational command line tools designed to stress test and demonstrate both capture and injection.  It’s the API though that allows low-level access to the hardware, where Sniffer10G really shines.  Sniffer can flow-hash or clone inbound traffic into one or more (up to 16) different user space queues.  It can also re-inject those queues, or other ones, directly onto the ethernet at wire-rate.
 
Imagine if you will synthetically spinning up a 5GB buffer containing a broad array of packets designed to achieve a specific military objective. Sniffer10G gives you the capability to blast all those packets out to your enemy over a single network port in a whopping four seconds!  With just two dual port cards in a single server and Sniffer10G, this could easily scale to 5GB/second.  At 5GB/s one could easily cripple over 40 traditional servers with Gigabit ethernet links.  As an offensive weapons platform, this could easily be assembled for under $5K.  The most challenging part would be positioning it on the Internet in such a way that the ISP wouldn’t shut you down after your first offensive.  Then again governments can easily bypass these trivial ISP issues.
 
If you want to learn more or discuss this further please don’t hesitate to contact me directly.

Black Hat USA 2012 – Emulex FastStack Sniffer10G Product Demo Shows 2x Suricata Performance Gain

scottcschweitzer Capture, networking, Security

This article was originally published in July of 2012 at 10GbE.net. 

At Black Hat USA this week at Caesars Palace in Las Vegas, Emulex & Myricom will be demonstrating Sniffer10G as it greatly improves Suricata performance by a factor of two. Suricata is a high-performance open source application for creating an intrusion prevention (IPS) or intrusion detection system (IDS). Snort is the leader in this space, but Suricata is the popular new application looking to capture early and mass adoption. While Snort was designed as a single threaded application, Suricata was built from the ground up to support multi-core servers. Sniffer10G takes this one step further, leveraging the processor found on Myricom and Emulex NX adapters to provide multiple capture buffers that can then be pinned to specific CPU cores.

This blog entry specifically outlines how to accomplish a 2X performance improvement for Suricata by using to Sniffer10G. First, you will need to purchase a high-performance dual port 10GbE adapter with Sniffer10G: Myricom dual port 10GbE (P/N 10G-PCIE2-8C2-2S+SNF2) or an Emulex NX adapter (P/N OCe12102-DX-SNF2). One of these cards is required to run Sniffer10G which is also required for performance gain. Both adapters can be purchased for $1,000.00.  This price includes Sniffer10G software. 
For the sake of this posting, we will assume you know how to install the server, the network adapter, and the OS. Prior to deployment, and for testing purposes only we will use a single DA cable to connect together both 10GbE ports. Also, make sure a gigabit ethernet port is connected to the independent server so we can download the software and a test suite. 
 
If you’ve not already downloaded Sniffer10G, then contact help@myricom.com and request a userid and password to access this product. Once downloaded issue the following command to install the product:
# rpm -i myri_snf-2.0.6.50271-2831.x86_64.rpm
 
This should put all the interesting stuff in /opt/snf. At this point, you should validate that you have purchased a card with the Sniffer10G license and that it’s ready to go. To do this issue:
 
# /opt/snf/sbin/myri_license
 
and it should return something similar to this: 
 
2 boards found
NIC 0 serial = 423691
License keys:
395e-e403-6a05-1a8f:2:423691:SNF:V2: # SNF, V2
NIC 1 serial = 423691
License keys:
395e-e403-6a05-1a8f:2:423691:SNF:V2: # SNF, V2
 
The reason it says two boards found is that each port is treated as a separate adapter. Next, you should confirm that the Sniffer10G code is running. The command to do this is myri_start_stop, you can use start, stop or restart. 
 
# myri_start_stop restart
Restarting Sniffer10G
Removing myri_snf
Loading myri_snf
 
Next, confirm that the system thinks things are ok by checking dmesg:
 
# dmesg | grep myri_snf
…
myri_snf INFO: 2 boards found and initialized
myri_snf INFO: eth4: Link0 is UP
myri_snf INFO: eth5: Link0 is UP
 
Now let’s confirm that traffic can pass easily between the ports. Here you’ll need two console windows open to your server, let’s call them Window-A and Window-B. On Window-A run the following command, it will setup your first port to receive traffic forever from the second port:
 
# /opt/snf/bin/tests/snf_simple_recv -p0 -t 1
snf_recv ready to receive
 
On Window-B run this command to generate one billion 60 byte packets on port 1:
 
# /opt/snf/bin/tests/snf_pktgen -p1 -s 60 -n 1000000000
 
Now in Window-A, you should see traffic along the following lines:
 
8650979 pkts (519058740B) in 1.001 secs (8641948 pps), Avg Pkt: 60, BW (Gbps): 4.148
14892539 pkts (893552340B) in 1.001 secs (14876784 pps), Avg Pkt: 60, BW (Gbps): 7.141
14894309 pkts (893658540B) in 1.001 secs (14878894 pps), Avg Pkt: 60, BW (Gbps): 7.142
…
 
Use Ctrl-C in both windows to stop things and in Window-B you should see:
 
Size Mbps Mpps Efficiency
60 7141.26 14.878 99.98%
 
This confirms that Sniffer10G is setup and working perfectly. Now on to Suricata.
 
Here are the steps to install Suricata, and build in the Sniffer10G libraries, note I’m not showing the output of the commands because it can be extensive:
 
 
# yum install file-devel
 
# tar -xvzf suricata-1.3.tar.gz
 
# mv suricata-1.3 suricata
 
# cd suricata
 
#./configure –with-libpcap-includes=/opt/snf/include/ –with-libpcap-libraries=/opt/snf/lib/ –prefix=/usr –sysconfdir=/etc –localstatedir=/var
 
# make
 
# make install-full
 
# cp classification.config /etc/suricata
 
# cp reference.config /etc/suricata
 
# cp suricata.yaml /etc/suricata
 
At this point we should confirm where Suricata is and if it was in-fact built with Sniffer10G. To do that use the following commands:
 
# which suricata
/usr/local/bin/suricata
# ldd /usr/local/bin/suricata | grep snf
libpcap.so.1 => /opt/snf/lib/libpcap.so.1 (0x00007f4359199000)
libsnf.so.0 => /opt/snf/lib/libsnf.so.0 (0x00007f4358b53000)
 
Now you need to edit your “suricata.yaml” file to make use of Sniffer10G. Suppose you have a dual socket system with eight cores per socket, you can then change your “suricata.yaml” file’s pcap section to look like this:
 
pcap:
– interface: eth4
threads: 16
buffer-size: 512kb
checksum-checks: no
 
We are assuming here that eth4 is your snf0 interface, you’ll need to confirm the actual interface names using “ifconfig”. Now to start Suricata and leverage all 16 cores, each with their own 512KB buffer use this command:
 
# SNF_NUM_RINGS=16 SNF_FLAGS=0x1suricata -c /etc/suricata/suricata.yaml -i eth4 –runmode=workers
 
For now, since the same system is both generating traffic and consuming it you should consider cutting this number back to 8 or 12 cores instead of all 16. During our testing, we typically use two systems connected back to back so we’ve not had to balance generation and consumption on the same system. 
 
This above command will start Suricata up, but the problem is where do you get traffic from. If you use Window-A for starting Suricata and Window-B for generating traffic then in Window-B we should first download some reasonable traffic. To do this use the following command from your home directory:
 
 
# mv 54 example.com-3.pcap
 
Then once suricata is running to generate test traffic you can use:
 
# /opt/snf/bin/tests/snf_replay -v -p0 -R 0.18 -i 2500 example.com-3.pcap
Thread 0> Packets: 5122500
Thread 0> Bytes: 1660497500
Thread 0> Rate: 0.27 Mpps
Thread 0> Throughput: 0.695 Gbps in 19.122 secs
 
Attached are three command output files:
 
 
and the two Suricata configuration files that were used for our testing:
 
 
These prove out the 2X performance improvement mentioned in the opening paragraph. It is possible to see gains well in excess of 2X, but those require additional tuning and tweaking of the suricata.yaml file beyond what was discussed above.

World War III is Now in Cyber Space

scottcschweitzer Security

This article was originally published in May of 2011 on 10GbE.net.

“We don’t know who struck first, us or them. But we do know it was us that scorched the sky.” Morpheus explaining to Neo how the final World War between the humans & the machines started.
 
Unlike the first two World Wars, or the final fictitious one is in the Matrix. WW III is being fought today exclusively in cyber space.  It is a silent war, rarely reported in mainstream media, primarily because there’s almost never a human casualty*, and most readers wouldn’t understand it.  Often the combatants have no idea who they’re fighting. If we look at one of the early major battles, Stuxnet, no one has officially claimed to have launched the attack, but the target was laser focused on Iran’s nuclear weapons program. The campaign was successful, and it delayed Iran’s weapons program by several years.
 
Now I’m sure you’re wondering, “Why is this being covered in a blog about 10GbE?”  Simple, all battles require supply lines. Today the supply lines for cyber warfare are moving from GbE to 10GbE.  Sure a lone hacker can use a Low Orbit Ion Cannon (LOIC) to take down an unsophisticated opponent using little more than a laptop, and a decent Internet connection. Frankly, this is synonymous in the real world to someone with a machine gun trying to shut down a store.
 
On the other hand governments, and corporations have substantially more resources.  As we saw with Stuxnet.  For those not familiar with Stuxnet, this was a very carefully engineered offensive cyber weapon delivered in virus form.  It had several different parallel attack vectors to ensure that it reached all the intended targets, which reports suggest that it did.  It contained a precise triggering system and a lethal warhead.   This battle would not have been possible with only a handful of intelligent people.  It took a collaborative effort by several governments, at least one corporation, and several unique pieces of equipment for testing to ensure the weapon’s success.  This went way beyond your garden variety LOIC class assault and defined a whole new category of offensive cyber weaponry.  While the LOIC is a machine gun obtainable by nearly anyone, Stuxnet, by comparison, is a state of the art cruise missile.
 
Now back to 10GbE.  Markets like HFT and HPC are breaking new ground in low latency, high packet rate & wire-rate bandwidth.  They are utilizing advanced 10GbE network adapters and switches, along with sophisticated algorithmic routines.  Today HFT’s are exploring lossless wire-rate 10GbE capture to do real time analysis and simulation.  They capture real trading data in buffers or spin up synthetic ones to emulate the market.  They then inject these buffers into an HFT trading engine to see how it responds.  Is this really any different than simulating 100s or 1,000’s of web surfers?  This technology is crossing over into cyber warfare.
 
All of the above was written over the holiday weekend.  This morning the Wall Street Journal reported that the US Government has announced that The Pentagon has adopted a new strategy that will classify major cyber attacks as acts of war, paving the way for possible real world military retaliation.  “If you shut down our power grid, maybe we will put a missile down one of your smokestacks,” said a military official as quoted in the WSJ.  So if you don’t believe the ongoing war in cyber space is real, then just consider it a prelude to it.
 
* John P. Wheeler III may very well be the first person slain in this cyber war.  He is a former Pentagon official and was a consultant for Mitre prior to his murder.  John was an outspoken proponent of Cyber defense.  In January 2011 he was found dead in a Maryland landfill, the investigation into his death is currently stalled.