Focus on the New Network Edge, the Server

For decades we’ve protected the enterprise at the network edges where the Internet meets our DMZ, and then again where our DMZ touches our Intranet. These two distinct boundary layers and the DMZ in-between makeup what we perceived as the network edge. It should be pointed out though that these boundaries were architected long before phishing and click-bate existed as part of our lexicon. Today anyone in the company can open an email, click on an attachment or a web page, and open Pandora’s box. A single errant click can covertly launch a platform that turns the computer into a beachhead for the attacker. This beachhead then circumvents all your usual well-designed edge focused defenses as it establishes an encrypted tunnel enabling the attacker access to your network whenever they like.

Once an attacker has established their employee hosted beachhead, they then begin the search for a secondary, server-based, vantage point from which to operate. A server affords them a more powerful hardware system and often one with a higher level of access across the entire enterprise. Finally, if the exploit is discovered in that server, the attacker can quickly revert to their fall back position on their initial beachhead system and wait out the discovery.

This is why enterprises must act as if they’ve already been breached. Accept the fact that there are latent attackers already inside your network seeking out your corporate jewels. So how do you prevent access to your companies most valuable data? Attackers are familiar with the defense in depth model so once they’re on your corporate networks, often all that stands between them and the data they desire is knowing where it is hidden, and obtaining the minimum required credentials to access it. So how do they find the good stuff?

They start by randomly mapping your enterprise network in hopes that you don’t have internal honey-pots or other mechanisms that might alert you to their activity. Once the network is mapped they’ll then use your DNS to assign names to the systems they’ve discovered in hopes that this might give them a clue where the good stuff resides. Next, they’ll do a selective port scan against the systems that look like possible targets to determine what applications are running on them to fill in their attack plan further. At this point, the attacker has a detailed network map of your enterprise, complete with system names, and the names of the applications running on those systems. The next step will be to determine the versions of the applications running on what appear to be the most critical systems, so they’ll know which exploits to leverage. It should be noted that even if your servers have a local OS based firewall, you’re still vulnerable. The attackers at this point know everything they need to, so if you haven’t detected the attack by this stage, then you’re in trouble because the next step is the exfiltration of data.

If we view each server within your enterprise as the new network edge, then how can we defend these systems? Solarflare will soon announce ServerLock, a system that leverages the Network Interface Card (NIC) in your server to provide a new defense in depth layer in hardware. A layer that not only shields it from attack, but it can also camouflage the server and report attempts made to access it. Two capabilities not found in OS based software firewalls. Furthermore, since all security is handled entirely within the NIC, there is no attackable surface area. So how does ServerLock provide both camouflage and reporting?

When a NIC has ServerLock enforcement enabled only network flows for which a defined policy exists are permitted to enter or exit that server. If a new connection request is made to that server which doesn’t align with a security policy, say from an invalid address or to an invalid port, then that network packet will be dropped, and optionally an alert can be generated. The attacker will not receive ANY response packet and assume that nothing is there. Suppose you are enforcing a ServerLock policy on your database servers which ONLY accepts connections from a pool of application servers, and perhaps two administrative workstations, on specific numeric ports. If a file server were compromised and used as an attack position once it reaches out to one of those database servers via a ping sweep or an explicit port scan it would get NOTHING back, the database server would appear as network dark space to the file server. On the ServerLock Manager console alerts would be generated, and the administrator would know in an instant that the file server was compromised. Virtually every port on every NIC that is under ServerLock enforcement is turned into a zero-interaction honeypot.

So suppose the attacker has established themselves on that file server, and the server then gets upgraded to ServerLock and put under enforcement. The moment that attacker steps beyond the security policies executing in that NIC on that server the jig is up. Assuming they’re on the server, once they attempt any outbound network access that falls outside the security policies those packets will be dropped in the NIC, and an alert will be raised at the ServerLock Management console. No data exfiltration today.

Also, it should be noted that ServerLock is not only firmware in the NIC to enforce security policies, but it is also an entire tamper-resistant platform within the NIC. Three elements make up this tamper-resistant platform, first only properly signed firmware can be executed, older firmware versions cannot be loaded, and any attempt to tamper with the hardware automatically destroys all the digital keys stored within the NIC. Valid NIC firmware must be signed with a 384-bit key utilizing elliptic curve cryptography. The Solarflare NIC contains the necessary keys to validate this signature, and as mentioned earlier tampering with the NIC hardware will result in fuses blowing that will corrupt the stored keys forever rendering the both unusable and unreadable.

Today enterprises should act as though they’ve already been compromised, and beef up their internal defenses to protect the new network edge, the server itself. In testing ServerLock, we put a web server protected by ServerLock directly on the Internet, outside the corporate firewall.

Compromised Server Supply Chains, Really?

2018 Was shaping up nicely to become “The Year of the CPU Vulnerability” what with Meltdown, Spectre, TLBleed, and Foreshadow we had something going then along came Bloomberg and “The Big Hack” story. Flawed CPU designs just weren’t enough; now we have to covertly install “system on a chip (SoC)” spy circuits directly into the server’s baseband management controller (BMC) at the factory. As if this weren’t enough today Bloomberg drops its second story in the series “New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom” which exposes compromised RJ45 connectors in servers.

We learned recently that Edward Snowden’s cache of secret documents from five years ago included the idea of adding an extra controller chip to motherboards for remote command and control. Is it astonishing that several years later a nation-state might craft just such a chip? Today we have consumer products like the Adafruit Trinket Mini-Microcontroller, pictured below, at $7USD the whole board is 27mm x 15mm x 4mm. The Trinket is an 8Mhz 8bit Atmel ATtiny85 minicomputer that can be clocked up to 20Mhz, with 8K flash, 512 bytes of SRAM and 512 bytes of EEPROM ($0.54USD for just the microcontroller chip) in a single 4mm x 5mm x 1.5mm package. In the pervasive Maker culture that we live in today, these types of exploits aren’t hard to imagine. I’m sure we’ll see some crop up this fall using off the shelf parts like the one mentioned above.

In the latest Bloomberg story, one source Yussi Appleboum, revealed that the SMC motherboards he found had utilized a compromised RJ45 Ethernet connector. This rogue connector was encased in metal providing both camouflage for the hidden chip and as a heat sink to dissipate the power it consumes. In this case all one would need to do would be to craft a simple microcontroller with an eight pin package, one for each conductor in the RJ45 connector. This controller would then draw it’s power directly from the network while also sniffing packets entering and leaving the BMC. Inconceivable, hardly, the metal covering such a connector is somewhere around 12mm square, similar to the RJ45 on the Raspberry Pi shown to the right, that’s four times more area than the ATtiny85 referenced above. Other micro-controllers, like the one powering the Raspberry Pi Zero, could easily fit into this footprint and deliver several orders more processing power. The point is that if someone suggested this five years ago, at the time of the Snowden breach, I’d have said it was possible but unlikely as it would have required leading-edge technology in the form of custom crafted chips costing perhaps ten million or more US dollars. Today, I could recommend a whole suite of off the shelf parts, and something like this could very likely be assembled in a matter of weeks on a shoestring budget.

Moving forward OEMs need to consider how they might re-design, build, and validate to customers that they’ve delivered a tamper-proof server. Until then for OCP compatible systems you should consider Solarflare’s X2552 OCP-2 NIC which can re-route the BMC through their network ports and which includes Solarflare’s ServerLock™ technology that can then filter ALL network traffic entering and leaving the server. That is provided of course that you’ve disconnected the servers own Gigabit Ethernet ports. If you’d like a ServerLock™ sample white-list filter file that shows how to restrict a server to internal traffic only (10.x.y.z or 192.168.x.y) then please contact me to learn more.

UPDATE: This weekend I discovered the item shown to the right which is offered as both a complete product called the “LAN Tap Pro” for $40 in a discrete square black case or as this throwing star kit for $15 with all the parts, some assembly and soldering required. This product requires NO external power source, and as such, it can easily be hidden. The chip which makes the product possible, but which is not shown, should answer the question of whether or not the above hacking scenario is a reality. While this product is limited to 10/100Mb, and can not do GbE, it has a trick up its sleeve to down speed a connection so that the network can be easily tapped. When it comes to server monitoring/management ports these often do not require high-speed connections so it’s highly unlikely that down speeding the connection would likely even be noticed. The point of all this rambling is that it’s very likely that the second Bloomberg article is true if the parts necessary to accomplish the hacking task are easily available through a normal retail outlet like the Hacker Warehouse.

Visibility + Control = Orchestration

In Taekwondo to win you watch your opponent’s center of gravity (CoG), for the eyes lie. For example, if the CoG moves toward their back foot you can expect a front kick, or if it begins a slight twist without moving forward or backward then a punch from the arm in the direction of the twist is coming. These are mandatory anticipatory movements which are precursors to a pending threat. If my opponent throws a punch or launches a kick without these movements it will be ineffectual. A punch without a twist is a tap. Of course the above is no secret. Skilled attackers lead with a feint to disguise their real intent, but that’s for another time. Cybersecurity is no different, you need to detect a threat, see it, classify it, then act on it. Detecting and seeing the threat is commonly referred to as Visibility. Classifying then acting on the threat is called Orchestration.

Imagine if you could watch the CoG of every server in your data center? In cyber terms that CoG might be every data flow in/out of the server. Placing boundaries and alerts on those flows is the primary role of orchestration. Placing these boundaries is now called micro-segmentation. Recently we suggested that the New Network Edge is the server itself. Imagine if you could watch every data flow from every server, set up zero trust policies to govern in advance which flows are permitted, then the system generates alerts to security operations when other flows are attempted. With solid governance comes the capability to quarantine applications or systems that have gone rogue.  All the while all of this is done within the server’s own NICs, without any host agents or utilizing any local x86 CPU cycles, that’s Solarflare ServerLock.

Below is a screenshot of ServerLock displaying seven groups of hosts, in the dark grey bubbles, with all the flows between those hosts in red. The Database servers group is highlighted, and all the network flows for this group are shown. Note this is a demonstration network. Click on the image below to see a larger version of it. 

The New Network Edge – The Server

Today cleverly crafted spear phishing emails and drive-by downloads make it almost trivial for a determined attacker to infect a corporate workstation or laptop. Wombat’s “State of the Phish 2018” report shows that 76% of InfoSec professionals experienced phishing attacks in 2017. Malware Remote Access Toolkits (RATs) like Remcos for Windows can easily be rebuilt with a new name and bound to legitimate applications, documents or presentations. Apple Mac users, myself included, are typically a smug group when it comes to Malware so for them, there’s MacSpy which is nearly as feature rich. A good RAT assumes total control over the workstation or server on which they are installed then it leverages a secure HTTPS connection back to their command and control server. Furthermore, they employ their own proprietary encryption techniques to secure their traffic prior to HTTPS being applied. This prevents commercial outbound web proxies designed to inspect HTTPS traffic from gaining any useful insights into the toolkits nefarious activities. With the existence of sophisticated RATs, we must reconsider our view of the enterprise network. Once a laptop or workstation on the corporate network is compromised in the above fashion all the classic network defenses, firewalls, IDS, and IPS are rendered useless. These toolkits force us to reconsider that the New Network Edge is the server itself, and that requires a new layer in our Defense in Depth model.

The data on our enterprise servers are the jewels that attackers are paid a hefty sum to acquire. Whether it’s a lone hacker for hire by a competitor, a hacktivist group or a rogue nation state, there are bad actors looking to obtain your companies secrets. Normally the ONLY defenses on the corporate network between workstations and servers are the network switches and software firewalls that exist on both ends. The network switches enforce sub-networks (subnets) and virtualized local area networks (VLANs) that impose a logical structure on the physical network. Access Control Lists (ACLs) then define how traffic is routed across these logical boundaries. These ACLs are driven by the needs of the business and meant to reflect how information should flow between different parts of the enterprise. By contrast, the software firewalls on both the workstations and servers also define what is permitted to enter and leave these systems. As defenses, both these methods fall woefully short, but today they’re the last line of defense. We need something far more rigorous that can be centrally managed to defend the New Network Edge, our servers.

As a representation of the businesses processes, switch ACLs are often fairly loose when permitting systems on one network access to those on another. For example, someone on the inside sales team sitting in their cubical on their workstation has access to the Customer Relationship Management (CRM) system which resides on a server that is physically somewhere else. The workstation and server are very likely on different subnets or VLAN within the same enterprise, but ACLs exist that enable the sales person’s workstation access to customer data provided by the CRM system. Furthermore, that CRM system is actually pulling the customer data from a third system, a database server. It is possible that the CRM server and the database server may be on the same physical server, or perhaps in the same server rack, but very possibly on the same logical network. The question is, is there a logical path from the inside sales person’s workstation to the database server, the answer should be no, but guess what? It doesn’t matter. Once the inside salesperson is successfully spear fished then it’s only a matter of time before the attacker has access to the database server through the CRM server.

The attacker will first enable the keylogger, then watch the sales person’s screen to see what they are doing, harvest all their user ids and passwords, perhaps turn on the microphone and listen to their conversations, and inspect all the outgoing network connections. Next, the attacker will use what they’ve harvested and learned to begin their assault, first on the CRM server. Their goal at this point is to establish a secondary beachhead with the greatest potential reach from which to launch their primary assault while keeping the inside sales person’s workstation as their fallback position. From the CRM server, they should be able to easily access many of the generic service machines: DNS, DHCP, NTP, print, file, and database systems. The point here is that where external attackers often have to actively probe a network to see how it responds, internal RAT based attacks can passively watch and enumerate all the ports and addresses typically used. In doing so they avoid any internal dark space honeypots, tripwires, or sweep detectors. So how do we protect the New Network Edge, the server itself?

A new layer needs to be added to our defense in depth model called micro-segmentation or application segmentation. This enforces a strict set of policies on the boundary layer between the server and the network. Cisco, Arista, and other switch providers, with a switch-based view of the world, would have you believe that doing it in the switch is the best idea. VMWare, with its hypervisor view of the world, would have you believe that their new NSX product is the solution. Others like Illumio and Tuffin would have you believe that a server-based agent is the silver bullet for micro-segmentation. Then there’s Solarflare, a NIC company, with its NIC based view of the world, and its new entrant in the market called ServerLock.

Cisco sells a product called Tetration designed to orchestrate all the switches within your enterprise and provide finely grained micro-segmentation of your network traffic. It requires additional Cisco servers be installed to receive traffic flow data from all the switches, processes the data, then provides network admins with both the visibility and orchestration of the security policies across all the switches. There are several downsides to this approach, it is complex, expensive, and can very possibly be limited by the ACL storage capabilities of the top of rack switches. As we scale to 100s of VMs per system or 1,000s of containers these ACLs will likely be stretched beyond their limits.

VMWare NSX includes both an advanced virtual switch and a firewall that both require host CPU cycles to operate. Again, as we scale to 100s of VMs per system the CPU demands placed on the system by both the virtual switch and the NSX firewall will become significant, and measurable. Also, it should be noted that being an entirely software-based solution NSX has a large attackable surface area that could eventually be compromised. Especially given the Meltdown and Spectre vulnerabilities recently reported by Intel. Finally, VMWare NSX is a commercial product with a premium price tag.

This brings us to the agent-based solutions like Illumio and Tuffin. We’ll focus on Illumio which comes with two components the Policy Compute Engine (PCE) and the Virtual Enforcement Node (VEN). The marketing literature states that the VEN is attached to a workload, but it’s an agent installed on every server under Illumio’s control and it reports network traffic flow data into the PCE while also controlling the local OS software firewall. The PCE then provides visualization and a platform for orchestrating security policies. The Achilles heel of the VEN is that it’s a software agent which means that it both consumes x86 CPU cycles and provides a large attackable surface area. Large in the sense that both its agent and the OS-based firewall on which it depends can both be easily circumvented. An attacker need only escalate their privileges to root/admin to hamstring the OS firewall or disable or blind the VEN. Like VMWare NSX, Illumio and Tuffin are premium products.

Finally, we have Solarflare’s NIC based solution called ServerLock. Unlike NSX and Illumio which rely on Intel CPU cycles to handle firewall filtering, Solarflare executes its packet filtering engine entirely within the chip on the NIC. This means that when an inbound network packet is denied access and dropped it takes zero host CPU cycles, compared to the 15K plus x86 cycles required by software firewalls like NSX or IPTables. ServerLock NICs also establish a TLS-based domain of trust with a central ServerLock Manager similar to Illumio’s PCE. The ServerLock Manager receives flow data from all the ServerLock NICs under management and provides Visibility, Alerting and Policy Management. Unlike Illumio though the flow data coming from the ServerLock NICs requires no host CPU cycles to gather and transmit, these tasks are done entirely within the NIC. Furthermore, once the Solarflare NIC is bound to a ServerLock Manager the local control plane for viewing and managing the NIC’s hardware filter table is torn down so even if an application were to obtain root privilege there is no physical path to view or manage the filter table. At this point the, it is only capable of being changed from the specific ServerLock Manager to which it is bound. All of the above comes standard with new Solarflare X2 based NICs that are priced at or below competitive Intel NIC price points. ServerLock itself is enabled as an annual service sold as a site license.

So when you think of micro-segmentation would you rather it be done in hardware or software?

P.S. Someone asked why there is a link to a specific RAT or why I’ve included a link to an article about them, simple it validates that these toolkits are in-fact real, and readily accessible. For some people, threats aren’t real until they can actually see them. Also, another person asked, what if we’re using Salesforce.com, that’s ok, as an attacker instead of hitting the CRM server I’ll try the file servers, intranet websites, print servers, or whatever that inside salesperson has access to. Eventually, if I’m determined and the bounty is high enough, I’ll have access to everything.

Onload Recovers Meltdown Lost Performance

The recently announced microprocessor architecture vulnerability known as Meltdown is focused on accessing memory that shouldn’t be available to the currently running program. Meltdown exploits a condition where the processor allows an unprivileged application the capability to continually harvest data unrestricted from anywhere in system memory. The flaw that enables Meltdown is based on microprocessor performance enhancements more than a decade old and are now common in Intel and some ARM processors. The solution to Meltdown is Kernel page-table isolation (KPTI), but it doesn’t come without a performance impact which ranges from 5-30%, every application behaves differently. Since Onload places the communications stack into that application’s userspace this dramatically reduces the number of kernel calls for network operations and as such avoids most of the performance impact brought on by KPTI. Redhat confirm this in a recent article on this topic. This means that applications leveraging Onload on KPTI patched kernels will see an even greater performance advantage.

By contrast Spectre tears down the isolation that exists between running applications. It allows a malicious application to trick error-free programs into leaking their secrets. It does this by scanning the process address space of those programs, and the kernel libraries on which they depend, looking for exploitable code. When this vulnerable code is executed it acts as a covert channel transmitting its secrets to the malicious application. This vulnerability affects a wider range of processors and requires both kernel and CPU microcode patches, and even then, the vulnerability hasn’t been 100% eliminated. More work remains to be done to shut down Spectre.

Post Spectre/Meltdown, Are We More Secure?

It’s human nature in time-critical environments to speculate and execute, remember Radar O’Reilly from M.A.S.H., it’s how we operate at our best. Sometimes you’re wrong and you throw out that work, but the majority of the time you’re right, and the payoff, in the case of an army field hospital could be life-saving. For computer processors, up until recently, that payoff was less critical and the savings could be measured in nanoseconds per successful speculation. Now while nanoseconds on their own don’t sound like much, consider that this savings comes with EVERY successful speculation. Also, it should be noted that there isn’t one block on the processor that is speculating and executing, but rather dozens of them in parallel. Current projections are that patching around the “speculate and execute” set of three flaws could reduce system performance from 5-30%. Friday Apple announced that they’ve patched all their OSes to mitigate just the Meltdown flaw and at worst the performance hit was only 2.5%, they are still working on Spectre. Redhat, which has worked closely with Google’s Project Zero, has not only addressed Meltdown but also Spectre by releasing patches for both sets of flaws.

Many articles and blog posts this week will cover the details of Spectre and Meltdown, and as such, this one won’t. Also, they will devote time on the lengths to which Intel, AMD, ARM, Linux, Apple, Microsoft, Google, Amazon, and others will or have gone to plug these holes, this post won’t. The real underlying question is are we more or less safe today as a result of these discoveries?

My view is that this is Shellshock only three years later and in silicon. Let’s flashback to 2014 when Stephane Chazelas discovered a critical security flaw in the Bash Shell after it had been in production for several decades. Within several days five more security flaws were found in the same code. Code that was widely considered stable and trusted. Clearly, it had never been revisited and retested using state-of-the-art hacking techniques. The same was true with speculative execution. It has been a microprocessor design component since the early days of pipelining, possibly as far back as the original IBM ROMP processor of 1981 (precursor to RISC and later the Power Architecture). It appears that chip security experts are now revisiting old trusted silicon techniques with new eyes looking for potential current exploits. This can only be a good thing as it will further secure the computing platforms on which we rely.

In the near term, patching for Spectre and Meltdown will improve security at the expense of performance, but it will only take another chip spin to likely recover that lost ground. At which point we’ll have safer and more trustworthy platforms.

While originally published on January 4th, 2018, this piece has since been updated on January 6, 2018.

2600, Attacking Enterprise Networks

Since 1984 the magazine “2600” has been the undisputed publication created by and for hackers and phone phreakers worldwide. The January 2017 cover to the right is misleading, the magazine isn’t named after the Atari system, but rather the 2600 Hz tone ATT used, that phreakers leveraged, to control long distance phone lines. Most article bylines are hacker handles, rather than proper names, as the articles themselves don’t always paint inside the numbers. In January 2017 a hacker with the handle “Daelphinux” published the first of a five-part series of articles titled “Successful Network Attacks – Phase One” and every quarter since then he’s published the next phase, with the final Phase Five hitting the streets today January 2, 2018. This collection of five articles is perhaps the most concise executive-level summary of how an attacker breaches an enterprise that I’ve read thus far. Hopefully, I won’t offend Daelphinux by attempting to summarizing the key points of his five articles in this single blog post. I strongly suggest that everyone reading this review the original text in these five issues of 2600.

Daelphinux classified a Successful Network Attack into five phases:

  1. Reconnaissance – “Gathering as much useful information about the targets as possible.”
  2. Scanning – “Gathering useful information about the target’s networks and any possible exploits.”
  3. Gaining Access – “Getting into the network to be able to accomplish the attack’s goal.”
  4. Maintaining Access – “Ensuring access to the network persists long enough to accomplish the attack’s goal.”
  5. Covering Tracks – “Obfuscating the attacker’s presence on the network such that they cannot be traced.”

Each phase builds on the first, with Daelphinux envisioning this as a pyramid with phase one on the bottom, and each successive phase building on the prior one. Attackers will need tools and skills at each level in order to conduct a successful attack. Defenders, the enterprise admins, will also need tools and skills for several phases to detect and defend against an attack.

Phase 1 – Reconnaissance. As defined by Daelphinux the attacker seeks to gather all the raw data they can on their target before actively engaging them. The key here is in gathering only the “useful” data, as an attacker will rapidly accumulate an enormous pile of information. This information should come from a wide variety of sources including, but not limited to: deep web searches, web crawling those results, calling all the targets publicly available phone numbers to learn everything they can about the target, draining “whois” databases for all known corporate DNS assets, launching phishing and email scams at targeted employes and generic email address, real-life social engineering by making new friends, and finally dumpster diving.

All these efforts will produce a heap of information, but most of it will be useless. Here is where intelligent sifting comes in. Important information is often in handwritten notes, or cast of printouts such as printer test configuration pages, IP table listings, usernames, equipment manufacturer shipping boxes, operating system manuals, internal organizational charts & structures, and corporate policies (especially password).

Recognizing these reconnaissance efforts is the initial step toward thwarting an attack. For example, reviewing security footage looking for dumpster divers might sound trivial, but differentiating between an attacker and someone looking something to sell for their next meal can be challenging. Other activities like social engineering become far more complex to detect via video surveillance as these activities appear as background noise. While this phase might be the toughest to detect, it is the easiest to defend against. If you can cut off the flow of information outside enterprise you can seriously hamper their reconnaissance efforts. To do this you can hide the whois records, destroy printed copies of purchase orders, destroy shipping boxes used to pack new servers or appliances, destroy discarded manuals, remove and clean printer hard drives, make sure all in-house shredders use cross-cut shredding and finally burn any really sensitive info that has already been shredded. Much of this can be addressed through procedures and training.

Phase 2 – Scanning, In April Daelphinux covered the details of this phase. He said that when an attacker moves on to phase two they are committed, and unlikely to walk away. This phase is the first real step where the attacker can’t evade detection as they have to deploy active tools to electronically gather as much insight as they can. Tools like: Nmap, Nessus, Metasploit, ZAP, Xenotix or Grabber. Here they are looking to generate IP Maps, enumerate subnets, determine network speeds, the resilience of networks, open ports on clients, appliances & servers used, along with applications and the versions in production.  All this scanning will provide the attacker with another huge heap of data to sift through with the eventual goal being to define the attack vectors that define the real “meat” of the attack.

This phase is the last chance for a defender to stop an attack before valuable assets are stolen. During this phase, administrators might notice network slowdowns, so on detecting these and investigating see if one IP address or a small range of addresses is touching a wide range of resources if so you are likely in the process of being scanned. Attackers will often launch the scanning phase remotely. So using network address translation (NAT) internally, next-generation firewalls, and current IPS & IDS appliances can in some cases detect this. It is always strongly recommended to be current in patching all your appliances and to follow proper admin process.

Phase 3 – Gaining Access, was published in 2600 in July of 2017. Without gaining access an attack isn’t an attack. Some of the key tools used are Metasploit and ZAP, but they will also leverage trojans, zombies, and backdoors to gain multiple toeholds into the enterprise’s network. Attackers often use remote shells instead of graphical tools, as shells often prove to be faster, more flexible and powerful.

Typically attackers operate from an endpoint that is not their intended target, for example leveraging another user’s machine to attack a server. By using another unsuspecting endpoint if it becomes compromised they can then move onto another workstation within the enterprise network. Attackers are typically interested in copying, moving, deleting and or altering files. In this article, we’re not interested in those seeking to launch a Denial of Service (DoS) attack, just those looking to exfiltrate value from the enterprise. Detecting attacks during this phase requires active monitoring, and the question is when, not if, you’ll detect an attacker. Doing active file audits, examining files that are changed outside of regularly expected intervals, watching for irregular traffic patterns, and reviewing access and error logs are all known methods of detecting an attack.

Phase 4 – Maintaining Access, was published in the October 2017 issue. This is the last phase where defending against an attack can successfully prevent data loss. Daelphinux stated that there are three things which should be kept in mind:

  1. The Attacker has already breached the network.
  2. The Attacker is actively attempting to achieve their goal.
  3. Less experienced attackers tend to get overly comfortable with their success at this point.

This is the stage at which attackers are at their most vulnerable. They are moving around in plain sight within your network and can be detected if you’re looking for them. Network monitoring is the key to detection in this phase, but also, unfortunately, detection at this phase is also based on hope. You as the defender are hoping you discover them, and they are hoping their connection won’t get them detected and severed.

Skilled attackers will setup precautions to prevent this. One of the attacker’s strategies will be to take over network monitoring tools in an effort to circumvent detection. Enterprises need a heavy level of paranoia at this point to ensure that they are checking everything. Looking for the use of uncommon ports or protocols is another method for detecting attackers. Typically Intrusion Prevention (IPS) and Intrusion Detection Systems (IDS) and Security Information and Event Management Systems (SIEMS) are useful tools in ferreting out hackers. SIEMS themselves often are now targets for attackers. Multiple instances of these devices or appliances should be leveraged to thwart a takedown. Consider having these monitors watch each other, so in the event, one goes down you can use that as a potential indicator of an attack. If both or all monitors go down simultaneously then it’s very possible you’re under attack. Killing connections that meet certain criteria are vital to cutting off an attackers access, for example terminating connections that last longer than 25 minutes. Think that scene in the original “Transformers” when the general shouts “Cut the hard lines!”

Phase 5 – Covering Tracks, today January 2, 2018, we saw the latest issue of 2600 with Daelphinux final installment in the series on “Successful Network Attacks – Phase Five – Covering Tracks.” Once an attacker has reached phase five they’ve taken what they need, and now the coverup begins. Attempting to defend against this phase “is a form of damage control,” at best you’re preserving forensic evidence to hopefully reconstruct what they took after the fact. An attacker has to leave as cleanly as they entered otherwise they could dilute some of the value of what was taken. As Daelphinux points out, what good is a list of users and passwords if the passwords have all been changed. The same holds true of Malware and backdoors, you can’t sell a backdoor into an enterprise if it has been removed.

The best defense against this is redundant, and perhaps even hidden copies of logs. Attackers will often sanitize the logs they find, but they rarely go looking for additional copies of logs, especially if some effort has been made to hide and even secure them. It’s possible that if you have automated your logging such that multiple copies are generated, and accesses are tracked the attacker will notice this in phase three and just avoid it. Attackers will normally obfuscate both their IP and MAC addresses to further frustrate tracking them, often using addresses already on the network. Again, as mentioned in phase three setting connection limits, timeouts, and alerts when these are reached is often a good way to thwart or even detect an attacker. It should also be noted that attackers will often escalate their privilege on a system so they can disable logging. As Daelphinux noted disabling logging will often generate its own log event, but then after that, you won’t know what was done. Some attackers may even just erase or corrupt log files on the way out the door, a sort of “salt the earth” strategy to make determining what was taken that much more difficult. Regardless, a company will need to make some determinations on what was stolen or affected and alert their customers. A recent Ponemon report states that just cleaning up and dealing with a data breach in the US often costs companies $244/record.

Hats off to Daelphinux for authoring “Successful Network Attacks,” and “2600, The Hacker Quarterly” for publishing it.

I hope everyone has a Happy and Secure New Year.

Security: DARPA, HFT & Financial Markets

Today nearly half of all Americans are invested in the financial markets. This past October the Dow Jones posted the “Pentagon Turns to High-Speed Traders to Fortify Markets Against Cyberattack.” The reporter had talked with a number of High-Frequency Trading (HFT) shops which had consulted directly with the Defense Advanced Research Projects Administration (DARPA). The objectives of these discussions were to determine how we could fortify the US financial markets against Cyber attacks.

The reporter learned that the following possible scenarios were discussed as part of the “Financial Markets Vulnerability Project:”

  1. Inject false information into stock data feeds
  2. Flood the stock market with fake orders and trigger a market crash
  3. Cripple a widely used payroll system
  4. Credit Card Processors
  5. Report fake news into systems used to algorithmically drive trading

While protecting the US financial markets is something we expect of our government, the markets themselves are actually already insulated from outside attackers. The first two threats in the above list are essentially the same, placing fake orders into the exchange with no intent to honor them. To connect to an exchange’s servers a trader must be a member in good standing on that exchange and pay significant connection fees for their server to participate in that exchange. Traders place a very high value on their access to each exchange, and while HFT shops may only hold a security for a few millionths of a second, they understand the long-term value of losing access to an exchange. Most HFT shops have leased many 10GbE connections on multiple exchange servers, across multiple exchanges, and big bank’s dark pool, and very often Solarflare NIC cards are on both sides of these connections. So while it is technically possible for an HFT shop to inject enormous volumes of orders into one or more exchanges, a type of Denial of Service attack, using one or more physical ports on one or more exchange servers it could quickly result in financial suicide for that the trading firm. The exchanges and the Securities and Exchange Commission (SEC) don’t take kindly to trading partners seeking to game the system. Quickly the exchanges, and soon after the SEC, would step in and shut down inappropriate activity. *It should be noted that the above image was taken on December 6, 2017, in New York City’s Times Square.

To further improve security for its trading customers later this month Solarflare will begin rolling out a beta of ServerLock™ which is a firmware update for these very same NICs powering the exchanges and HFT shops worldwide. With ServerLock™ the HFT shops and the exchanges themselves could rapidly pump the breaks on any given logical connection directly within the NIC hardware.  This is the point at which DARPA and others should be interested. If the logic within the exchange were to detect and validate a threat they could then within a few millionths of a second install a filter into the NIC hardware to drop all subsequent packets from that threat. At that point, the threat would be eliminated, and it would no longer consume exchange CPU cycles. For HFT shops if they were to detect an algorithm had gone rogue they could employ ServerLock™ to physical cut a trading platform from the exchange without having to actually touch the platforms precious code. Much like throwing a cover over Schrodinger’s box, by applying the filter in the NIC hardware the trading platform itself remains intact for later investigation.

Number three on the list above is crippling a widely used payroll processor like ADP who processes payroll checks for one out of six Americans? First ADP uses at least two different networks. One permits inbound payroll data from their client companies, over the public internet via SSL secured connections, and a second which is a private Automated Clearing House (ACH) network. The ACH network is a member network connecting banks to clearinghouses like the Federal Reserve. Much like the exchanges above, being a paid member of an ACH network then attacking that same network would not be a wise move for a business. As for the public Internet-facing connections that ADP maintains, they likely are practicing the latest defense in depth technologies coupled with least privilege in an effort to avoid the issues faced earlier this year by Equifax.

Next, we have the Credit Card Processors also know as Payment Card Industry (PCI) players from Amex to Square who are fighting a never-ending battle to secure their systems against outsider threats. Much like the ACH network the PCI industry has its own collection of private networks for processing credit card transactions, ex. the Mastercard network, or Visa network, etc… These networks, like the ACH networks, are member networks, and attacking them would also be counterproductive. The world economy would likely not be in Jeopardy if at any point say the Amex or Discover networks were to stop processing credit cards for a few hours. We have seen the Internet websites of these providers, ex. Mastercard, have been targets of some of the most substantial Distributed DoS (DDoS) attacks the world has ever seen, and they’ve all faired it pretty well. Most have learned from these assaults how to further harden their networks.

Who would have thought two years ago that “Fake News” could possibly have turned the tide of a US Presidential election, or be used as a tool to dramatically shift a financial market? While at DEFCON 2015 I watched as Charlie Miller and Chris Valasek presented their now infamous hack of a Jeep Grand Cherokee. At the start of their talk, Charlie joked that had they thought the wired article would have moved Chrysler stock more than a point or two he would have partnered up with a VC to fund shorting their stock. He said that had he done that he’d now be sitting on the beach of his private island now sipping his favorite frozen drink through a straw, rather than lecturing us. Charlie explained that he expected their announcement would be similar to Google or Microsoft announcing a bug, but he was very wrong. It led to a recall of 1.4 million vehicles and the stock dropped double-digit percentage points following the story and the recall. While this was real news, it was a controlled news release from someone outside the company. They could have easily made hundreds of millions of US dollars shorting the stock. Now what most people aren’t aware of is that there are electronic news systems that some HFT algorithmic platforms are subscribed to. Some of these systems even “read” tweets from key people (ex. our president) to determine if their comments might move a particular security or market in one direction or another. Knowing this, these systems can then be gamed by issuing false stories expecting that the HFT algorithms will then “read” these stories and stock prices will move appropriately. When retractions are issued later it might also be expected that they will place orders that would also benefit from these retractions. So how do we suppress the impact of “fake news” on our financial markets?

These news services know that HFT systems trade on their output. Given that, they should be investing heavily in machine learning based systems to rapidly fact-check and score the potential truthfulness of a given story. For those stories that score beyond belief, they should then be kicked to humans for validation or potentially be delayed until they are backed up by additional sources or even held until after the US markets close to further limit their impact.

Kernel Bypass = Security Bypass

As we move our performance focused applications to kernel bypass techniques like DPDK and Solarflare’s Onload this does not come without a price, and one component of that price is often security. When one bypasses the Linux kernel, they are also bypassing its security mechanisms (ex. XDP and NFTables, formerly IPTables). These security mechanisms have evolved over the past decade to ensure that your server doesn’t get compromised. Are they perfect no, software rarely is, but they are an excellent starting point to secure your Linux server. So as we move to kernel bypass platforms what options are available to us? We need to define lower level network security checkpoints that can be used as gatekeepers to keep the good stuff in and the bad stuff out. With one exception these are often hardware products that are managed using several different networking segmentation metaphors: micro, macro, and application which is also known as workload.

Micro-segmentation is the marketing term that has been co-opted by VMWare to represent its NSX security offering. When you’re a hypervisor company all the worlds a virtual machine (VM) so moving security into the hypervisor is a natural fit. VMWare then plays a clever trick and abstracts the physical network from the VM by installing a virtual network to which it then connects the VM. The hypervisor then works as the switch between the physical and virtual networks. To support coordinating workloads and security across multiple hypervisors running on different physical servers VMWare goes one step further and encapsulates traffic. This enables it to take traffic running on one virtual network and bridge it over the physical network to a virtual network on another host. So if your kernel bypass application can run from within a VM without having to rely on hypervisor bypass, then this model might work for you. Illumio has also attached itself to micro-segmentation, but rebranding it “smart micro-segmentation.” Our understanding is that they essentially run an agent that then programs NFTables in real time, so for kernel bypass applications this would offer no security.

Macro-segmentation, as you might guess, means creating segmented networks that span multiple external physical network devices. This is the term that Arista Networks has chosen (originally they used micro-segmentation, perhaps until VMWare stepped in). Macro-segmentation is the foundation for Arista’s CloudVision line of products. While this too does an awesome job of securing your network it doesn’t come without cost, which is complexity. CloudVision connects into VMWare NSX, OpenStack and other OVS DB based controllers to enable you to seamlessly configure various vendors hardware through a single interface. Furthermore, it comes with configuration modules called configlets for a wide variety of hardware that enables you to quickly and easily duplicate data center functions across one or more data centers. It also includes a configlet builder tool to quickly empower an administrator to craft a configlet for a device for which one does not exist.

The last solution is application or workload segmentation. In techie terms, this is five-tuple filtering and enforcement of network traffic. Which to the layperson means opening the network packet up, inspecting the protocol it uses, along with the source and destination addresses and ports. Then taking these five values and comparing them to some collection of filter tables to determine the appropriate action to take on the packet. Today this can be done by Solarflare ServerLock NICs or applications like XDP or NFTables. ServerLock NICs do this comparison in 50 to 250 nanoseconds within the firmware of the NIC itself, entirely transparent to the server the NIC is installed in. In doing it this way the process of filtering consumes no host CPU cycles, is agnostic to the OS or applications running, and it scales with every NIC card added to the server. Packets are filtered at wire-rate, 10Gbps/port, and there can be one filter table for every locally hosted IP address with a total capacity exceeding over 5,000 filters/NIC. As mentioned, all of this filtering is done in the NIC hardware without any awareness of it by the DPDK or Onload applications running above it.

So if you’re using DPDK or Onload, and the security of your application, or the data it shares, is of concern to you, then perhaps you should consider engaging with one of the vendors mentioned above.

If you’d like to learn more about ServerLock, please drop me an email.

Security Entirely Chimerical, SEC

On September 20th SEC Chairman Jay Claton released a “Statement on Cybersecurity.” It is an extremely dry read, but if one suffers through it they’ll find several interesting points.

“I recognize that even the most diligent cybersecurity efforts will not address all cyber risks that enterprises face. That stark reality makes adequate disclosure no less important.”

How does the SEC define “adequate disclosure?” The federal government has requirements that in some extreme breach cases require a report within one hour to DHS’s CERT. When faced with this class of breach recently it was found that the SEC waited 14 days, is this adequate disclosure? Much further down in the SEC Statement they disclosed the following.

“In August 2017, the Commission learned that an incident previously detected in 2016 may have provided the basis for illicit gain through trading. Specifically, a software vulnerability in the test filing component of our EDGAR system, which was patched promptly after discovery, was exploited and resulted in access to nonpublic information.”

So in the best case, the SEC waited only eight months to inform the public of this breach, but it could have been as much as 20 months. Unlike the publicly traded companies, the SEC regulates it isn’t legally required to tell investors or the public if it is ever breached. It is ONLY required to inform a law enforcement agency. EDGAR was also breached in 2014, but that saw little attention.

Now it’s one thing to breach an entity and remove data, but how about intentionally leaving false data behind for the purpose of capitalizing on that deposit? In at least two cases over the past few years, false business acquisition reports for Avon and the Rocky Mountain Chocolate Factory have been inserted into EDGAR. In the Avon case, the stock ran up 10 points. Does the SEC own up to these, well kinda of…

“As another example, our Division of Enforcement has investigated and filed cases against individuals who we allege placed fake SEC filings on our EDGAR system in an effort to profit from the resulting market movements.”

Ok, so EDGAR is a 30-year-old piece of swiss cheese riddled with potential attack surfaces some by design, others by just not keeping current on penetration testing of their systems. What about their physical assets?

“For example, a 2014 internal review by the SEC’s Office of Inspector General (“OIG”), an independent office within the agency, found that certain SEC laptops that may have contained nonpublic information could not be located.”

All the above quotes were from the Wednesday SEC Statement, but in a 2016 GAO report on the SEC, it stated that the SEC:

“…wasn’t always using encryption, supported software, well-tuned firewalls, and other key security tools while going about its business.”

Banking, in fact, our financial market structure as a whole is based on a singular concept, TRUST. The SEC was created in the wake of the Great Depression in 1934 as a way to restore trust in the markets. Technology savvy individuals will always attempt to exploit this trust for their own gain, it’s a part of how the game is played. In our financial system, the SEC plays the role of the gambling commission to ensure that the players, dealers, pit bosses, and the house are all working from the same set of published public rules. To his credit Chairman Clayton is working within the system in an attempt to shine daylight on an agency in trouble and out of touch with the technology driving the markets its charged with regulating. Today it is now possible to trade a stock based on a tick (a signal that something moved) within 150 billionths of a second, but it takes the SEC 1.2 million seconds (14 days) to report a serious breach of their security to law enforcement. Clearly, work remains to be done.