Focus on the New Network Edge, the Server

For decades we’ve protected the enterprise at the network edges where the Internet meets our DMZ, and then again where our DMZ touches our Intranet. These two distinct boundary layers and the DMZ in-between makeup what we perceived as the network edge. It should be pointed out though that these boundaries were architected long before phishing and click-bate existed as part of our lexicon. Today anyone in the company can open an email, click on an attachment or a web page, and open Pandora’s box. A single errant click can covertly launch a platform that turns the computer into a beachhead for the attacker. This beachhead then circumvents all your usual well-designed edge focused defenses as it establishes an encrypted tunnel enabling the attacker access to your network whenever they like.

Once an attacker has established their employee hosted beachhead, they then begin the search for a secondary, server-based, vantage point from which to operate. A server affords them a more powerful hardware system and often one with a higher level of access across the entire enterprise. Finally, if the exploit is discovered in that server, the attacker can quickly revert to their fall back position on their initial beachhead system and wait out the discovery.

This is why enterprises must act as if they’ve already been breached. Accept the fact that there are latent attackers already inside your network seeking out your corporate jewels. So how do you prevent access to your companies most valuable data? Attackers are familiar with the defense in depth model so once they’re on your corporate networks, often all that stands between them and the data they desire is knowing where it is hidden, and obtaining the minimum required credentials to access it. So how do they find the good stuff?

They start by randomly mapping your enterprise network in hopes that you don’t have internal honey-pots or other mechanisms that might alert you to their activity. Once the network is mapped they’ll then use your DNS to assign names to the systems they’ve discovered in hopes that this might give them a clue where the good stuff resides. Next, they’ll do a selective port scan against the systems that look like possible targets to determine what applications are running on them to fill in their attack plan further. At this point, the attacker has a detailed network map of your enterprise, complete with system names, and the names of the applications running on those systems. The next step will be to determine the versions of the applications running on what appear to be the most critical systems, so they’ll know which exploits to leverage. It should be noted that even if your servers have a local OS based firewall, you’re still vulnerable. The attackers at this point know everything they need to, so if you haven’t detected the attack by this stage, then you’re in trouble because the next step is the exfiltration of data.

If we view each server within your enterprise as the new network edge, then how can we defend these systems? Solarflare will soon announce ServerLock, a system that leverages the Network Interface Card (NIC) in your server to provide a new defense in depth layer in hardware. A layer that not only shields it from attack, but it can also camouflage the server and report attempts made to access it. Two capabilities not found in OS based software firewalls. Furthermore, since all security is handled entirely within the NIC, there is no attackable surface area. So how does ServerLock provide both camouflage and reporting?

When a NIC has ServerLock enforcement enabled only network flows for which a defined policy exists are permitted to enter or exit that server. If a new connection request is made to that server which doesn’t align with a security policy, say from an invalid address or to an invalid port, then that network packet will be dropped, and optionally an alert can be generated. The attacker will not receive ANY response packet and assume that nothing is there. Suppose you are enforcing a ServerLock policy on your database servers which ONLY accepts connections from a pool of application servers, and perhaps two administrative workstations, on specific numeric ports. If a file server were compromised and used as an attack position once it reaches out to one of those database servers via a ping sweep or an explicit port scan it would get NOTHING back, the database server would appear as network dark space to the file server. On the ServerLock Manager console alerts would be generated, and the administrator would know in an instant that the file server was compromised. Virtually every port on every NIC that is under ServerLock enforcement is turned into a zero-interaction honeypot.

So suppose the attacker has established themselves on that file server, and the server then gets upgraded to ServerLock and put under enforcement. The moment that attacker steps beyond the security policies executing in that NIC on that server the jig is up. Assuming they’re on the server, once they attempt any outbound network access that falls outside the security policies those packets will be dropped in the NIC, and an alert will be raised at the ServerLock Management console. No data exfiltration today.

Also, it should be noted that ServerLock is not only firmware in the NIC to enforce security policies, but it is also an entire tamper-resistant platform within the NIC. Three elements make up this tamper-resistant platform, first only properly signed firmware can be executed, older firmware versions cannot be loaded, and any attempt to tamper with the hardware automatically destroys all the digital keys stored within the NIC. Valid NIC firmware must be signed with a 384-bit key utilizing elliptic curve cryptography. The Solarflare NIC contains the necessary keys to validate this signature, and as mentioned earlier tampering with the NIC hardware will result in fuses blowing that will corrupt the stored keys forever rendering the both unusable and unreadable.

Today enterprises should act as though they’ve already been compromised, and beef up their internal defenses to protect the new network edge, the server itself. In testing ServerLock, we put a web server protected by ServerLock directly on the Internet, outside the corporate firewall.

Compromised Server Supply Chains, Really?

2018 Was shaping up nicely to become “The Year of the CPU Vulnerability” what with Meltdown, Spectre, TLBleed, and Foreshadow we had something going then along came Bloomberg and “The Big Hack” story. Flawed CPU designs just weren’t enough; now we have to covertly install “system on a chip (SoC)” spy circuits directly into the server’s baseband management controller (BMC) at the factory. As if this weren’t enough today Bloomberg drops its second story in the series “New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom” which exposes compromised RJ45 connectors in servers.

We learned recently that Edward Snowden’s cache of secret documents from five years ago included the idea of adding an extra controller chip to motherboards for remote command and control. Is it astonishing that several years later a nation-state might craft just such a chip? Today we have consumer products like the Adafruit Trinket Mini-Microcontroller, pictured below, at $7USD the whole board is 27mm x 15mm x 4mm. The Trinket is an 8Mhz 8bit Atmel ATtiny85 minicomputer that can be clocked up to 20Mhz, with 8K flash, 512 bytes of SRAM and 512 bytes of EEPROM ($0.54USD for just the microcontroller chip) in a single 4mm x 5mm x 1.5mm package. In the pervasive Maker culture that we live in today, these types of exploits aren’t hard to imagine. I’m sure we’ll see some crop up this fall using off the shelf parts like the one mentioned above.

In the latest Bloomberg story, one source Yussi Appleboum, revealed that the SMC motherboards he found had utilized a compromised RJ45 Ethernet connector. This rogue connector was encased in metal providing both camouflage for the hidden chip and as a heat sink to dissipate the power it consumes. In this case all one would need to do would be to craft a simple microcontroller with an eight pin package, one for each conductor in the RJ45 connector. This controller would then draw it’s power directly from the network while also sniffing packets entering and leaving the BMC. Inconceivable, hardly, the metal covering such a connector is somewhere around 12mm square, similar to the RJ45 on the Raspberry Pi shown to the right, that’s four times more area than the ATtiny85 referenced above. Other micro-controllers, like the one powering the Raspberry Pi Zero, could easily fit into this footprint and deliver several orders more processing power. The point is that if someone suggested this five years ago, at the time of the Snowden breach, I’d have said it was possible but unlikely as it would have required leading-edge technology in the form of custom crafted chips costing perhaps ten million or more US dollars. Today, I could recommend a whole suite of off the shelf parts, and something like this could very likely be assembled in a matter of weeks on a shoestring budget.

Moving forward OEMs need to consider how they might re-design, build, and validate to customers that they’ve delivered a tamper-proof server. Until then for OCP compatible systems you should consider Solarflare’s X2552 OCP-2 NIC which can re-route the BMC through their network ports and which includes Solarflare’s ServerLock™ technology that can then filter ALL network traffic entering and leaving the server. That is provided of course that you’ve disconnected the servers own Gigabit Ethernet ports. If you’d like a ServerLock™ sample white-list filter file that shows how to restrict a server to internal traffic only (10.x.y.z or 192.168.x.y) then please contact me to learn more.

UPDATE: This weekend I discovered the item shown to the right which is offered as both a complete product called the “LAN Tap Pro” for $40 in a discrete square black case or as this throwing star kit for $15 with all the parts, some assembly and soldering required. This product requires NO external power source, and as such, it can easily be hidden. The chip which makes the product possible, but which is not shown, should answer the question of whether or not the above hacking scenario is a reality. While this product is limited to 10/100Mb, and can not do GbE, it has a trick up its sleeve to down speed a connection so that the network can be easily tapped. When it comes to server monitoring/management ports these often do not require high-speed connections so it’s highly unlikely that down speeding the connection would likely even be noticed. The point of all this rambling is that it’s very likely that the second Bloomberg article is true if the parts necessary to accomplish the hacking task are easily available through a normal retail outlet like the Hacker Warehouse.

IPv6, an Appropriate Glacier Turns 21

This month IPv6 hit its 21st anniversary, but outside of Google, cloud providers, cell phone companies and ISPs, who really cares? One would think by now it would be widely adopted or dead, as technologies rarely last two decades if they’re unsuccessful. Estimates are that between 5% and 25% of Internet traffic is IPv6, and adoption rates vary greatly between countries. So what about the other 75% to 95% of Internet traffic? It’s using IPv4, you know those addresses of the form 192.168.1.1, technology which is 35 years old! These addresses resolve into four bytes that provide 4.3 billion unique numbers that can be assigned to publicly routed devices.

Back in 1983 having a worldwide network with a potential capacity of 4.3 billion connected devices was inconceivable. The IPv4 system was designed to link supercomputers together across the globe, military installations, large company mainframes and perhaps minicomputers used by smaller companies. For those who didn’t live through 1983, Radio Shack ruled the personal computer market with the TRS80 Model I, III, and the Color Computer. If you were online you paid CompuServe and dialled in via a 300 baud (bit/sec modem). Personal computers from Apple and IBM were just hitting the market, and cell phones were nothing more than glorified two-way radios addressable via a phone number, while selfies were called Polaroids, and you waited a minute or so to see a low quality “instant” picture. Who could have imagined then that soon a significant percentage of the people on the planet would have networked computers in their pockets, strapped to our wrists, controlling our automobiles, refrigerators and septic systems?

Now for those who might not be keeping up, we “officially” ran out of publicly available IPv4 address blocks back in January 2011, yeah right, who knew? Now to be fair the January 31, 2011 date is when we exhausted the top-level blocks which are doled out to the five regional Internet registries (RIRs). Going one level farther down, the last of the RIRs to consume its final block was North America in September 2015. So how have we survived the end of available IPv4 addresses? Simple, since 1994 we’ve been playing a series of tricks to expand the available address space beyond 4.3 billion. Suppose like me you have a single IPv4 Internet address on your home router, inside your home network the router and all the devices use an address on that network of 192.168.1.X. Your router then uses a trick called Network Address Translation (NAT) to map all the 192.168.1.X devices on your network to that single IPv4 address. Brantley Coile founded a company in 1994 called Network Translation that patented NAT, then rolled it out in a device that Cisco later bought and rebranded the PIX Firewall (Private Internet eXchange). Today the vast majority of Internet-connected devices are using a NAT’d address. These days it’s highly unlikely that a company will assign a publicly routed Internet address to a laptop or workstation. I joined IBM Research back in late 1983, and by 1987 was standing up servers with their own unique publicly routed 9.X.X.X addresses. This was before the days of firewalls and security appliances. At the time IBM was one of roughly 100 entities worldwide who had their own class A Internet address space (an IPv4 address starting with 126 or less, for example, General Electric has 3, IBM 9, HP 15, and Ford 19). If you control a class A address you have 16 million publicly routable Internet addresses at your disposal.

Administrators for decades have trained ourselves to grasp a four-byte number like 10.5.17.23, for example, so we could then key it into another device and manage or networks. We invested time in knowing IPv4 and building networks to meet our needs. IPv6 address look like 2001:0db8:85a3:0000:0000:8a2e:0370:7334, this is not human-friendly. That is why inside large companies, where the administrators are familiar with IPv4, there’s resistance to moving to IPv6 addresses. IPv6 is designed for automated machine management. Personally, this spring I was assigned the first IPv6 address I took note of when we converted over to Google Fiber at home. So what was the first thing I did once the fibre was active? I requested an IPv4 address. It turns out there are several servers in my house which I need to reach remotely, and I wasn’t about to begin pasting in an IPv6 address whenever I needed to connect with them. There may be a better way, but I fall back on what I know and trust, it’s human nature. 

Today most of our new Internet edge devices, for example, routers and smartphones, are intelligent enough that they self-configure and the whole issue of IPv4 to IPv6 conversion will slowly fade into the background. Within the home or the Enterprise though, where devices need a human touch, IPv4 will live long and prosper. 

Container Performance Doesn’t Need to Suck

Recently the OpenShift team at Red Hat, working with Solarflare Engineering, rolled out new code that was benchmarked by a third party, STAC Research, which demonstrated networking performance from within a container that was equivalent to that of a bare metal server. We’re talking 1.2 microseconds for 99% of network traffic in a 1/2RT (half round trip), that’s a TCP receive to an application coupled with a TCP send from that application.

Network performance like this was considered leading edge in High-Performance Computing (HPC) a little more than a decade ago when Myricom rolled out Myrinet10G which debuted at 2.4 microseconds back in 2006. Both networks are 10Gbps so it’s sort of an apples to apples comparison. Today, this level of performance is available for containerized applications using generic network socket calls. It should be noted that the above numbers were for zero byte packets, a traditional HPC measurement. More realistic performance using 256-byte packets yielded a 1/2RT time for the 99th percentile of traffic which was still under 1.5 microseconds, that’s amazing! It should be noted that everything was done to both the bare metal server and the Pod configuration to optimize performance. A graph of the complete results of that testing is shown below.

Anytime we create abstractions to simplify application execution or management we introduce additional layers of code that can result in potentially unwanted delays, known as application latency. By running an application inside a container, then wrapping that container into a Pod we are increasing the distance between what we intend to do, and what is actually being executed. Docker containers are fast becoming all the rage and methods for orchestrating them using tools like Kubernetes are extremely popular. If you dive into this OpenShift blog post there are ways to cut through these layers of code for performance while still retaining the primary management benefits.

What the FEC?
Auto-Detect Finally Here for 25G!

As technology marches forward new challenges arise that were not previously an issue. Consider as mankind moved from walking to horseback we cleared trails where there was once brush covered paths. As we transitioned from horseback to carriages those paths needed to become dirt roads, and the carriages added suspension systems. With the move from carriages to automobiles, we further smoothed the surface traveled by adding gravel. As the automobiles moved faster, we added an adhesive to the gravel creating paved roads. With the introduction of highways, we required engineered roads with multi-layered surfaces. Each generation reduced the variability in the road surface by utilizing new techniques that enabled greater speed and performance. The same holds true for computer networks.

Over the past three decades as we transitioned from 10Mbps to 25Gbps Ethernet we’ve required many innovations to support these greater speeds. The latest of these being Forward Error Correction (FEC). The intent of FEC is to reduce the bit error rate (BER) as the cable length increases. In 2017 we saw the ratification of the IEEE 25GbE specification which provides two unique methods of FEC. There is BASE-R FEC (also known as Firecode) and RS-FEC (known also as Reed Solomon). Both of these FEC algorithms introduce additional network latency as the signal is decoded, BASE-R is about 80 nanoseconds while RS-FEC is about 250 nanoseconds. The complexities don’t end here though, it turns out there are three different Direct Attach (DA) cable types with varying levels of quality, from good, to best we have:

  • CA-25G-L: up to 5m, requires RS-FEC
  • CA-25G-S: up to 3m, lower loss, requires either RS-FEC or BASE-R FEC
  • CA-25G-N: up to 3m, even lower loss, can work with RS-FEC, BASE-R FEC, or no FEC

But wait there’s more, if you order now we’ll throw in auto-negotiation (AN) and link training (LT) as both are required by the 25GbE IEEE standard (10GbE didn’t need these tricks). So what does AN actually negotiate? Two things, link speed and which type, if any, FEC will be utilized. It should be noted that existing 25GbE NICs that have been on the market likely only support one type of FEC. As for LT, it helps to improve the quality of the 25GbE link itself. It turns out though that the current generation of 25GbE switches came out prior to AN being worked out so support is at best poor to mixed. Often manual switch and adapter configuration are required. Oh, and did I mention that optical modules don’t support AN/LT? Well, they don’t, but some will support short links with no FEC.

So where does this leave people who want to deploy 25GbE? You need to be careful that both your network switch and server NICs will work well together. We strongly advise that you do a proof of concept prior to a full deployment. Not all 25G server NICs do both AN/LT because their chips (ASICs) were designed and fabricated prior to the completion of the IEEE specification for 25GbE last year. Solarflare’s 25GbE X2522 server NICs which debut next month include support for all the above, in fact, when initially powered up they will begin by:

  • First looking at cable, is it SFP or SFP28?
  • If it’s SFP28 it will attempt AN/LT, then 25G no AN/LT, then 10G
  • If it’s a 25G link, then it will try and detect which FEC is being used by the switch

Additionally, the server administrator can manually override the defaults and select AN/LT and the FEC type and setting (auto, on, off).

I grew up in New York, and remember listening to Sy Sims on TV say “an educated consumer is our best customer…”

P.S. I’d like to give a special thanks to Martin Porter, Solarflare’s VP of Engineering, for pulling all this together into a few slides.

Visibility + Control = Orchestration

In Taekwondo to win you watch your opponent’s center of gravity (CoG), for the eyes lie. For example, if the CoG moves toward their back foot you can expect a front kick, or if it begins a slight twist without moving forward or backward then a punch from the arm in the direction of the twist is coming. These are mandatory anticipatory movements which are precursors to a pending threat. If my opponent throws a punch or launches a kick without these movements it will be ineffectual. A punch without a twist is a tap. Of course the above is no secret. Skilled attackers lead with a feint to disguise their real intent, but that’s for another time. Cybersecurity is no different, you need to detect a threat, see it, classify it, then act on it. Detecting and seeing the threat is commonly referred to as Visibility. Classifying then acting on the threat is called Orchestration.

Imagine if you could watch the CoG of every server in your data center? In cyber terms that CoG might be every data flow in/out of the server. Placing boundaries and alerts on those flows is the primary role of orchestration. Placing these boundaries is now called micro-segmentation. Recently we suggested that the New Network Edge is the server itself. Imagine if you could watch every data flow from every server, set up zero trust policies to govern in advance which flows are permitted, then the system generates alerts to security operations when other flows are attempted. With solid governance comes the capability to quarantine applications or systems that have gone rogue.  All the while all of this is done within the server’s own NICs, without any host agents or utilizing any local x86 CPU cycles, that’s Solarflare ServerLock.

Below is a screenshot of ServerLock displaying seven groups of hosts, in the dark grey bubbles, with all the flows between those hosts in red. The Database servers group is highlighted, and all the network flows for this group are shown. Note this is a demonstration network. Click on the image below to see a larger version of it. 

The New Network Edge – The Server

Today cleverly crafted spear phishing emails and drive-by downloads make it almost trivial for a determined attacker to infect a corporate workstation or laptop. Wombat’s “State of the Phish 2018” report shows that 76% of InfoSec professionals experienced phishing attacks in 2017. Malware Remote Access Toolkits (RATs) like Remcos for Windows can easily be rebuilt with a new name and bound to legitimate applications, documents or presentations. Apple Mac users, myself included, are typically a smug group when it comes to Malware so for them, there’s MacSpy which is nearly as feature rich. A good RAT assumes total control over the workstation or server on which they are installed then it leverages a secure HTTPS connection back to their command and control server. Furthermore, they employ their own proprietary encryption techniques to secure their traffic prior to HTTPS being applied. This prevents commercial outbound web proxies designed to inspect HTTPS traffic from gaining any useful insights into the toolkits nefarious activities. With the existence of sophisticated RATs, we must reconsider our view of the enterprise network. Once a laptop or workstation on the corporate network is compromised in the above fashion all the classic network defenses, firewalls, IDS, and IPS are rendered useless. These toolkits force us to reconsider that the New Network Edge is the server itself, and that requires a new layer in our Defense in Depth model.

The data on our enterprise servers are the jewels that attackers are paid a hefty sum to acquire. Whether it’s a lone hacker for hire by a competitor, a hacktivist group or a rogue nation state, there are bad actors looking to obtain your companies secrets. Normally the ONLY defenses on the corporate network between workstations and servers are the network switches and software firewalls that exist on both ends. The network switches enforce sub-networks (subnets) and virtualized local area networks (VLANs) that impose a logical structure on the physical network. Access Control Lists (ACLs) then define how traffic is routed across these logical boundaries. These ACLs are driven by the needs of the business and meant to reflect how information should flow between different parts of the enterprise. By contrast, the software firewalls on both the workstations and servers also define what is permitted to enter and leave these systems. As defenses, both these methods fall woefully short, but today they’re the last line of defense. We need something far more rigorous that can be centrally managed to defend the New Network Edge, our servers.

As a representation of the businesses processes, switch ACLs are often fairly loose when permitting systems on one network access to those on another. For example, someone on the inside sales team sitting in their cubical on their workstation has access to the Customer Relationship Management (CRM) system which resides on a server that is physically somewhere else. The workstation and server are very likely on different subnets or VLAN within the same enterprise, but ACLs exist that enable the sales person’s workstation access to customer data provided by the CRM system. Furthermore, that CRM system is actually pulling the customer data from a third system, a database server. It is possible that the CRM server and the database server may be on the same physical server, or perhaps in the same server rack, but very possibly on the same logical network. The question is, is there a logical path from the inside sales person’s workstation to the database server, the answer should be no, but guess what? It doesn’t matter. Once the inside salesperson is successfully spear fished then it’s only a matter of time before the attacker has access to the database server through the CRM server.

The attacker will first enable the keylogger, then watch the sales person’s screen to see what they are doing, harvest all their user ids and passwords, perhaps turn on the microphone and listen to their conversations, and inspect all the outgoing network connections. Next, the attacker will use what they’ve harvested and learned to begin their assault, first on the CRM server. Their goal at this point is to establish a secondary beachhead with the greatest potential reach from which to launch their primary assault while keeping the inside sales person’s workstation as their fallback position. From the CRM server, they should be able to easily access many of the generic service machines: DNS, DHCP, NTP, print, file, and database systems. The point here is that where external attackers often have to actively probe a network to see how it responds, internal RAT based attacks can passively watch and enumerate all the ports and addresses typically used. In doing so they avoid any internal dark space honeypots, tripwires, or sweep detectors. So how do we protect the New Network Edge, the server itself?

A new layer needs to be added to our defense in depth model called micro-segmentation or application segmentation. This enforces a strict set of policies on the boundary layer between the server and the network. Cisco, Arista, and other switch providers, with a switch-based view of the world, would have you believe that doing it in the switch is the best idea. VMWare, with its hypervisor view of the world, would have you believe that their new NSX product is the solution. Others like Illumio and Tuffin would have you believe that a server-based agent is the silver bullet for micro-segmentation. Then there’s Solarflare, a NIC company, with its NIC based view of the world, and its new entrant in the market called ServerLock.

Cisco sells a product called Tetration designed to orchestrate all the switches within your enterprise and provide finely grained micro-segmentation of your network traffic. It requires additional Cisco servers be installed to receive traffic flow data from all the switches, processes the data, then provides network admins with both the visibility and orchestration of the security policies across all the switches. There are several downsides to this approach, it is complex, expensive, and can very possibly be limited by the ACL storage capabilities of the top of rack switches. As we scale to 100s of VMs per system or 1,000s of containers these ACLs will likely be stretched beyond their limits.

VMWare NSX includes both an advanced virtual switch and a firewall that both require host CPU cycles to operate. Again, as we scale to 100s of VMs per system the CPU demands placed on the system by both the virtual switch and the NSX firewall will become significant, and measurable. Also, it should be noted that being an entirely software-based solution NSX has a large attackable surface area that could eventually be compromised. Especially given the Meltdown and Spectre vulnerabilities recently reported by Intel. Finally, VMWare NSX is a commercial product with a premium price tag.

This brings us to the agent-based solutions like Illumio and Tuffin. We’ll focus on Illumio which comes with two components the Policy Compute Engine (PCE) and the Virtual Enforcement Node (VEN). The marketing literature states that the VEN is attached to a workload, but it’s an agent installed on every server under Illumio’s control and it reports network traffic flow data into the PCE while also controlling the local OS software firewall. The PCE then provides visualization and a platform for orchestrating security policies. The Achilles heel of the VEN is that it’s a software agent which means that it both consumes x86 CPU cycles and provides a large attackable surface area. Large in the sense that both its agent and the OS-based firewall on which it depends can both be easily circumvented. An attacker need only escalate their privileges to root/admin to hamstring the OS firewall or disable or blind the VEN. Like VMWare NSX, Illumio and Tuffin are premium products.

Finally, we have Solarflare’s NIC based solution called ServerLock. Unlike NSX and Illumio which rely on Intel CPU cycles to handle firewall filtering, Solarflare executes its packet filtering engine entirely within the chip on the NIC. This means that when an inbound network packet is denied access and dropped it takes zero host CPU cycles, compared to the 15K plus x86 cycles required by software firewalls like NSX or IPTables. ServerLock NICs also establish a TLS-based domain of trust with a central ServerLock Manager similar to Illumio’s PCE. The ServerLock Manager receives flow data from all the ServerLock NICs under management and provides Visibility, Alerting and Policy Management. Unlike Illumio though the flow data coming from the ServerLock NICs requires no host CPU cycles to gather and transmit, these tasks are done entirely within the NIC. Furthermore, once the Solarflare NIC is bound to a ServerLock Manager the local control plane for viewing and managing the NIC’s hardware filter table is torn down so even if an application were to obtain root privilege there is no physical path to view or manage the filter table. At this point the, it is only capable of being changed from the specific ServerLock Manager to which it is bound. All of the above comes standard with new Solarflare X2 based NICs that are priced at or below competitive Intel NIC price points. ServerLock itself is enabled as an annual service sold as a site license.

So when you think of micro-segmentation would you rather it be done in hardware or software?

P.S. Someone asked why there is a link to a specific RAT or why I’ve included a link to an article about them, simple it validates that these toolkits are in-fact real, and readily accessible. For some people, threats aren’t real until they can actually see them. Also, another person asked, what if we’re using Salesforce.com, that’s ok, as an attacker instead of hitting the CRM server I’ll try the file servers, intranet websites, print servers, or whatever that inside salesperson has access to. Eventually, if I’m determined and the bounty is high enough, I’ll have access to everything.

Onload Recovers Meltdown Lost Performance

The recently announced microprocessor architecture vulnerability known as Meltdown is focused on accessing memory that shouldn’t be available to the currently running program. Meltdown exploits a condition where the processor allows an unprivileged application the capability to continually harvest data unrestricted from anywhere in system memory. The flaw that enables Meltdown is based on microprocessor performance enhancements more than a decade old and are now common in Intel and some ARM processors. The solution to Meltdown is Kernel page-table isolation (KPTI), but it doesn’t come without a performance impact which ranges from 5-30%, every application behaves differently. Since Onload places the communications stack into that application’s userspace this dramatically reduces the number of kernel calls for network operations and as such avoids most of the performance impact brought on by KPTI. Redhat confirm this in a recent article on this topic. This means that applications leveraging Onload on KPTI patched kernels will see an even greater performance advantage.

By contrast Spectre tears down the isolation that exists between running applications. It allows a malicious application to trick error-free programs into leaking their secrets. It does this by scanning the process address space of those programs, and the kernel libraries on which they depend, looking for exploitable code. When this vulnerable code is executed it acts as a covert channel transmitting its secrets to the malicious application. This vulnerability affects a wider range of processors and requires both kernel and CPU microcode patches, and even then, the vulnerability hasn’t been 100% eliminated. More work remains to be done to shut down Spectre.

Post Spectre/Meltdown, Are We More Secure?

It’s human nature in time-critical environments to speculate and execute, remember Radar O’Reilly from M.A.S.H., it’s how we operate at our best. Sometimes you’re wrong and you throw out that work, but the majority of the time you’re right, and the payoff, in the case of an army field hospital could be life-saving. For computer processors, up until recently, that payoff was less critical and the savings could be measured in nanoseconds per successful speculation. Now while nanoseconds on their own don’t sound like much, consider that this savings comes with EVERY successful speculation. Also, it should be noted that there isn’t one block on the processor that is speculating and executing, but rather dozens of them in parallel. Current projections are that patching around the “speculate and execute” set of three flaws could reduce system performance from 5-30%. Friday Apple announced that they’ve patched all their OSes to mitigate just the Meltdown flaw and at worst the performance hit was only 2.5%, they are still working on Spectre. Redhat, which has worked closely with Google’s Project Zero, has not only addressed Meltdown but also Spectre by releasing patches for both sets of flaws.

Many articles and blog posts this week will cover the details of Spectre and Meltdown, and as such, this one won’t. Also, they will devote time on the lengths to which Intel, AMD, ARM, Linux, Apple, Microsoft, Google, Amazon, and others will or have gone to plug these holes, this post won’t. The real underlying question is are we more or less safe today as a result of these discoveries?

My view is that this is Shellshock only three years later and in silicon. Let’s flashback to 2014 when Stephane Chazelas discovered a critical security flaw in the Bash Shell after it had been in production for several decades. Within several days five more security flaws were found in the same code. Code that was widely considered stable and trusted. Clearly, it had never been revisited and retested using state-of-the-art hacking techniques. The same was true with speculative execution. It has been a microprocessor design component since the early days of pipelining, possibly as far back as the original IBM ROMP processor of 1981 (precursor to RISC and later the Power Architecture). It appears that chip security experts are now revisiting old trusted silicon techniques with new eyes looking for potential current exploits. This can only be a good thing as it will further secure the computing platforms on which we rely.

In the near term, patching for Spectre and Meltdown will improve security at the expense of performance, but it will only take another chip spin to likely recover that lost ground. At which point we’ll have safer and more trustworthy platforms.

While originally published on January 4th, 2018, this piece has since been updated on January 6, 2018.

2600, Attacking Enterprise Networks

Since 1984 the magazine “2600” has been the undisputed publication created by and for hackers and phone phreakers worldwide. The January 2017 cover to the right is misleading, the magazine isn’t named after the Atari system, but rather the 2600 Hz tone ATT used, that phreakers leveraged, to control long distance phone lines. Most article bylines are hacker handles, rather than proper names, as the articles themselves don’t always paint inside the numbers. In January 2017 a hacker with the handle “Daelphinux” published the first of a five-part series of articles titled “Successful Network Attacks – Phase One” and every quarter since then he’s published the next phase, with the final Phase Five hitting the streets today January 2, 2018. This collection of five articles is perhaps the most concise executive-level summary of how an attacker breaches an enterprise that I’ve read thus far. Hopefully, I won’t offend Daelphinux by attempting to summarizing the key points of his five articles in this single blog post. I strongly suggest that everyone reading this review the original text in these five issues of 2600.

Daelphinux classified a Successful Network Attack into five phases:

  1. Reconnaissance – “Gathering as much useful information about the targets as possible.”
  2. Scanning – “Gathering useful information about the target’s networks and any possible exploits.”
  3. Gaining Access – “Getting into the network to be able to accomplish the attack’s goal.”
  4. Maintaining Access – “Ensuring access to the network persists long enough to accomplish the attack’s goal.”
  5. Covering Tracks – “Obfuscating the attacker’s presence on the network such that they cannot be traced.”

Each phase builds on the first, with Daelphinux envisioning this as a pyramid with phase one on the bottom, and each successive phase building on the prior one. Attackers will need tools and skills at each level in order to conduct a successful attack. Defenders, the enterprise admins, will also need tools and skills for several phases to detect and defend against an attack.

Phase 1 – Reconnaissance. As defined by Daelphinux the attacker seeks to gather all the raw data they can on their target before actively engaging them. The key here is in gathering only the “useful” data, as an attacker will rapidly accumulate an enormous pile of information. This information should come from a wide variety of sources including, but not limited to: deep web searches, web crawling those results, calling all the targets publicly available phone numbers to learn everything they can about the target, draining “whois” databases for all known corporate DNS assets, launching phishing and email scams at targeted employes and generic email address, real-life social engineering by making new friends, and finally dumpster diving.

All these efforts will produce a heap of information, but most of it will be useless. Here is where intelligent sifting comes in. Important information is often in handwritten notes, or cast of printouts such as printer test configuration pages, IP table listings, usernames, equipment manufacturer shipping boxes, operating system manuals, internal organizational charts & structures, and corporate policies (especially password).

Recognizing these reconnaissance efforts is the initial step toward thwarting an attack. For example, reviewing security footage looking for dumpster divers might sound trivial, but differentiating between an attacker and someone looking something to sell for their next meal can be challenging. Other activities like social engineering become far more complex to detect via video surveillance as these activities appear as background noise. While this phase might be the toughest to detect, it is the easiest to defend against. If you can cut off the flow of information outside enterprise you can seriously hamper their reconnaissance efforts. To do this you can hide the whois records, destroy printed copies of purchase orders, destroy shipping boxes used to pack new servers or appliances, destroy discarded manuals, remove and clean printer hard drives, make sure all in-house shredders use cross-cut shredding and finally burn any really sensitive info that has already been shredded. Much of this can be addressed through procedures and training.

Phase 2 – Scanning, In April Daelphinux covered the details of this phase. He said that when an attacker moves on to phase two they are committed, and unlikely to walk away. This phase is the first real step where the attacker can’t evade detection as they have to deploy active tools to electronically gather as much insight as they can. Tools like: Nmap, Nessus, Metasploit, ZAP, Xenotix or Grabber. Here they are looking to generate IP Maps, enumerate subnets, determine network speeds, the resilience of networks, open ports on clients, appliances & servers used, along with applications and the versions in production.  All this scanning will provide the attacker with another huge heap of data to sift through with the eventual goal being to define the attack vectors that define the real “meat” of the attack.

This phase is the last chance for a defender to stop an attack before valuable assets are stolen. During this phase, administrators might notice network slowdowns, so on detecting these and investigating see if one IP address or a small range of addresses is touching a wide range of resources if so you are likely in the process of being scanned. Attackers will often launch the scanning phase remotely. So using network address translation (NAT) internally, next-generation firewalls, and current IPS & IDS appliances can in some cases detect this. It is always strongly recommended to be current in patching all your appliances and to follow proper admin process.

Phase 3 – Gaining Access, was published in 2600 in July of 2017. Without gaining access an attack isn’t an attack. Some of the key tools used are Metasploit and ZAP, but they will also leverage trojans, zombies, and backdoors to gain multiple toeholds into the enterprise’s network. Attackers often use remote shells instead of graphical tools, as shells often prove to be faster, more flexible and powerful.

Typically attackers operate from an endpoint that is not their intended target, for example leveraging another user’s machine to attack a server. By using another unsuspecting endpoint if it becomes compromised they can then move onto another workstation within the enterprise network. Attackers are typically interested in copying, moving, deleting and or altering files. In this article, we’re not interested in those seeking to launch a Denial of Service (DoS) attack, just those looking to exfiltrate value from the enterprise. Detecting attacks during this phase requires active monitoring, and the question is when, not if, you’ll detect an attacker. Doing active file audits, examining files that are changed outside of regularly expected intervals, watching for irregular traffic patterns, and reviewing access and error logs are all known methods of detecting an attack.

Phase 4 – Maintaining Access, was published in the October 2017 issue. This is the last phase where defending against an attack can successfully prevent data loss. Daelphinux stated that there are three things which should be kept in mind:

  1. The Attacker has already breached the network.
  2. The Attacker is actively attempting to achieve their goal.
  3. Less experienced attackers tend to get overly comfortable with their success at this point.

This is the stage at which attackers are at their most vulnerable. They are moving around in plain sight within your network and can be detected if you’re looking for them. Network monitoring is the key to detection in this phase, but also, unfortunately, detection at this phase is also based on hope. You as the defender are hoping you discover them, and they are hoping their connection won’t get them detected and severed.

Skilled attackers will setup precautions to prevent this. One of the attacker’s strategies will be to take over network monitoring tools in an effort to circumvent detection. Enterprises need a heavy level of paranoia at this point to ensure that they are checking everything. Looking for the use of uncommon ports or protocols is another method for detecting attackers. Typically Intrusion Prevention (IPS) and Intrusion Detection Systems (IDS) and Security Information and Event Management Systems (SIEMS) are useful tools in ferreting out hackers. SIEMS themselves often are now targets for attackers. Multiple instances of these devices or appliances should be leveraged to thwart a takedown. Consider having these monitors watch each other, so in the event, one goes down you can use that as a potential indicator of an attack. If both or all monitors go down simultaneously then it’s very possible you’re under attack. Killing connections that meet certain criteria are vital to cutting off an attackers access, for example terminating connections that last longer than 25 minutes. Think that scene in the original “Transformers” when the general shouts “Cut the hard lines!”

Phase 5 – Covering Tracks, today January 2, 2018, we saw the latest issue of 2600 with Daelphinux final installment in the series on “Successful Network Attacks – Phase Five – Covering Tracks.” Once an attacker has reached phase five they’ve taken what they need, and now the coverup begins. Attempting to defend against this phase “is a form of damage control,” at best you’re preserving forensic evidence to hopefully reconstruct what they took after the fact. An attacker has to leave as cleanly as they entered otherwise they could dilute some of the value of what was taken. As Daelphinux points out, what good is a list of users and passwords if the passwords have all been changed. The same holds true of Malware and backdoors, you can’t sell a backdoor into an enterprise if it has been removed.

The best defense against this is redundant, and perhaps even hidden copies of logs. Attackers will often sanitize the logs they find, but they rarely go looking for additional copies of logs, especially if some effort has been made to hide and even secure them. It’s possible that if you have automated your logging such that multiple copies are generated, and accesses are tracked the attacker will notice this in phase three and just avoid it. Attackers will normally obfuscate both their IP and MAC addresses to further frustrate tracking them, often using addresses already on the network. Again, as mentioned in phase three setting connection limits, timeouts, and alerts when these are reached is often a good way to thwart or even detect an attacker. It should also be noted that attackers will often escalate their privilege on a system so they can disable logging. As Daelphinux noted disabling logging will often generate its own log event, but then after that, you won’t know what was done. Some attackers may even just erase or corrupt log files on the way out the door, a sort of “salt the earth” strategy to make determining what was taken that much more difficult. Regardless, a company will need to make some determinations on what was stolen or affected and alert their customers. A recent Ponemon report states that just cleaning up and dealing with a data breach in the US often costs companies $244/record.

Hats off to Daelphinux for authoring “Successful Network Attacks,” and “2600, The Hacker Quarterly” for publishing it.

I hope everyone has a Happy and Secure New Year.