Trick or Treating and Why Failing Open is Failing Safe

This article was originally purchased in October of 2012 on
Revised on 11/1 with additional technical details found at the end.

Tomorrow children across the US will be going door to door begging for candy, my teenager will be one of them.  Suppose you weren’t home, would you appreciate it if they let themselves in, and rummaged through your home looking for treats?  How about if they also stole your extra front door key, and used your home to party whenever you weren’t home. This is what botnets do. Some years ago it was demonstrated that a new “out of the box Windows PC” without updates survived exactly 20 minutes when attached directly to the Internet.
Some network hardware vendors consider this approach acceptable.  They’ve gone to great lengths to design hardware so that when the server dies the network “fails-closed” (in the electrical sense) meaning that traffic continues to flow through the network unchecked.  Now if this approach was used to enhance the networking experience, perhaps traffic shaping or load balancing, I could see how this might be a selling point. What we need to avoid is having engineers design security appliances that “fail-closed”. Security adapters should always “fail-open” or “fail-safe”. Today unknowingly our firewalls, and systems are probed & assaulted hourly as pawns in a cyber war waged by individuals, corporations & nation states. I bring this up because we recently came across a line of Intrusion Prevention Systems (IPS) that offered this as a feature, seriously.
Imagine you hire a doorman to increase the safety of your home, and early on Halloween evening he dozes and falls off the porch into the bushes sound asleep. This is “failing-closed”. I’d rather if my doorman was to fall asleep that he slumped against the door totally blocking entry “failing-open” or “failing-safe”. How does your 10GbE IPS or IDS fail?
Additional Technical Details
I’ve been asked how “failing-closed” actually works. The simple analogy is a single lens reflex camera. These are the big bulky cameras that professional photographers have used for decades. How they work is simple, light enters through some very expensive lenses on the front of the camera. A mirror reflects that light up to an eye piece so you can compose your picture. When the camera is on, and you depress the shutter release that mirror quickly flips out of the way to expose your film or the CCD in your digital SRL, then the mirror flips back much like the shutter in even older style cameras.
With network adapters that fail open consider the first network port as the lens and the second network port as the eye piece. When the card is powered off it is in bypass mode, “failing-closed”, a mirror (actually an optical MEMS) takes the light entering the first port and sends it to the second and vice-versa. When the network adapter is fully powered on, and the drivers have all successfully loaded the mirror can be flipped open and the 10GbE ASIC chip sees all the traffic and can selectively pass traffic back and forth between the ports. It can also send traffic up to the server, and the server can then send traffic out one or both ports. It’s a very flexible approach, but the use case is not very common and the cards are often priced five times higher than competing dual port cards.
Thanks to Gary Archer from Emulex for suggesting this topic.

Leave a Reply