bright ideas

Bash Bug Patching

IT Infrastructure


30 Sep 2014


Does the Bash Bug have to mean a whac-a-mole approach to patching?

Tech and regular media channels are awash with news of the Bash Bug (or Shellshock as it is otherwise known) and how it could be even more serious than the Heartbleed Bug seen earlier this year. If you don’t already know, it’s a security flaw whose vulnerability can be triggered by placing malicious code at the end of a function before it is handled by the Bash shell, which is a default inclusion in the Linux operating system.

This means that a number of VMware virtual appliances are exposed, and even though theses appliances will typically reside behind corporate firewalls and no vulnerability has been demonstrated, VMware will take the pre-cautionary measure of re-releasing them. Bash is also found in Apple's Mac OS X.

Some are saying that this security breach could be one of the most powerful exploits possible in the IT world, and one of the easiest to take advantage of. The bug dates back to version 1.13, making it a 22 year old bug (!) and is rated a perfect 10 out of 10 in terms of severity (impact and exploitability), according to the National Institute of Standards and Technology (NIST). The bug potentially allows hackers to take control of a victim’s computer and run almost any operation, from opening, altering and deleting files to shutting down networks and launching attacks on websites. However, the main threat will be to servers, and in particular those behind websites. Attackers have already repurposed a script written by a cybersecurity blogger that was intended to help people scan for vulnerabilities. It has been modified so as to download malware when vulnerable systems respond. And they even thank him for his contributions with a “Thanks-Rob” in the code now hosted on GitHub. The results of a breach could be a botnet payload that could potentially be used to send spam, participate in denial-of-service attacks on websites or to steal confidential data.

While the Bash exploit has big ramifications, it's unlikely to be something that will impact customers who segment off their servers from wide-open proxy access. However, if an exploit does breach the perimeter then all vulnerable systems would potentially be exposed. This is explained with the popular aardvark analogy, where the exterior perimeter is a hard shell but the interior is soft and squishy – and it’s also another reason to seriously consider and then incorporate a micro-segmentation approach (like VMware NSX) into your security strategy, where internal traffic (East-West traffic) is guided through virtual security control points (firewall, IDS, IPS etc.) implemented in the virtual networking layer.

First - fight the immediate fire

The first thing that you need to do is emergency patching. A partial fix has been released, and a full fix is expected to be released shortly.

The only challenge with this is that Bash is embedded in a huge number of different devices and it may take some time to find and fix them all, meanwhile leaving your system in a vulnerable state. And obviously having to patch and re-patch isn’t ideal and can create a significant operational burden on your team.

Second - take steps against future attacks

Get over the patching hurdle

Our experience when it comes to patching is that many organisations have trouble just managing regular, scheduled patching and maintenance, let-alone an emergency patch situation. In most large organisations the negotiation of maintenance windows between teams in Infrastructure, Applications and the Business is akin to a climate control summit in Copenhagen. The net result is that they can only be done infrequently (if at all in some extreme cases), and when they can be done it often results in high cost as there is a lot of co-ordination planning (multiple roll-up updates) and typically incurs afterhours overtime.

And the problem with scheduled patching is that you remain vulnerable from the moment the hackers find the vulnerability to the moment you can install the patch. For many this could be months or weeks, but even an hour is an hour too long to be comfortable with that sort of security risk.

Trend Micro virtual patching

One of the solutions to patching and maintenance challenges that we’ve been exploring and implementing with a few of our customers is the Trend Micro virtual patching solution. Primarily we’ve made this recommendation to reduce the cost associated with patch management, but it also has a significant parallel benefit of providing rapid (near immediate) protection against vulnerabilities such as that of the Bash Bag.

Trend Micro call this Deep Security and essentially what it does is provide virtual patches to protect your servers and endpoints until such time as the patches can be installed - or indefinitely for out-of-support or un-patchable systems. The solution is able to deliver immediate protection whilst also eliminating the costs and operational pains of emergency patching, frequent patch cycles, breaches, and costly downtime.

Virtual patching in a VMware context

In the VMware context Trend Micro virtual patching can provide a complete agent-less protection capability for virtual machines, which has a number of benefits:

Ability to provide “virtual patching” as a compensating control, without having to install anything on the virtual machine
High performance via offloading to the (now protected!) Virtual Appliances on the ESXi hosts
Operational improvements and efficiencies to significantly reduce risk
Trend Micro offers a free trial of the software on their website, however in our experience this is not something that can be rushed into production environments as it requires pre-requisite and interoperability validations, and generally a Proof of Concept in order to justify the investment.

There are huge benefits for organisations who continue to struggle with the potential risk and incurred downtime from large scale patch deployments, as well as benefits for managing the spiralling costs that come with managing the patching and maintenance processes. One customer’s quick calculations showed that the cost of the Trend Micro solution was more than offset by the reduction that would result to the maintenance window operating costs they bear.

Other options

There may be even more options, but a notable call-out is to use application-centric, next-generation firewalls such as those from Palo Alto to provide immediate (HTTP in the case of Bash) exploit protection.


Whilst the horse may have bolted on this particular vulnerability and emergency outage windows for patching (and re-patching) are the only option for now, it is well worth considering implementing a rapid response solution such as Trend Micro virtual patching to provide immediate and less disruptive protection for the next one. This approach is very much in line with the vision that we share at ViFX for the Software Defined Data Centre (SDDC), where pre-defined policy supported by the implementation of the most appropriate technologies, means less human intervention, and more agility, risk mitigation and automation.

Do you believe the Bash Bug will have serious implications? What solutions is your organisation putting in place to address these ever-increasing security vulnerabilities?

Author: James Knapp

James focuses on the cloud technology landscape, developing cloud strategies for our clients, as well as being a regular conference presenter on new IT service delivery models.

30 September 2014 / 0 Comments