Virtual Private Onions

If you’ve not checked out Security Onion (SO) yet, you really should. It’s a powerhouse Linux distro, running everything an analyst could need to carry out effective Network Security Monitoring (NSM). The latest beta is looking fantastic; watch the video and try it out, you won’t be sorry!

I’m not going to talk about SO itself (Mother Google has plenty of good things to say); instead I’m going to look at the underlying network infrastructure.

A major component of SO is Sguil, a distributed monitoring system that has sensor and server components that are used for monitoring the network and aggregating output from multiple sensors (you can see a Sguil schematic here to understand how it all fits together). The comms between the sensors and the server are protected by SSL, and SO builds on this with autossh tunnels to protect other traffic streams from interception. Ubuntu’s Uncomplicated Firewall (UFW) opens only the necessary ports to allow SO sensors to talk to their server, so all the installations are protected to a high degree.

If all of the connections between the sensors and the server are on your own infrastructure, this is fine (perhaps you have a management VLAN specifically for this purpose, for example). There is however another use case I can think of, where we like to use a slightly different method of sensor-to-server communications.

NSM As A Service?

Let’s say you’re in the business of selling NSM to clients, either short-term (“help me understand what’s on my network”) or long-term (“please be our NSM operators in return for huge bushels of cash”). This will probably involve dropping off SO sensor boxes on customer networks and having them all report in to an SO server back at base. The SO sensors will probably be behind the customer’s NAT and firewalls, but this doesn’t matter, the comms will still work. The network of sensors will look a bit like this:

This particular use case isn’t without issue, though – here is where I think that a slightly different approach to the underlying SO infrastructure has value. Consider the following:

• The Sguil interface shows you that one of the agents on one of the sensors is down. How do you ssh in to fix it if the sensor is behind someone else’s NAT and firewalls? How do you even know which Internet IP address you should use to try?
• You suspect one of the sensors is under excessive load. It’d be great to install an snmp agent on the sensor and point Munin at it to try and find the stressed metric. Again, how do we contact the sensor from back at base?
• You want to implement a custom information flow from the sensor back to the server. You need to think about the security of the new flow – can you implement another autossh tunnel? What if the flow is UDP (e.g., snmp-trap)?
• One of the sensors is reported lost or stolen. How can you easily stop it from connecting back to base when it’s possibly now in the hands of threat actor du jour?

One possible answer is to build a custom “underlay” network to carry all of the traffic between SO sensors and the server, catering for existing flows and anything else you think of in the future. As a side benefit, the sensors will all have static and known IP addresses, and contacting them from the server will be no problem at all.

Virtual Private Onions

We’ll accomplish this by using OpenVPN to create a secure, privately- and statically-addressed cloud that the sensors and server will use to talk to each other (sorry for using the word “cloud”!). Take our starting point above, showing our distributed sensors at client sites behind client firewalls. By “underlaying” the SO comms with an OpenVPN cloud we’ll end up with is something like this:

The client’s firewalls and NAT are still there, but they’re now hidden by the static, bi-directional comms layer provided by OpenVPN. The sensors all talk over the VPN to the server at 10.0.0.1; the server in turn is free to contact any of the sensors via their static addresses in the 10.0.0.0/24 subnet.

Details

I don’t intend for this to be a HOWTO; I want to wait until people tell me that what I’m proposing isn’t a galactic waste of time and completely unnecessary! Conceptually, it works like this:

The SO Server takes on the additional role of an OpenVPN server. Using OpenVPN’s “easy-rsa” feature, the Server also becomes a Certificate Authority capable of issuing certificates to itself and to the sensors.

OpenVPN gives the SO Server a routed tun0 interface with the IP address 10.0.0.1/24 (note routed – if we wanted to, we can give other subnets back at base that are “behind” the Server access to the SO Sensors via the VPN).

The Sensors become OpenVPN clients, using certificates from the Server to mutually authenticate themselves to it. The Sensors know to point their OpenVPN clients at the static Internet-facing IP address of the server back at base, getting a tun0 interface and a static IP address out of 10.0.0.0/24 in return.

When setting up the SO components on the Sensor boxes, the sosetup script is told that the server is at 10.0.0.1 via tun0, ensuring that all SO comms go over the VPN.

The VPN carries all traffic between Sensors and Server, be it Sguil’s SSL connections, autossh tunnels, SNMP traps or queries, raw syslog forwarding, psychic lottery predictions, anything. The Server is also free to ssh onto the Sensors via their static VPN addresses.

Advantages?

We’ve implemented OpenVPN with SO in this manner, and it works well for us. I hope this approach has some advantages:

• It makes for easier firewalling at the Server and Sensor ends. You only need to open up one port, which is by default 1194/UDP. All of the VPN traffic is carried over this port.
• You have deterministic sensor IP addressing, something you wouldn’t otherwise have had when your sensors were deployed on foreign networks behind NAT, firewalls, and dynamic IP addresses on customer DSL routers.
• You don’t need to worry about setting up additional autossh tunnels when you dream up a new flow of information  between Sensor and Server.
• You can easily talk to the sensors from the server, for ssh sessions, SNMP polls, or anything else.
• If a sensor is lost or stolen, all you need to do is revoke its certificate at the Server end and it won’t be able to connect to the VPN (more on sensor security in a moment).

There are of course some disadvantages:

• Complexity, which as we all know is the enemy of security.
• Double encryption. OpenVPN is encrypting stuff, including stuff already encrypted by Sguil and autossh.
• <insert other glaring flaws here>

Security considerations

On the Server side, keep in mind that it is now a Certificate Authority capable of signing certificates that grant access to your SO VPN. Guard the CA key material well – maybe even use a separate system for certificate signing and key generation. Also, once you’ve generated and deployed certificates and private keys for your sensors, remove the private keys from the Server – these belong on each specific sensor and nowhere else.

On the Sensor side, we have a duty of care towards the clients purchasing the “NSM As A Service”. OpenVPN should be configured to prohibit comms between Sensor nodes, so that a compromise of one Sensor machine doesn’t automatically give the attacker a path onto other customer networks.

If a Sensor machine is lost or stolen whilst deployed at a customer site, we can of course revoke its certificate to prevent it from connecting to the VPN. That’s great, but the /nsm directory on it is still going to contain a whole bunch of data that the client wouldn’t want turning up on eBay along with the stolen sensor.

I have a plan for this, but I’ve not implemented it yet (comments welcome!). The plan is to have the /nsm directory encrypted (a TrueCrypt volume? something else?), but to have the decryption key stored on the SO Server rather than on the Sensor. When the Sensor starts up, it builds its OpenVPN connection to the Server before the SO services start. Using the VPN, the Sensor retrieves the decryption key (one unique key per Sensor) from the Server (wget? curl? something else?), taking care never to store it on the Sensor’s local disk. Decryption key in hand, the Sensor can decrypt /nsm and allow the SO services to kick off.

In this way, a stolen Sensor box represents minimal risk to the client as the /nsm directory is encrypted, with the keymat necessary for decryption stored on the Server. Provided that the theft is communicated to the NSMAAS provider in an expedient fashion, that keymat will be out of reach of the thief once the OpenVPN certificate is revoked.

(Other, crazier, plans involve giving the Sensors GPS receivers and 3G comms for covert phone-home-in-an-emergency situations…)

So there you have it…

…Virtual Private Onions. Any comments?

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

Eyesight to the Blind – SSL Decryption for Network Monitoring

Here’s another post I wrote for the InfoSec Institute. This time, the article shows how to add SSL decryption to your NSM infrastructure, restoring the eyesight of sensors blinded by the use of SSL.

You can read the article here; comments welcome, as always!

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

Tweaking EAP’s weak link – sucking WiFi PMKs out of RADIUS with pmkXtract

When faced with a WiFi network using WPA2/EAP-TLS, options for a direct attack are limited. You can’t feasibly attack the encryption (most likely CCMP), and if the client devices are configured properly attacking the authentication mechanism directly isn’t likely to bear fruit either.

One of the advantages of the “enterprise” variants of WPA is that the Pairwise Master Keys are specific to a client device and to a single association (WPA Personal has the same PMK shared amongst all devices). To oversimplify somewhat, with WPA Enterprise the PMK for a given device is negotiated between the device and the RADIUS server as part of the EAP process. The PMK is subsequently delivered to the access point via RADIUS. Once both the client device and the AP have the PMK the normal WPA four-way handshake can complete, the output of which is a Pairwise Transient Key which is used to encrypt the session. There’s a nice diagram of the process here.

However, there is a weak link in the chain. Hacking Exposed Wireless, 2nd edition describes a possible attack against the delivery of the PMK from the RADIUS server to the access point. The theory of operation is as follows:

1. Start capturing the wireless traffic of interest.
2. Start capturing the RADIUS traffic between the RADIUS server and the access point.
3. Using an obtained RADIUS shared secret, decrypt the RADIUS traffic that contains the PMKs.
4. Use the extracted PMKs and the captured four-way handshakes to derive each device’s PTK.
5. Decrypt every single data frame you’ve captured from the wireless network 🙂

The book didn’t cite any tools capable of extracting the PMKs from the RADIUS conversation, so I thought I’d have a go at writing one myself. The reason the book didn’t cite any tools is probably that if an attacker is in a position to capture your RADIUS traffic, you probably have bigger things to worry about. Nevertheless, it seemed like a challenge worth attempting.

Of the five points above, 1 and 2 are straightforward (assuming the attacker is well placed). 4 and 5 can be easily carried out with Wireshark. Step 3 breaks down into obtaining the shared secret and decrypting the PMKs.

Obtaining the RADIUS shared secret

As noted in Hacking Exposed Wireless, a brute force attack on the shared secret is quite feasible. Alternatively, one can just poke around the RADIUS server and find the list of secrets. For example:

• FreeRADIUS stores all of this in clients.conf
• Internet Authentication Service on Windows 2K3 stores everything in system32\ias\ias.mdb which you can open in MS Access (look in the Objects table)
• Network Policy Server on Windows 2k8 stores everything in system32\ias\ias.xml in plaintext.

Secrets in hand, we can move on to…

Decrypting the PMKs

We first need to start capturing the WiFi traffic of interest. We can do this with airodump-ng:

airmon-ng start wlan0
airodump-ng –channel 1 –bssid 00:1c:df:8b:50:66 –write pmkXtract mon0

Now we can turn our attention to capturing the PMKs. The PMKs are transmitted in the final RADIUS message of the transaction (the Access-Accept message), and they’re stored encrypted in a vendor-specific attribute, MS-MPPE-Recv-Key:

Given the correct shared secret, Wireshark is apparently capable of decrypting RADIUS traffic but I couldn’t make it work for MS-MPPE-Recv-Key. Shying away from actually writing decryption code from scratch, I invoked the age-old software engineering tradition of “code re-use”, aka pinching someone else’s stuff.

hostapd is a user-space access point daemon, and has full support for EAP-TLS. After nosing around and debugging the code, I found the procedure that does the decryption – decrypt_ms_key() in radius.c.

pmkXtract

pmkXtract is a quick and dirty hack that wraps up decrypt_ms_key(); all we need to do is feed it:

• The RADIUS shared secret
• The request authenticator from the final Access-Request message
• The MS-MPPE-Recv-Key from the following Access-Accept message

A neater, future version of pmkXtract would capture the latter two straight off the wire, but for now they are CLI parameters. The most convenient way to get these values is with tshark, supplying the appropriate capture filter:

tshark -T fields -e radius.code -e radius.id -e radius.authenticator -e radius.MS_MPPE_Recv_Key -E separator=! ip host access.point.ip.addr

This will produce output like this:

<snip>
1!10!a0:29:27:a6:60:a8:f4:cb:4c:1d:b3:fa:46:b4:63:e4!
11!10!8b:eb:bf:dd:47:b8:46:35:92:33:6d:bb:ec:23:e8:2f!
1!11!a2:9f:92:ec:93:5b:05:ef:f6:a1:82:0a:6c:dd:37:52!
11!11!85:f7:35:08:39:da:f1:0c:4a:3e:fa:af:00:e7:af:d4!
1!12!5d:5c:3c:99:86:fc:d6:2e:a1:fc:fb:85:7e:6d:69:5d!
2!12!9e:13:52:10:ab:b9:74:99:37:ab:8d:b9:ec:3f:b0:9a!80:8a:7c:
d5:62:6c:0a:d9:ad:5b:0f:e5:5b:1d:56:d5:59:38:7a:06:ab:
b1:65:8a:a6:b4:a6:9f:b2:ea:d9:32:49:d2:3a:f0:ba:d0:b1:
77:fc:e5:b9:77:fe:dc:c2:04:7d:79

What we’re looking for is a pair of code1/code2 messages with the same ID, as is the case with the last two lines above with ID 12. Code 1 denotes Access-Request, code 2 is Access-Accept. Now we can pass the message authenticator from the access-request and the MS-MPPE-Recv-Key from the access-accept to pmkXtract, along with the shared secret:

C:\>pmkXtract secret 5d:5c:<snip>:69:5d 80:8a:<snip>:7d:79
PMK is:c8cad1f342e431bdf97b3fd1c37bf2fe
d74b9c785d302430ea418a04998b9ee4

Score! We’ve got the PMK.

Decrypting the captured WiFi traffic

The first thing we need to do is look through our airodump capture for the WPA four-way handshake:

Frames 134, 136, 138, and 140 are the four-way handshake

As we might expect, the contents of the capture is unintelligible:

Now we can enter the PMK that pmkXtract gave us, by going Edit->Preferences, and selecting IEEE802.11 under Protocols:

And hey presto, we have decrypted the traffic. Note the “Decrypted CCMP data” sub-tab at the bottom of the window:

As I’ve already said, if an attacker can pull this off, they already have a deep foothold in your infrastructure. Use strong RADIUS secrets, secure the list of secrets on the server (clients.conf, ias.mdb, etc) as well as you can, and if it’s possible, protect the RADIUS exchange with IPSec.

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

Decrypting SSL traffic with Wireshark, and ways to prevent it

A neat feature of Wireshark is the ability to decrypt SSL traffic. This post is about why you might want to do it, how to do it, why it works, and how to decrease the chances of other people being able to decrypt your “secure” traffic.

Why decrypt SSL?

Aside from the obvious malicious uses, decrypting SSL has uses such as:

• Debugging applications that run over SSL (HTTP, SMTP, POP3, IMAP, FTP, etc).
• Feeding a decrypted traffic stream to an IDS. Having the best signatures in the world won’t help if all your sensors see is encrypted traffic.
• Learning about SSL. What better way to understand something than to take it apart and put it back together again?

I’m running the 2012 Brighton Half Marathon in aid of Help for Heroes – please sponsor me if you can by clicking the link to the right:

How to decrypt SSL with Wireshark

Step one – set up an SSL-protected server to use as a testbed

To illustrate the process, we’re going to use OpenSSL to generate a certificate and act as a web server running HTTP over SSL (aka HTTPS) – it’s quite straightforward.

To begin with, we need to get ourselves a self-signed certificate that our HTTPS server can use. We can do this with a single command:

openssl req -x509 -nodes -newkey rsa:1024 -keyout testkey.pem -out testcert.pem

OpenSSL will ask you for some input to populate your certificate with; once you’ve answered all the questions, the output of this command is two files, testkey.pem (containing a 1024 bit RSA private key) and testcert.pem (containing a self signed certificate). PEM (Privacy Enhanced Mail) format files are plaintext, and consist of a BASE64 encoded body with header and footer lines. You can look at the contents of your key and certificate files in more detail like this:

openssl rsa -in testkey.pem -text -noout (output here)
openssl x509 -in testcert.pem -text -noout (output here; more info here)

We need to perform one tiny tweak to the format of the private key file (Wireshark will use this later on, and it won’t work properly until we’ve done this):

openssl rsa -in testkey.pem -out testkey.pem

Now we’re ready to fire up our HTTPS server:

openssl s_server -key testkey.pem -cert testcert.pem -WWW -cipher RC4-SHA -accept 443

The -key and -cert parameters to the s_server command reference the files we’ve just created, and the -WWW parameter (this one is case sensitive) causes OpenSSL to act like a simple web server capable of retrieving files in the current directory (I created a simple test file called myfile.html for the purposes of the test).

The -cipher parameter tells the server to use a particular cipher suite – I’m using RC4-SHA because that’s what’s used when you go to https://www.google.com. The RC4-SHA cipher suite will use RSA keys for authentication and key exchange, 128-bit RC4 for encryption, and SHA1 for hashing.

Having got our server up and running, we can point a browser at https://myserver/myfile.html and retrieve our test file via SSL (you can ignore any warnings about the validity of the certificate). If you’ve got this working, we can move on to…

Step two – capture some traffic with Wireshark

Fire up Wireshark on the server machine, ideally with a capture filter like “tcp port 443” so that we don’t capture any unnecessary traffic. Once we’re capturing, point your browser (running on a different machine) at https://myserver/myfile.html and stop the capture once it’s complete.

Right-click on any of the captured frames and select “Follow TCP stream” – a window will pop up that’s largely full of SSL-protected gobbledegook:

Step three – configuring Wireshark for decryption

Close the TCP Stream window and select Preferences from Wireshark’s Edit menu. Expand the “Protocols” node in the tree on the left and scroll down to SSL (in newer versions of Wireshark, you can open the node and type SSL and it will take you there).

Once SSL is selected, there’s an option on the right to enter an “RSA keys list”. Enter something like this:

10.16.8.5,443,http,c:\openssl-win32\bin\testkey.pem

You’ll need to edit the server IP address and path to testkey.pem as appropriate. If this has worked, we’ll notice two things:

• Wireshark’s SSL dissector can look into otherwise encrypted SSL packets and dissect the protocol inside:
  
• We can right-click on any of the captured frames that are listed as SSL or TLS and select “Follow SSL stream”:

Nice 🙂

You can read about this step in the Wireshark Wiki here.

Why it works

So, why does this work? Our test server, in common with a very large proportion of HTTP-over-SSL webservers, is using RSA to exchange the symmetric session key that will be used by the encryption algorithm (RC4 in this case). Below is an extract from RFC2246:

F.1.1.2. RSA key exchange and authentication

With RSA, key exchange and server authentication are combined. The public key may be either contained in the server’s certificate or may be a temporary RSA key sent in a server key exchange message.

After verifying the server’s certificate, the client encrypts a pre_master_secret with the server’s public key.

The server can of course decrypt the pre_master_secret passed to it by the client (by using the server’s private key in testkey.pem), and subsequently both the client and the server derive the master_secret from it – this is the symmetric key that both parties will use with RC4 to encrypt the session.

But the server isn’t the only one with the private key that corresponds to the public key in the server’s certificate – Wireshark has it as well. This means it is able to decrypt the pre_master_secret on its way from the client to the server, and thereafter derive the master_secret needed to decrypt the traffic.

How to prevent decryption

There are at least two methods to approach this:

Method one – Protect the server’s private key

Protection of one’s private key is at the core of any system using asymmetric keys. If your private key is compromised, the attacker can either masquerade as you or they can attempt to carry out decryption as outlined above. Keys stored in separate files like the ones above are particularly vulnerable to theft if access permissions are not set strictly enough, or if some other vulnerability allows access. Certain operating systems like Windows and Cisco’s IOS will try to protect the keys on your behalf by marking them as “non-exportable”. This is meant to mean that the OS won’t divulge the private key to anyone under any circumstances, but clearly there comes a point where some software running on the box has to access the key in order to use it. This simple fact can sometimes be exploited to export non-exportable keys – a practical attack is described in a past edition of Hakin9 magazine. You can download it for free; here’s a quote from the article:

The Operating System (Windows XP in this case) does not let you export the private key of a certificate if it is marked as non-exportable. However, the OS must have access to read the private key in order to use it for signing and encrypting. If the OS can access the private key and we control the OS, then we can also access the private key.

Method two – Don’t use RSA for key exchange

As we’ve seen, the RSA key exchange is susceptible to interception if one is fortunate enough to have the server’s private key. By using a flavour of Diffie-Hellman for key exchange instead, we can rule out any chance of an attacker feasibly decrypting our SSL traffic even if they are in possession of the server’s private key. With your existing server certificates and keys, run the server like this and get your browser to fetch https://myserver/myfile.html again:

openssl s_server -key testkey.pem -cert testcert.pem -WWW -cipher AES256-SHA -accept 443

We’ve changed the cipher from RC4 to 256 bit AES – this step is just to prove that Wireshark can decrypt AES as well as RC4.

Now, to break the decryption, alter the cipher suite to use Diffie-Hellman instead of RSA for key exhcange:

openssl s_server -key testkey.pem -cert testcert.pem -WWW -cipher DHE-RSA-AES256-SHA -accept 443

If you’re using a browser other than IE (my IE8 doesn’t seem to support DH with RSA certificates), Wireshark is totally unable to decrypt the HTTP traffic, even though it is in possession of the server’s private key.

A DH key exchange is by design resistant to eavesdropping, although can be susceptible to a man-in-the-middle attack unless both parties identify themselves with certificates. It’s also, as we’ve seen, not universally supported by common SSL clients. But at least it rules out the possibility of some wiseguy with Wireshark sticking his fins where they’re not wanted!

Automatic inline SSL decryption

I have made some improvements to the viewssld package, which allows inline SSL decryption on your Snort/Sguil/etc boxes. You can read all about it here.

If you are interested in becoming an IT professional
you may want to consider a CCNA training course.

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)dataline.co.uk