Detecting encrypted traffic with net-entropy, part two

Back here I described my setup of a modified version of net-entropy, which I was going to use in my quest to detect encrypted traffic. Well, it’s been running for a week or so now, and I’ve got some results.

• Did it detect encrypted traffic? Yes!
• Did it detect stuff that wasn’t encrypted? Yes!
• Did it fail to detect traffic that definitely was encrypted? Yes!
• Was the experiment a total failure and a complete waste of time? No! Far from it, in fact.

The theory I was testing was that traffic with sufficient entropy might be encrypted, since high entropy is a property of decent encryption algorithms. net-entropy was set to trigger an alert if it saw any connections whose entropy crossed an arbitrarily chosen threshold of 7.9 (8.0 is the maximum), and protocols that were expected to be encrypted (HTTPS, etc.) were filtered out.

Here’s a summary of what I’ve found so far:

• Encrypted traffic that crossed the 7.9 threshold included Windows Remote Desktop (RDP), Skype (both calls and signalling traffic), SSH-2, and Google Talk.
• Unencrypted traffic that crossed the threshold was mainly RTMP (streaming Flash audio/video), and possibly Spotify (I don’t know for sure if Spotify uses encryption or not, but high entropy was observed both on the inbound media from port 4070 and the outbound media on random ports). Media protocols like this are usually highly compressed – high entropy is a side effect of compression as well as encryption.
• Encrypted traffic that was not detected included SSH-1 (1.5, to be exact). SSH-2 was detected as one would hope, provided that the session was long enough.

Clearly my blunt approach of a single threshold isn’t the most effective one, as we have both false positives and false negatives. But after applying some visualisations to my results, an intriguing possibility presents itself.

net-entropy was installed in this instance on a Sguil box mainly so that it was in a position where it could see a large volume of real-world traffic. A happy side effect of this is that it’s quite simple to extract the raw traffic captures that each net-entropy alert is referring to. If we’ve built net-entropy with the –enable-statistics option, we are then in a position to draw graphs of the entropy of an individual TCP stream:

• First, use the net-entropy alert to extract the specific TCP stream. The easiest way to do this is to search for it using the Sguil client, and then export the results to Wireshark. Let’s save the capture as session.raw
• Then we run net-entropy over it in statistics mode:

  $ net-entropy -r session.raw -s mystatsdir -F
   -b -c net-entropy.conf

• The output of this is a .dat file whose name is made up of a timestamp and the source and dest IP addresses and ports.
• We can now plot this file in gnuplot:

  plot 'mystatsdir/whateveritwascalled.dat'

By way of a baseline, here is a plot showing the entropy of the first 64KB of an HTTPS and an SSH-2 session. The blue line marks the 7.9 alerting threshold:

Zooming in a little, we can see that HTTPS crossed the threshold after about 2.2KB of data, and SSH-2 took a little longer:

Let’s zoom in a little on a different area of the graph – the little “wobble” on the SSH-2 line:

What we’re looking at here is the part of the conversation where the various parameters of the SSH-2 session are being exchanged (key exchange protocol, encryption/hashing algorithms, etc). These are passed as cleartext, hence the low entropy at this point.

It’s an interesting little pattern, though. Let’s overlay some more SSH sessions onto the one above and see what they look like:

There are three sessions illustrated here:

• The blue line is an SSH-2 session, which in the context of this experiment is a “true positive” since it was encrypted and it did cross the 7.9 threshold
• The red line is another SSH-2 session which was so short in duration it didn’t manage to make it above 7.9. This is a “false negative” because we’ve missed something that definitely was encrypted.
• The green line is an SSH-1 session. At no point during this session’s life did it cross the 7.9 threshold – another false negative.

As far as detecting encrypted traffic goes, this clearly isn’t as useful as I’d have hoped. But look at the red and blue lines – look how tightly they follow one another:

This brings us to the intriguing possibility I alluded to earlier – using entropy analysis not for the detection of encrypted traffic, but for the classification of traffic.

What if the entropy of certain types of traffic is reasonably consistent? What if the patterns above represent “fingerprints” for SSH-1 and SSH-2 traffic? If we could match traffic against a library of fingerprints, we’d have a port-independent classifier of sorts.

I’ve not had time yet to analyse sample sets large enough to be anywhere approaching conclusive, but let’s look at some other kinds of traffic:

The following graph shows four RTMP sessions:

Whilst RTMP isn’t encrypted, all four sessions have a similar visual fingerprint.

Now let’s look at nine RDP sessions (Windows Remote Desktop):

The most obvious outlier here is the black line – this was an RDP session to a server whose encryption level was set to “Low”. If we zoom in a bit, we’ll see another outlier:

The orange line is significantly different to the others. This particular session sent the string “Cookie: mstshash=machinename” in the first data segment sent from the client to the server – the other sessions had mostly zeroes instead, hence the lower entropy at this point. Since this is the very first data segment in the session, we could possibly infer that we’re looking at different client software used to make the connection. Indeed, if we look at RDP sessions from rdesktop (rather than the Windows client), the entropy looks different still:

The entropy is low, relative to the Windows client, and there’s a slightly different signature at the start of the connection:

One might be tempted to think that one could look at graphs like these and infer something about both the server (encryption level in use) and the client (type of software used).

OK. Enough graphs. Summary time.

Detecting encrypted traffic with a straightforward entropy threshold doesn’t seem to be useful. However, we may be able to use entropy analysis as a means to classify traffic in a port-independent manner, but I’ve analysed nowhere near enough traffic to assess whether this could be a viable technique or not (there are bound to be loads of outlying cases that don’t fit the profile). And even if it is a viable technique, are the bases already covered by other port-independent classifiers (l7filter, et al)? That said, I’m not the first person to explore various visualisations of encrypted traffic, so someone somewhere considers the broad concept useful.

Comments welcome!

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)dataline.co.uk