Detecting encrypted traffic with net-entropy, part one

I’ve been pondering the possibility of detecting encrypted traffic crossing a network, and I think I’m getting somewhere (not necessarily closer to the goal, but somewhere nonetheless!). My initial thoughts were to put some kind of frequency analysis to the task, and whilst I was researching this I came across net-entropy.

net-entropy is a clever tool that can learn the expected cumulative packet entropy (“randomness”) for a given protocol, and raise alerts if an observed connection falls out of bounds (there’s a very detailed writeup here). The theory is that if someone is attacking a flaw in some cryptographic software (a SSH server, for example), the observed entropy of the connection will decrease unexpectedly once the attack has been executed and the attacker is delivering shellcode or whatever (figures two and three here illustrate the principle).

net-entropy was designed to focus on its list of pre-learned protocols, each of which is described in a protospec file. Here is the file for SSH:

Port: 22
Direction: both
Cumulative: yes
RangeUnit: bytes
# Range: start    end      min_ent        max_ent
Range:   1        63       0              4.38105154
Range:   64       127      4.22877741     4.64838314
Range:   128      255      4.95194340     5.02499151
Range:   256      511      4.86894369     7.28671360
Range:   512      1023     4.86310673     7.59574795
Range:   1024     1535     4.94409609     7.74570751
Range:   1536     2047     5.77497149     7.81915951
Range:   2048     3071     6.44314718     7.85139179
Range:   3072     4095     7.17234325     7.92034960
Range:   4096     8191     7.46498394     7.96606302
Range:   8192     65536    7.82608652     7.99687433

Each range is defined in terms of start byte and end byte, and minimum and maximum entropy. For example, for the first 63 bytes, the entropy is expected to be between 0 and 4.38105154 – an alert is raised if the entropy at this point is either too high or too low.

We could have a go at detecting encrypted traffic (rather than profiling its properties) with a very simple protospec file. What I’m interested in seeing is anything with an observed entropy that’s greater than some defined threshold – this will be my indicator that what we’re looking at could possibly be encrypted. So, we could have a protospec file that looks like this:

Port: "whatever"
Direction: both
Cumulative: yes
RangeUnit: bytes
# Range: start    end      min_ent        max_ent
Range:   1        65536    0              7.9

This file will cause net-entropy to raise an alert if the entropy for a connection on port “whatever” exceeds my arbitrarily chosen threshold of 7.9 in the first 64KB of its life; the problem here is that I’d have to write thousands of these files to cover the complete set of all TCP ports. I spoke to net-entropy’s author, Julien Olivain, about this and he very kindly implemented me an “all” feature, whereby a single protospec file can be applied to the complete range of TCP ports (updated source code is available here).

Now we can start to experiment! net-entropy will accept the usual variety of capture filter, so we can use this to exclude:

  • The protocols that we expect to be encrypted (SSH, HTTPS, etc.)
  • High volume protocols that are scrutinised by other means (SMTP, HTTP, etc.)
  • Non-TCP protocols (net-entropy only works for TCP at the moment)

So, our net-entropy.conf file looks like this:

Interface: eth1
RuntimeUser: nobody
MemoryLimit: 131072
MaxTrackSize: 65536
PcapFilter: tcp and not port 80 and not port 25 and not port 22 and
            not port 443 and not port 993 and not port 995
ProtoSpec: /usr/local/share/net-entropy/protospec/proto-tcp-all.nes

I installed the software on a Sguil box and fired it up; pretty soon, things like this were popping up in /var/log/messages:

Sep 17 11:15:03 morpheus net-entropy[2689]: RISING ALARM on -> offset=2406 packets=7 entropy=7.90993547 range=0 (1 65536 0.000000 7.900000)

Woohoo! Data! Now all we have to do is work out if it’s useful or not. I’m not one for leaving logs lying idly on the server that generated them so I send the messages to a remote syslog collector, in this case a Cisco CS-MARS. The MARS certainly has its flaws and niggles, but it does let you write custom parsers for devices it doesn’t know about. Once the MARS has been educated in the ways of net-entropy, you can use its querying mechanism to start exploring the data.

I’ve written the required custom parsers, and exported them as a Device Support Package that you can import into your own MARS, if you happen to have one and want to play along (download it here). The net result is that I can do stuff like:

  • Ask about the kinds of messages from net-entropy:

    Event Types

    Event Types

  • See the details of sessions seen:



  • Drill down onto a single session:
    A single session

    A single session

    Note that the MARS has noticed that there are two events talking about the same session (based on the IP addresses and ports), and has been able to correlate them together into a single session.

  • Get the raw messages as raised by net-entropy:

    Raw messages from net-entropy

    Raw messages from net-entropy

So, here’s where we’re at:

  • net-entropy has been enhanced to support a protospec file that applies to all ports
  • This allows us to do “generic” entropy detection
  • Events from net-entropy are being exported to my MARS, from which I can run queries and reports

Next steps:

  • Work out if the things that net-entropy is alarming on are actually encrypted, or if the reason for their high entropy is something else (effective compression, for example). If I’m not reliably detecting encryption, then I can either tweak my entropy threshold, or give up the whole idea 😉
  • If the technique is really yielding useful results, perhaps write an agent for Sguil so that net-entropy’s alerts appear in the client for easy drill-down onto the session transcripts (there’s an agent available for modsec, so it could be feasible to try this)
  • In the far future, how about a mod to the Sguil client that lets you right-click and say “Graph session entropy”? This would extract the relevant session from the full-content capture (just like the Wireshark option does at present), run it through net-entropy in statistics mode, and use gnuplot to visualise the result.

This post is most definitely filed under “Crazy Plans”. Comments on my insanity are welcome.

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: