Detecting encrypted traffic with net-entropy, part one
I’ve been pondering the possibility of detecting encrypted traffic crossing a network, and I think I’m getting somewhere (not necessarily closer to the goal, but somewhere nonetheless!). My initial thoughts were to put some kind of frequency analysis to the task, and whilst I was researching this I came across net-entropy.
net-entropy is a clever tool that can learn the expected cumulative packet entropy (“randomness”) for a given protocol, and raise alerts if an observed connection falls out of bounds (there’s a very detailed writeup here). The theory is that if someone is attacking a flaw in some cryptographic software (a SSH server, for example), the observed entropy of the connection will decrease unexpectedly once the attack has been executed and the attacker is delivering shellcode or whatever (figures two and three here illustrate the principle).
net-entropy was designed to focus on its list of pre-learned protocols, each of which is described in a protospec file. Here is the file for SSH:
Port: 22 Direction: both Cumulative: yes RangeUnit: bytes # Range: start end min_ent max_ent Range: 1 63 0 4.38105154 Range: 64 127 4.22877741 4.64838314 Range: 128 255 4.95194340 5.02499151 Range: 256 511 4.86894369 7.28671360 Range: 512 1023 4.86310673 7.59574795 Range: 1024 1535 4.94409609 7.74570751 Range: 1536 2047 5.77497149 7.81915951 Range: 2048 3071 6.44314718 7.85139179 Range: 3072 4095 7.17234325 7.92034960 Range: 4096 8191 7.46498394 7.96606302 Range: 8192 65536 7.82608652 7.99687433
Each range is defined in terms of start byte and end byte, and minimum and maximum entropy. For example, for the first 63 bytes, the entropy is expected to be between 0 and 4.38105154 – an alert is raised if the entropy at this point is either too high or too low.
We could have a go at detecting encrypted traffic (rather than profiling its properties) with a very simple protospec file. What I’m interested in seeing is anything with an observed entropy that’s greater than some defined threshold – this will be my indicator that what we’re looking at could possibly be encrypted. So, we could have a protospec file that looks like this:
Port: "whatever" Direction: both Cumulative: yes RangeUnit: bytes # Range: start end min_ent max_ent Range: 1 65536 0 7.9
This file will cause net-entropy to raise an alert if the entropy for a connection on port “whatever” exceeds my arbitrarily chosen threshold of 7.9 in the first 64KB of its life; the problem here is that I’d have to write thousands of these files to cover the complete set of all TCP ports. I spoke to net-entropy’s author, Julien Olivain, about this and he very kindly implemented me an “all” feature, whereby a single protospec file can be applied to the complete range of TCP ports (updated source code is available here).
Now we can start to experiment! net-entropy will accept the usual variety of capture filter, so we can use this to exclude:
- The protocols that we expect to be encrypted (SSH, HTTPS, etc.)
- High volume protocols that are scrutinised by other means (SMTP, HTTP, etc.)
- Non-TCP protocols (net-entropy only works for TCP at the moment)
So, our net-entropy.conf file looks like this:
Interface: eth1 RuntimeUser: nobody MemoryLimit: 131072 MaxTrackSize: 65536 PcapFilter: tcp and not port 80 and not port 25 and not port 22 and not port 443 and not port 993 and not port 995 ProtoSpec: /usr/local/share/net-entropy/protospec/proto-tcp-all.nes
I installed the software on a Sguil box and fired it up; pretty soon, things like this were popping up in /var/log/messages:
Sep 17 11:15:03 morpheus net-entropy: RISING ALARM on 212.7x.aaa.bbb:1708 -> 82.4x.aaa.bbb:60970 offset=2406 packets=7 entropy=7.90993547 range=0 (1 65536 0.000000 7.900000)
Woohoo! Data! Now all we have to do is work out if it’s useful or not. I’m not one for leaving logs lying idly on the server that generated them so I send the messages to a remote syslog collector, in this case a Cisco CS-MARS. The MARS certainly has its flaws and niggles, but it does let you write custom parsers for devices it doesn’t know about. Once the MARS has been educated in the ways of net-entropy, you can use its querying mechanism to start exploring the data.
I’ve written the required custom parsers, and exported them as a Device Support Package that you can import into your own MARS, if you happen to have one and want to play along (download it here). The net result is that I can do stuff like:
- Ask about the kinds of messages from net-entropy:
- See the details of sessions seen:
- Drill down onto a single session:
Note that the MARS has noticed that there are two events talking about the same session (based on the IP addresses and ports), and has been able to correlate them together into a single session.
- Get the raw messages as raised by net-entropy:
So, here’s where we’re at:
- net-entropy has been enhanced to support a protospec file that applies to all ports
- This allows us to do “generic” entropy detection
- Events from net-entropy are being exported to my MARS, from which I can run queries and reports
- Work out if the things that net-entropy is alarming on are actually encrypted, or if the reason for their high entropy is something else (effective compression, for example). If I’m not reliably detecting encryption, then I can either tweak my entropy threshold, or give up the whole idea
- If the technique is really yielding useful results, perhaps write an agent for Sguil so that net-entropy’s alerts appear in the client for easy drill-down onto the session transcripts (there’s an agent available for modsec, so it could be feasible to try this)
- In the far future, how about a mod to the Sguil client that lets you right-click and say “Graph session entropy”? This would extract the relevant session from the full-content capture (just like the Wireshark option does at present), run it through net-entropy in statistics mode, and use gnuplot to visualise the result.
This post is most definitely filed under “Crazy Plans”. Comments on my insanity are welcome.
Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)dataline.co.uk