Visualising Sguil session data with NetFlow

I think this is the first time I’ve explicitly mentioned Sguil, and I’m not going to talk too much about the package itself as many others have already done it for me. Basically, Sguil is a nicely integrated suite of (free) tools that will help you put NSM principles into practice. It has a wonderful investigative interface, and allows easy access to all the alert, session, and full-content data you’ve been capturing via a TAP or SPAN port.

I want to talk about Sguil’s session data capture mechanism. SANCP is used for this purpose, and it’s nicely tied into the alert and full-content data captured by Sguil. It does a jolly good job, too.

SANCP isn’t not the only tool for handling session data, though – Cisco’s NetFlow is another. What I’m going to propose might sound a little odd, but I find it useful – I hope you will, too. Here goes:

My Sguil sensors maintain two stores of session data. One is the standard set as captured by SANCP, the other is a database of NetFlow records that express the same traffic.

“Why?”, I hear you ask. Isn’t this a duplication of data? Don’t you need some fancy Cisco router to export flows? The answers to these are “sort of” and “no”, respectively. Before we get onto how we make this happen, let’s talk about why we want to bother in the first place.

• There is no easy and cheap visualisation of SANCP data. If you’ve got a TAP on a link and you’re feeding it into a Sguil sensor, it’d be great to see a nice graph of how much traffic is passing in each direction. If you can break it down by IP address and port, so much the better. Yes, you can export the SANCP data into something like Excel to graph it, but that’s hardly “easy” or “cheap”. If we can somehow extract NetFlow data from our TAPped link, we can use any number of NetFlow analysis tools to look at what’s going on.
• SANCP isn’t (to my knowledge) timely. If you’ve got a long running session (e.g., a huge download) it won’t get logged by SANCP until it has finished (possibly many hours later). NetFlow devices by contrast can be instructed to prematurely export flows for sessions that haven’t finished yet, giving you a more real-time view of the data.
• SANCP logs one row of data for every session it observes, in terms of source IP/port, destination IP/port, src->dest bytes/packets, dst->src bytes/packets, etc. By storing these “bidirectional” records, you sometimes get entries like this:

  1.2.3.4:1034 -> 10.11.12.13:80
  8.7.6.5:3412 -> 10.11.12.13:80
  10.20.30.40:4431 -> 10.11.12.13:80
  10.11.12.13:80 -> 101.102.103.104:8456

  All of the rows above show sessions to or from a webserver on 10.11.12.13. Look at the last row, though – somehow 10.11.12.13 is listed as the source rather than the destination (you’d have to ask someone familiar with the SANCP codebase why this happens from time to time). What this means is that if I want to write some SQL to show me all of the sessions hitting my webserver, I can’t just say “WHERE dst_ip = INET_ATON(‘10.11.12.13′) and dst_port = 80″ because I would miss out on the last row. I need to alter my SQL to include rows where “src_ip = INET_ATON(‘10.11.12.13′) and src_port = 80″ as well (I guess this is why there are UNION options in Sguil’s query builder!). IMHO SANCP’s “source” and “destination” columns might be better referred to as “peer A” and “peer B”, since there’s no consistency in the way sessions are listed. NetFlow, by contrast, will store unidirectional flows with separate database rows for traffic from a->b and from b->a, thereby avoiding this confusion.

Irrespective of whether you agree with the latter points, I think the visualisation aspect of this is worth it’s weight in gold. Here’s how to do it, and actually make it work in a useful fashion.

Firstly, we need some way of getting NetFlow exports from our TAPped traffic stream. softflowd is your friend here – get it installed on your Sguil box, and run it like this:

/usr/sbin/softflowd -i eth1 -t maxlife=60 -n 127.0.0.1:9996

Let’s look at the parameters in turn:

“-i eth1″ tells softflowd to listen on eth1, which is also Sguil’s monitoring interface in this case.

“-t maxlife=60″ tells softflowd to export a flow record for non-expired flows after sixty seconds. This gives us the “timeliness” that SANCP lacks.

“-n 127.0.0.1:9996″ tells softflowd where to send its NetFlow exports to, in this case a NetFlow collector running locally on the Sguil sensor on port 9996.

So now softflowd is looking at Sguil’s traffic stream, and is converting what it sees to NetFlow exports which it is sending to a collector listening on port 9996. All we need now is a NetFlow collector to receive the exports.

There are many collectors available, but the one I’m going to show here is ManageEngine’s Netflow Analyzer (NFA). I use NFA because the free version will work perfectly in this scenario, and because it has one crucial feature that makes all of this useful (we’ll get to that later on).

So, download and install the free version of NFA on your Sguil sensor, and tell it to collect flows on port 9996. In very short order, NFA should start receiving flows and drawing a nice graph. A graph which will have one obvious and fatal flaw about it:

Whoops. We’re only showing “in” traffic, and it’s actually the sum of the “in” and “out” traffic on our monitored link. When you think about it, what else was softflowd supposed to do? After all, the only traffic it sees is coming “in” to eth1, so that’s all it can report.

However, NFA has a nifty feature we can use to make sense of the data – IP Groups. Set up a new IP Group (call it whatever you like), and add the IP addresses that Sguil/Snort considers to be HOME_NET. NFA will then use these IP addresses to determine which flows should be interpreted as “in” and which as “out”. If we navigate to our IP group in NFA we now get a nice graph that looks like this:

Woohoo! Now we’re talking! Now we can use NFA’s other cool features like drag-to-zoom and breakdowns by IP address and port.

Yes, we’ve bloated up our Sguil sensor in the process. Yes, we’re doubling up on session information. But if you’ve just arrived at a customer site and have plugged your sensor into what is (for you) an unfamiliar network, what better amd quicker way to get an idea of the kind of traffic you’re monitoring?

Comments welcome. I’m sure there’s a gigantic flaw in the plan somewhere!