Archive for May, 2009

Collection is King, part one

Posted in NSM on 29 May, 2009 by Alec Waters

So we’re now at the point where a SPAN port is spewing mirrored traffic at our sensor. What do we do with it?

Well, we’re going to apply a methodology to it (I’m deliberately avoiding mentioning any specific products here – security is a process, not a product). The methodology in question is called “Network Security Monitoring”, or just NSM for short.

NSM wasn’t my idea, nor do I claim to be any kind of authority on the subject. I do, however, practice it every day, and I can tell you that it works. One of the foremost NSM authorities is Richard Bejtlich, who has an excellent blog and has written some fine books. I would strongly recommend reading them.

Richard defines NSM as the “collection, analysis, and escalation of indications and warnings to detect and respond to intrusions.” It sets out to deliver to you, the analyst, the following four categories of network-based information:

  1. Alert data. These are the alerts raised by a detection device, such as an IDS. Alert data by itself doesn’t really go far enough though – it isn’t possible to validate a security incident based solely on an IDS alert. We need more.
  2. Session data. This is the networking equivalent of an itemised telephone bill. It shows you who spoke to whom, at what times, and for how long (expressed in terms of IP addresses, protocols, ports, byte volumes, etc). It won’t tell you what was said, but it will at least tell you that a conversation took place. By mining session data, you can find out which hosts are busy sending and receiving lots of traffic, and also what kind of traffic that was based upon the protocol and port (FTP, HTTP, etc).
  3. Statistical data. Given our collection of session data, we can build baselines of “normal” usage and report on anything that deviates from this baseline.
  4. Full-content data. These are the actual packets themselves, and their content. The full-content data can be mined to extract things like URLs requested, emails sent/received, IM chats, FTP sessions. etc.

Given this lot, you can go a great deal further than you could if you only had an IDS. A trigger for an investigation could be:

  • An IDS alert. If you’re practicing NSM, you can:
    • See the alert in the context of the session that carried it. For example, if an IDS alert fires because it has spotted an exploit URL being requested of your web server, an NSM practitioner will be able to see the web server’s response to the potentially harmful request. If the web server responded with a 404 “not found” error, for example, then the attack was unsuccessful.
    • Investigate the attacker’s precursor activities by leveraging the session data. Has the attacker been here before? Has he conducted reconnaissance? Has he tried many different attacks? Or perhaps the same attack over and over?
    • See if the victim machine exhibited any unexpected behaviour during the time after the IDS alert was raised. Post compromise, an attacker will often download tools to consolidate their position – traces of this will be present in our session and full-content data.
  • A statistical alarm. Has our mailserver started sending more mail than usual? Are there an unusual number of connections to or from our public web server?
  • A report based on session data. For example, a public facing web server should be busy serving web pages to the public. If we can filter these out of our session data, we are left with things like:
    • FTP sessions where new content is uploaded
    • The server’s own hits on auto-update services, like Windows Update.
    • Anything else! This is where you’ll see traces of an attacker downloading their tools, or malware communicating with a command and control server.

An analyst practicing NSM is clearly well equipped to detect and investigate the Badness. Next time, I’ll show a practical application of the NSM methodology.

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

Quis custodiet ipsos facis?

Posted in Why watch the wire? on 15 May, 2009 by Alec Waters

According to the Magic Internet, that means “who watches the packets?” I bet my latin teacher would have a few comments on that translation…

Anyway, we’ve decided that we’re interested in the network traffic crossing the point between our switch and our small-office router. We have made this decision because we do not wish to trust our security solely to preventative measures which will inevitably let us down. We want to try to spot the Badness getting in and out, and this is the way to do it.

The practicalities of this are twofold – we need a sensor with which to collect the traffic, and we need some way of directing our traffic to it so that it can be examined.

For the most part, the sensor is some kind of computer with at least one network interface of a type appropriate to your infrastructure (e.g., perhaps it’s an ordinary copper ethernet interface, or a fibre-optic interface, or even a wifi one). The sensor will not interfere with the collected traffic in any way, as it will usually be fed a copy of the traffic you want to inspect. This prevents the mere presence or absence of the sensor from breaking any of your stuff, and also has the happy side effect of preventing the Baddies from even knowing you’ve deployed it.

There are a number of ways we can get a copy of our network traffic to our sensor:

  • If our switch is capable of it, we might be able to set up a SPAN port. We can configure a SPAN port that will output a copy of everything sent and received on the switchport that connects to our router. By plugging our sensor into this SPAN port, it will be able to see all of our Internet traffic.
  • If our sensor has two or more monitoring interfaces, we can use a network tap. A tap will physically sit on the path between our switch and our router, and will “syphon off” a copy of the network traffic for our sensor to look at. Although the tap is inline, it doesn’t alter the observed traffic and it won’t permit the sensor to inject any traffic of its own. It’s the purest form of capture, and can be dropped in without altering the configuration of any other devices. Tap manufacturer Netoptics has a comparison between tap and SPAN here.
  • A final option might be something like Cisco’s Raw IP Traffic Export (RITE). This is something of a last choice, though; generally speaking tap and SPAN are superior options. However, for some topologies this may be the only option – you may not be physically able to use tap or SPAN if you want to capture the traffic crossing a virtual IP interface on a layer three switch, for example.

For the sake of simplicity, let’s say our sensor has just one monitoring interface and we’re feeding it from a SPAN port on our office’s switch. We are now ready to spot the Badness! However, our sensor needs to be able to do something with the traffic we’re spewing at it. We need to be able to receive it, store it, inspect it, mine it.

Hold tight – things are about to get interesting!

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

Baby steps, part two

Posted in Why watch the wire? on 12 May, 2009 by Alec Waters

Baby Steps ended by asking how we can use the network itself “to extract information from strategic points that will tell us what is going on”. To start to explain, let’s have an example of a “strategic point”.

When I was at junior school (at about ten years of age), we were given a practical maths assignment which meant us leaving the school grounds. This was a big deal, because our school was quite literally a fortress – it had fifteen foot high brick and flint walls topped off with another six feet of chickenwire, only interrupted by heavy iron gates. Don’t get me started about the guard towers and searchlights. As such, it was a rare treat to get outside during school hours without resorting to re-enacting the Great Escape.

Our maths task for the day was to conduct a traffic survey. We were to sit with our backs to the outside of the fortress wall, count the passing cars, and group them by colour. After twenty minutes or so of this, we were shepherded inside by the guards teachers to prepare a report on which colours of car were most popular based upon our observations.

So, what has this got to do with strategic points and network security? It’s all to do with visibility, in terms of the quantity and type of traffic you want to observe.

Our school was on a main road, so we had a fair sample set to show for our twenty minutes’ worth of observations. If our school was instead next to a motorway, our sample set would have been huge (and possibly beyond the ability of a ten year old to accurately collect with a pencil and paper). If the school was in a cul-de-sac, we’d have hardly seen anything at all. So, if we want a large sample set the motorway is probably the best choice, albeit with the risk that we won’t be able to record everything we see.

On the other hand, the type of traffic may be more important to us than the quantity. If we want to observe trucks hauling huge loads, the motorway is the best place to look. If we’re interested in local bus services, our school’s main road might fit the bill. If we want to know about people’s milk deliveries, then the cul-de-sac would be a more fruitful place to conduct your sampling.

Coming back to network security, we have already established that we want to look at what is going on on the network; the next step is to pick one or more strategic points where we can do the looking. This will depend entirely on what your own infrastructure looks like, and what it is you’re hoping to see.

For the purposes of our baby steps, we’ll take the simple example of a small office. It has a dozen or so workstations connected via a switch, a single server (also on the switch), and a router that connects the whole lot to the Internet. We have already established that there is Badness on the Internet, and that we should watch for it. Given this objective, a suitable strategic point to monitor would be the point where the switch plugs into the router. All traffic either to or from the Interet will cross this point – if Badness is getting in or out, this is the route it has to take and with a bit of luck, we’ll catch it in the act.

(I should emphasise at this point that monitoring our little office’s border with the Internet will not tell us anything about the conversations that the workstations and the server have between themselves – we’ll only see traffic that involves the Internet. If you’re interested in “local” traffic, you’ll need to conduct your monitoring elsewhere on your network, possibly at more than one location).

Having picked the place to do our monitoring, we now need to decide how we’re actually going to do this. Clearly, a ten year old with a pencil isn’t going to cut it. Perhaps there’s some technical marvel that can help us out? Stay tuned!

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

Baby steps

Posted in Why watch the wire? on 7 May, 2009 by Alec Waters

Referring back to my initial post, I said:

“I believe that the network itself (remembering my specific interpretation of the word “network”) is a great and often untapped source of security information”

Why do I think this, and what does it all boil down to at the end of the day?

Well, the “why” is based on the following rash assumptions:

  • Rash assumption #1 – “Badness” exists on the Internet (by “Badness” I’m talking about threats including viruses, trojans, bots and other such malware)
  • Rash assumption #2 – If the Badness reaches your computer, the most likely (but far from only) means by which it got there is via the network. It could have come in via an email, or a drive-by download, or via a peer-to-peer filesharing application – at this point in the discussion, the precise infection vector is unimportant. I just want to stress the fact that having a network connection provides a potential way in for Badness
  • Rash assumption #3 – Once present on your computer, the Badness will use the network in some way, either to receive a list of Evil Tasks from the Baddies, or to send the Baddies your banking credentials, or to perpetrate some other naughtiness.

Hopefully none of these assumptions can be reasonably refuted!

The common theme amongst these assumptions is the network itself. By connecting yourself to the Internet, you’re tapping into a vast pool of Badness, all of which wants to make its way onto your computer. Once there, the byproducts of the Badness will leave your computer via the same route – the network. It therefore follows that it ought to be productive to monitor the traffic crossing the network, and look out for signs of Badness.

Still with me?

Given the existence of the Badness, and the potential route it has to get onto your computer, what can we do about it? We clearly need some defences here. A three-layered approach might be to strive to achieve the following:

  1. Stop the Badness in its tracks. This is the ideal situation, and can be addressed with preventative measures such as:
    • Network- and host-based firewalls
    • Network Intrusion Prevention Sensors (IPS, actually a specialised class of firewall)
    • Email firewalls that scan messages for Badness well before the message actually reaches your computer
    • Network-based URL filters or proxies that intercept your web requests and stop you from fetching anything that is known to be Bad
    • Anti-virus software
    • Keeping the software on your computer patched in a timely fashion

    However, sooner or later, one or more of these will let you down and the Badness will get to your computer. So, at some point or another, we’ll need to fall back to the second line, which is to:

  2. Detect the Badness in a timely fashion. If we can’t stop the Badness, at least give us a chance to detect the Badness so that we can act before something really catastrophic goes down. We have to watch the network like a hawk, not just for the alerts raised by the preventative measures listed above, but for much more subtle things. We’re interested in anything anomalous, like:
    • Traffic at strange times of the day
    • Traffic on strange ports
    • Traffic to or from strange destinations
    • Unexpected traffic volumes or per-flow packet counts

    Your preventative measures are unlikely to report on indicators like this. Given the vagueness of what actually defines “anomalous” in your own context, it is pretty much a given that it is a person (not a machine) that makes this determination.

    Finally, we have to cover the case where Badness has got in undetected, and has wrought whatever carnage and mayhem its creator had in mind. In short, we need to:

  3. Gather enough forensic information to work out exactly what has happened, after you’ve found out about your own security breach in either the popular press or the legal papers that have just been served upon you. Take the relatively benign example where some rogue anti-virus software is popping up on your computer, telling you there are zillions of problems, and that it can fix them all (for a price!). We need to be able to determine:
    • Where the Badness came from
    • If anything else got downloaded along with the rogue anti-virus software
    • If there have been any unexpected network connections from the suspect machine

    Now, we can’t readily ask these questions of the infected machine. If the infection has been thorough enough, your computer will lie to you. It will tell you everything is OK, that there are no strange processes running, and that there is definitely no unusual network activity.

The only way to provide for all of this is to make use of the network itself. We need to be able to extract information from strategic points that will tell us what is going on now and also what happened at half past four last Tuesday afternoon. We need to be able to mine this information for usable snippets of intelligence, expressed in terms of low-level things like IP addresses and ports all the way through to higher-level things like URLs visited and emails sent and received. With this information at our fingertips, it becomes feasible to attempt to “Detect the Badness in a timely fashion” and to “Gather enough forensic information to work out exactly what has happened”.

So, how do we do all of this? That’s a topic for next time – stay tuned.

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

Here we go!

Posted in Misc on 6 May, 2009 by Alec Waters

Well, here it is – my first foray into web2.0! Apparently, blogs are a good way to document and collate your thoughts, so in the spirit of scientific experimentation I’m going to test the theory by documenting and collating my thoughts on “why have I started a blog”?

To understand the “why”, it might help to understand the “who”. By which I mean me, rather than The Who. They rock; I type.

Anyway, in no particular order…

  • My name is Alec Waters
  • I work for an IT company in Brighton in the UK
  • My job title is Infrastructure Manager, which means I’m responsible for pretty much all aspects of IT other than an application written by our development teams
  • Part of my job is the provision and management of network infrastructure. “Network” is a term that sometimes has different meanings to different people – when some of my colleagues talk about their “networks”, they’re talking about their domain controllers, active directories, and group policy objects. Not me – when I say “network”, I’m talking about the cabling plant, the switches, routers, firewalls, intrusion detection sensors and other devices that enable the server admins to even have their shiny domain controllers.
  • I have a current Cisco CCNP certification
  • The lion’s share of my day to day work is information security.

And so we come to the point of this blog. Information security is something that I’m greatly enthusiastic about, and I think I’ve got to the point where I’ve amassed enough experience to actually have something hopefully useful to say on the subject.

Information security spans a great many disciplines; my specialist subjects are network security monitoring and network forensics. I believe that the network itself (remembering my specific interpretation of the word “network”) is a great and often untapped source of security information that should be complementary to other, more conventional, defensive measures like anti-virus. I am in part talking about network specific devices like intrusion detection sensors (IDS), but I’m also talking about going a lot further than this. IDS boxes are extremely valuable and useful devices, but they just don’t go far enough on their own. You can do a lot more, and I hope to be able to get this point across in subsequent posts.

I don’t claim to be a luminary. I don’t claim to be the only person on planet Earth who has ever done any of this. I strongly doubt that I’ll ever come up with something original and groundbreaking. All I want to do is get over the point that listening to your network is a truly worthwhile thing to be doing, and will give you operational visibility that you absolutely cannot have in such fidelity and with such integrity by any other means.

With that said, let battle commence!

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)