Archive for the General Security Category

HackFu 2015 – The Badgening

Posted in General Security, Hardware, Packet Challenge on 16 June, 2015 by Alec Waters

Flashback to August 2014. Planning for HackFu 2015 is well underway:

Alec: Hmm, maybe HackFu could use a bit of DefCon-style badge hacking..?
Martyn (MWR): Can we do something cool for £10-£20 per badge? Max 100.
Alec: No problem.

Fast-forward to June 2015, skipping out many months of design, construction, frustration, late nights and burned fingers:

Lots...

Lots…

What you see here is a box containing 102 of these:

badge4

P1070720

Note the important instruction written at the top of the board

More pics from the event can be found on MWR’s Facebook page, and there’s a video report from the event courtesy of SC Magazine.

Here’s the spec of the badge:

  • Ciseco/WirelessThings RFu-328 Arduino-compatible radio transceiver
  • 3.3v TC1015 PowerPOD power regulator
  • 1.8 inch colour TFT display, 128×160 resolution
  • SD card slot
  • 5-way joystick
  • Serial port
  • 3xAA battery box
  • I2C edge connector. More on that later on.
  • Hackable gameplay! More on that later on, too.

It’s a long way from the prototype:

badge1

This is based on an RFu development board and a Nokia 5110 mono LCD (48×84 resolution).

The venue for HackFu 2015 was ex-HM Prison Ashwell, closed in 2011 and now run by The Gaol Events as an urban airsoft site. The venue was chosen to match HackFu’s theme – the premise was that all of MWR (and guests!) were incarcerated, as some of them were suspected of being involved in plotting acts of cyber-nastiness. The name of the game was to identify the guilty and exonerate the innocent, although I’m not sure how many of these people look innocent:

crims

So how did the badges fit into the gameplay? They had two primary functions:

  1. They contained each prisoner’s characteristics – their convictions, addictions, skills, and tattoos. You can see this on the menu screen on the picture above.
  2. They allowed the prison to track the inmates around the venue by means of their on-board SRF radio modules. There were three base stations around the prison which would “ping” each badge in turn. A reply from a badge equated to “loyalty” (because you’ve not absconded) and increased a global loyalty score for all the teams combined. At different point levels, new areas of the venue became accessible and rewards were given (“party” and “free booze” being two of them!)

The base stations looked like this, and comprised a WirelessThings Xino-RF, an Arduino ethernet shield, a clear acrylic enclosure, and some elastic bands:

P1070716

 

P1070717

P1070719

It’s “upside down” in the enclosure so that I could have a wire whip antenna poking out. The ethernet shield is so that the base station can call web services on the game network to register the presence of the badges.

There were dozens of challenges for the inmates to attempt – the reward for successful completion of each wasn’t points (as usual), it was a clue to the identity of one of the guilty parties. Clues were of the form “the mole does not have a conviction for Racketeering”, or “the mole does not have a tattoo of tempura battered prawn” – this meant knowing all of the prisoner characteristics for all of the badges (even those of people on other teams) was critical to success. Now, you could just go around and ask everyone what’s on their badges, or perhaps you could find another way to get the information that doesn’t involve bartering with other teams…

The final menu item on the badge is “Maintenance Mode”. Selecting it shows you some stats and config about the radio module; it also warns you that the radio is inactive – this means it’s no longer responding to polls and is no longer contributing to the overall loyalty score (this is Bad! Remember there’s a party and booze at stake!)

Why is the radio inactive? The SRF module on the RFu board is attached to the “Arduino’s” serial port – if you want to talk to the Arduino, you have to turn off the radio – you can’t do both at the same time. If the inmates connected to the serial port whilst in Maintenance Mode they were presented with a request to enter a PIN – the challenge here is to write a simple brute-forcer that would operate over the serial port.

The reward for getting the PIN is a download of a “badge toolkit”. This consisted of a Python script and a dissector for Wireshark written in Lua. The purpose of the script was to allow the teams to use their issued SRF-Stick USB radios to sniff the radio network and have Wireshark parse the packets. I was using the RadioHead Packet Radio library – the badges would listen for RHReliableDatagrams (sent via the RH_Serial class), send an ack back to the sender, and act on the contents.

The problem here was that the stick could only see traffic to team’s own badges – the SRF supports the concept of logical separation of traffic via a PANID, and each team had their own (think of PANIDs like VLANs on an ethernet switch). If the teams looked at the contents of the Python script they’d find a simple tweak they could make. The script put the SRF into ATZD1 mode, allowing it to hexdump all traffic on its configured PANID. Commented out was a line which put it into ATZD2 mode instead – this hexdumps traffic on all PANIDs.

So now the teams can see all the traffic; if they look into the Lua dissector, they’ll see all of the message types the badge supports:

  • ping – this is the standard “are you there” message sent by the prison
  • IDNUM – causes the badge to output the prisoner ID number over the radio net
  • SKILLx – causes the badge to output skill number X over the radio net
  • CONVIx – same for conviction X
  • TATTOx – same for tattoo X
  • ADDICx – same for addiction X
  • CHPIDx – causes the badge to swap PANIDs (i.e., cause a badge to “change teams”)

So, how does one send one of these messages? Inside the badge toolkit zipfile was a file called .gitignore. Which most people ignored. Because it’s .gitignore. Except it wasn’t – it was another Python script that allowed the user to send a ping to a badge and included all the necessary code to calculate the packet’s CRC. This could then be modified to send any of the other messages, with the results captured in Wireshark. Now the teams can start harvesting prisoner characteristics from all badges, and the answers to the other challenges will make sense.

So what’s the CHPID message for? Why cause a badge to change PANID? Changing PANID will also cause the badge to change the primary colour of the display – each team has their own colour and PANID, and if you change PANID the badge will change colour to match.

It turns out there’s a side benefit to accruing loyalty points, namely cold, hard cash. At the end of every sweep, the prison would work out how many badges were present and which PANIDs they were on. Money was then distributed to each team captain based on the number of badges seen on their team’s PANID. If you can command a badge to change its PANID to yours, that gets more money for your team. Cue “Badge Wars”, where people’s badges were rapidly changing colour as teams vied for control!

So what about the “Loyalty Enhance” menu option? Selecting it merely says “Enhancer not found”, with no other clue as to its purpose. However, one of the items on sale at the HackFu shop was a “nunchuk” (note singular, not plural). Purchasing one of these gave you a Wii Nunchuk, and connecting this to the edge connector made Loyalty Enhance do this:

Tetris, baby!

Tetris, seen here on an earlier version of the board

By levelling-up in Tetris your badge’s response to a prison poll counted for more (a “loyalty multiplier” if you like) – up to sixteen times more if you played it for long enough, potentially allowing a team to reap huge rewards. But there weren’t many Nunchuks to go around, and it takes ages to get to that kind of level in the game. Surely there’s an easier way?

“Easier” probably isn’t the right word, but the Yellow Team (the “Framed Packets”) figured it out. At the peak of their activities, they were netting over £30,000 of in-game currency per hour – you can read about how they did it here. Extra credit also goes to the Green Team (the “Barred Coders”) for downloading the firmware from the badge, removing all the troublesome CHPID and SKILL/etc commands, and reflashing their patched code. Credit goes to all of the teams for their efforts – they all dug deep.

Lessons learned

This was the first time I’d designed a board and made something “proper”. There were a few things I picked up along the way:

  • Make sure you get the Eagle files right before sending them off to the board house! My first run of boards had the four pins at the top of the display 180 degrees out, preventing the use of the SD card slot.
  • Test the boards when you get them back, and before you solder any components on. Some of my boards had dodgy soldermask, much as you see here. It’s quite frustrating to discover this later rather than sooner, but that’s what sometimes happens when you use a cheap Chinese board house (although apart from a few dodgy ones, the boards and the service from DirtyPCBs represented excellent value for money!)
  • Get a Panavise Jr! Assembling over 100 boards without one would have been more torture than it actually was.
  • Don’t do something so monumentally daft as committing yourself to a project that involves making over 7,000 hand-soldered joints (each badge has 70 joints) :)

At the end of this year’s event, I was issued an instruction for HackFu 2016 – “come up with something awesome”. Hmmm, let’s think…

Who are you?

Posted in General Security, NSM on 19 September, 2014 by Alec Waters

Unwanted email is as near a certainty in life as death and taxes. “Selling” spam is a nuisance; phishing emails or messages bearing hostile attachments have the potential to really ruin your day. A lot of the time there are dead giveaways that the message isn’t what it appears to be – the grammar is usually poor, or perhaps the message is claiming to be from a company based in a foreign country that you’re unlikely to be doing business with.

We’ve all chuckled at poorly written messages, but what if the message looks like this?

cdsOr this?

furnituremarketThese are a little more convincing, because they’re copies of actual emails from these two companies – the companies, people and phone numbers all exist and are genuine. The messages are also targeted a little better – they claim to be from a UK company, and are sent to a recipient in the UK, meaning they’re more likely to be read and perhaps acted upon.

The attachments are of course not what they claim to be – the CDS message carries this; the Furniture Market message has this. Neither of these are things you want anywhere near your computer.

Messages like these cause two Problems:

Problem Number One is if a message like this was sent to, for example, your accounts department, would they consider it suspicious or would they open the (hostile) attachment without a second thought? After all, it seems legit – the usual red flags are mostly absent making the message more believable than most. However, if you go to the trouble of opening the attachment you’re running a definite risk of having your computer become part of a botnet, and at that point your real problems are only just beginning.

Problem Number One can be defended against in the usual ways:

  • Educate your users – keep them vigilant. A legitimate looking invoice would have better provenance if you’d actually placed an order with the company it claims to come from. Do you do business with the sender organisation, regardless of how authentic the message looks? Context is important!
  • Keep all your software patched and up to date
  • Run current anti-virus (although as usual there’s no guarantee of success here, judging by the VirusTotal links above)
  • Disable JavaScript in Adobe Reader
  • Don’t log into your computer with admin rights
  • etc!

Problem Number Two affects the sender of the message. Not the actual criminal who sent the message, but the organisation the message claims to be from. Here are some very recent scrapbook snippings from the websites of affected companies:

cds2

tfm2

bhliveimpersonation

broadoak

…and the list goes on. The unfortunate companies above have done absolutely nothing wrong – they’ve not been hacked, they’ve not lost their customer lists, nothing – yet they’ve been forced to put prominent messages like these on their websites, and their customer service staff are suddenly inundated with calls and emails. Under the circumstances, it’s just about the best thing they can do – it shows they care by reassuring their customers that nothing’s been compromised, and hopefully it’ll decrease the load on their service staff. But it’s still not an ideal thing to have to put on the company website.

Is there an effective defence against this kind of impersonation? Email, by its very nature, is insecure (it’s the Simple Mail Transfer Protocol, after all) – it’s trivial to make an email appear to be from anyone you like. Copying an organisation’s email template is just the icing on the cake.

You could employ one or more of the following techniques:

All of these are mechanisms which are designed to detect email spoofing, as in the above examples. A shortcoming of this approach is that it is the reciever’s responsibility to do the checking. If you’ve set up SPF, for example, it’s all for naught if the receiver doesn’t do the SPF check. Think about your own email arrangements – does your receiving mailserver perform SPF or DKIM checks?

Problem Number Two can therefore affect just about anybody, regardless of how careful you are in setting up anti-spoofing measures. The best defence against Problem Number One probably lies with the acutal human being opening the email – take the advice of these guys and ask the question, “Who Are You?”


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

I love it when a plan comes together

Posted in General Security, NSM on 9 January, 2014 by Alec Waters

As defenders, we have many reasons to do our jobs. We want to comply with regulations, protect our employers (and protect our pay cheques!), and just maybe we enjoy the challenge despite the certain knowledge that someday an exploit with our name on it is going to smack us between the eyes.

The “why we do it” is therefore straightforward; what about the “how”? How do we defend? I don’t mean from a technical perspective – at a generic level, what are we trying to achieve?

There are certainly many answers to that question, but I quite like the idea that there are three mutually supporting objectives for defenders. The idea I’m presenting below is probably a little monitoring-centric, but then so am I!

At the top tier, there’s the “Defenders’ Utopia”, namely Prevent-It:

planaIt doesn’t matter what “It” is – if we can prevent all badness, we’ve won! Let’s patch stuff, pentest stuff, educate our users, harden our deployments, use SDL principles and products X, Y and Z to prevent all the badness, guaranteeing us “corporate security hero” status.

Prevent-It is therefore our “Plan A”. Sadly, it’s not enough because Prevention Eventually Fails – we need a Plan B.

preventioneventuallyfailsIf we can’t Prevent-It, Plan B should be to “Detect-It” in a timely fashion:

planabAnd no, third-party breach notification doesn’t count as “timely”! As well as responding to obvious indicators such as AV or IDS hits, we need to be proactive in detection by hunting through all of the instrumentation at our disposal looking for indicators like uncommon or never-before-seen events, unusual volumes of network traffic or event types, Bob logging in from Antigua when you know he’s in Scotland, etc. A solid ability to Detect-It will help you invoke the Intruder’s Dilemma.

“Detect-It” is bigger than “Prevent-It” in the diagram because prevention is hard, and we are more likely to be able to detect things than we are to prevent them if we try hard enough. Attackers have more tactics at their disposal than defenders have preventative measures so you need to be as thorough and as business-tailored as you can be in your monitoring. Know your infrastructure in as much detail as possible in terms of the platform (e.g., MVC app on IIS8 behind a Cisco ASA) and make certain it’s behaving as expected. For a self-contained security team, this latter part might be hard – I’d possibly venture the opinion that an app’s functional spec might be a useful thing for the security team to have. Should app X be sending emails? Doing FTP transfers? Talking to SkyDrive? If the security team don’t understand these details, they may miss things.

Eventually, you’ll likely run up against an attacker who slips past detection and conjures up the defenders’ nightmare of third-party breach notification – you’ve been compromised, you didn’t notice it happen, and now you’re front page news. Plan B has failed – your final recourse is Plan C:

planabc

If we can’t Prevent-It, and we didn’t Detect-It then we have to maintain the ability to Investigate-It. This bit’s biggest because it represents your gathering of logs, network traffic data and other indicators – your monumental programme of “hay collection” (the Detect-It part can be thought of as a “hay removal” process whose output is needles, if you follow my metaphor).

Even boring, routine logs may be worth their weight in gold when Investigating-It – collect as much as is feasible and legal. Even if we don’t have the resource to actually look at all the collected logs as a matter of course, at least we’ve got a huge pool of evidence to trawl through as part of the Incident Response process.

We can also use Investigate-It for retrospective analysis. If a new set of IOCs (Indicators of Compromise) comes to light, we can check them against what we’ve collected so far. Investigate-It also supports your corporate forensic readiness plan – knowing what information you need as part of IR, where to get it, and what you can get quickly without outside help is key.

Stairway to Heaven

If our org has sufficient resources we can move back up the stack, leveraging each tier to improve the one above. If we have an excellent capability to Investigate-It, it means we can improve our ability to Detect-It by hunting through the collected logs in different ways, producing more insightful reports/alerts/dashboards that pull together many disparate log sources. Lessons learned as part of Investigate-It can be incorporated into Detect-It – make sure that the indicators that were missed this time won’t be missed next time.

Moving up again, improving our ability to Detect-It can improve our ability to Prevent-It because we may discover things we don’t like that we can put a stop to before they become a problem (e.g. people emailing confidential docs to their personal email account so they can “work on them from home”, people logging in as local admin, people running apps you don’t like, why is there an active Bluetooth serial port on the CEO’s MacBook, etc) or we may even discover Existing Badness that we can zap.

So now we’re back at the top of the diagram, hopefully in better stead than before we started. Like I say, it’s a monitoring-centric way of looking at things, but if you’re in the game hopefully it’s an interesting perspective!


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

Hacker Countermeasures, 1984 style

Posted in General Security, Retro on 30 April, 2012 by Alec Waters

Some number of years ago, I was lucky enough to get a Sinclair ZX-81 for Christmas. There was much wonderment and joy to be had amid the frustration of RAM-pack wobble, the agonising waits for software to load from tape, and the never-ending search for a replacement keyboard that wasn’t as bad (or worse) than the original.

The best thing about the computers of olde was the built-in interpreter, usually for BASIC – here was an item of consumer electronics that wouldn’t do a single thing unless you told it to, an unthinkable concept to the marketeers of today. Putting in a tape and typing LOAD “” was the easy way out of this predicament; however, the real solution to the inert nature of your newly purchased box of future was to open the manual and learn how to code.

So learn we did. One day, my father proudly showed me a program he’d written – a version of the card game “snap”, with graphics and everything. After whiling away a good part of the afternoon playing, I looked over the source code. Showing an early leaning towards white-box pentesting, it didn’t take long to find a simple flaw. By simply keeping your finger pressed on your “snap” key (regardless of the two cards on the top of the deck) you could beat even the quickest opponent when a true “snap” finally came around. If both players were aware of this exploit (or “expliot” as I’d almost certainly have spelled it at the time) you had to make sure that you were player 2 since the subroutine that checked which key was pressed during a “snap” condition checked for player 2’s “snap” key before player 1’s.

Easily exploitable vulnerabilities? Some things never change, huh? In terms of Incident Detection, prime Indicators of Compromise included my little sister complaining to our parents that Daddy’s game was no fun because Alec always won.

To restore the game back to a test of speedy reactions my father rolled out some countermeasures in the form of a patch. The next time we played, my tactic resulted in the computer labelling me a cheat and docking me five cards every time I pressed my snap key when it wasn’t snap. To further add injury to insult, the losing player was crushed by a one-ton weight falling from the ceiling. I’m sure you can imagine what a terrifying visual experience this must have been, especially if you remember the graphics capabilities of the ZX-81…

To sample the full glory of this dance of measure and countermeasure, here’s the actual source code as submitted to ZX Computing magazine nearly thirty years ago. Lines 570 and 580 show the horrifying corpses of the losing players, squashed flat by Newtonian Justice From Above. Enjoy :)

Newtonian Justice from Above

Lines 570 and 580, Newtonian Justice from Above


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

Quick Book Review – The Alexandria Project, by Andrew Updegrove

Posted in General Security on 19 April, 2012 by Alec Waters

THANK YOU FOR YOUR CONTRIBUTION TO THE ALEXANDRIA PROJECT

In a slight departure from my usual reading material of John le Carré and non-fiction technical tomes, I recently read The Alexandria Project by Andrew Updegrove; it turned out to be a nice mix of the two.

Without wishing to spill too many beans, it’s a fun read featuring mystery attackers with mystery motives, three-letter-agencies butting heads whilst manipulating people down their chosen path, military coups and crazy politicians with their finger on the Big Red Button.

The plot is spookily close to reality, especially in the Big Red Button department – I was reading the story on the actual dates featured in the book, at which time events were playing out on the world stage much as they were in my Kindle (plot spoiler, courtesy of BBC News, here). Coincidence? Or does Mr Updegrove have a crystal ball?

For the grand total of £1.95, you can’t really go wrong. Swing by Amazon and pick up a copy!


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

Next Generation Naughtiness at the Dead Beef Cafe

Posted in General Security, IPv6, Networking, NSM on 23 December, 2010 by Alec Waters

The IPocalypse is nearly upon us. Amongst the FUD, the four horsemen are revving up their steeds, each bearing 32 bits of the IPv6 Global Multicast Address of Armageddon, ff0e::dead:beef:666.

Making sure that the four horsemen don’t bust into our stables undetected is something of a challenge at the moment; IPv6 can represent a definite network monitoring blind spot or, at worst, an unpoliced path right into the heart of your network. Consider the following:

Routers and Firewalls

Although a router may be capable of routing IPv6, are all the features you use on the router IPv6-enabled? Is the firewall process inspecting IPv6 traffic? If it is, is it as feature-rich as the IPv4 equivalent (e.g., does it support application-layer inspection for protocols like FTP, or HTTP protocol compliance checking?)

End hosts

If you IPv6-enable your infrastructure, you may be inadvertently assigning internal hosts global IPv6 addresses (2001::) via stateless address autoconfiguration. If this happens (deliberately or accidentally), are the hosts reachable from the Internet directly? There’s no safety blanket of NAT for internal hosts like there is in IPv4, and if your network and/or host firewalls aren’t configured for IPv6 you could be wide open.

IDS/IPS

Do your IDS/IPS boxes support IPv6? Snort’s had IPv6 support since (I think) v2.8; the Cisco IPS products are also IPv6-aware, as I’m sure are many others.

Session tracking tools

SANCP has no IPv6 support; Argus does, as does cxtracker. netflow can also be configured to export IPv6 flows using v9 flow exports.

Reporting tools

Even if all of your all-seeing-eyes support IPv6, they’re of little use if your reporting tools don’t. Can your netflow analyser handle IPv6 exports? What about your IDS reporting tools – are they showing you alerts on IPv6 traffic? What about your expensive SIEM box?

The IPv6 Internet is just as rotten as the IPv4 one

We’ve seen some quite prolific IPv6 port scanning just as described here, complete with scans of addresses like 2001:x:x:x::c0:ffee and 2001:x:x:x::dead:beef:cafe. The same scanning host also targeted UDP/53 trying to resolve ‘localhost’, with the same source port (6689) being used for both TCP and UDP scans. I have no idea if this is reconnaissance or part of some kind of research project, but there were nearly 13000 attempts from this one host in the space of about three seconds.

Due to the current lack of visibility into IPv6, it can also make a great bearer of covert channels for an attacker or pentester. Even if you’re not running IPv6 at all, an attacker who gains a foothold within your network could easily set up a low-observable IPv6-over-IPv4 tunnel using one of the many IPv6 transition mechanisms available, such as 6in4 (uses IPv4 protocol 41) or Teredo (encapsulates IPv6 in UDP, and can increase the host’s attack surface by assigning globally routable IPv6 addresses to hosts behind NAT devices, which are otherwise mostly unreachable from the Internet).

The IPocalypse is coming…

…that’s for certain; we just have to make sure we’re ready for it. Even if you’re not using IPv6 right now, you probably will be to some degree a little way down the road. Now’s the time to check the capability of your monitoring infrastructure, and to conduct a traffic audit looking for tunneled IPv6 traffic. Who knows what you might find!


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

The Cisco Kid and the Great Packet Roundup, part two – session data

Posted in Cisco, General Security, NSM on 26 October, 2010 by Alec Waters

In part one, I covered how to use Cisco routers and firewalls to perform full packet capture. This exciting installment will cover how to get network session data out of these devices.

Network session data can be likened to a real-world itemised telephone bill. It tells you who “called” who, at what times, for how long, and how much was said (but not what was said). It’s an excellent lightweight way to see what’s going on armed only with a command prompt.

There are several ways to extract such information from Cisco kit; we’ll look at each in turn, following Part One’s support/troubleshooting/IR scenario of accessing remote devices where you’re not able to make topological changes or install any extra software or hardware.

Netflow

The richest source of session information on Cisco devices is Netflow (I’ll leave it to Cisco to explain how to turn it on). If you’re able to set up a Netflow collector/analyser (like this one (free for two-interface routers), or many others) you can drill down into your session info as far as you like. If you haven’t got an analyser or you can’t install one in time of need, it’s still worth switching on Netflow because you can interrogate the flow cache from the command line.

The command is “show ip cache flow”, and the output is split into two parts. The first shows some statistical information about the flows that the router has observed:

router#sh ip cache flow
IP packet size distribution (3279685 total packets):
 1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
 .000 .184 .182 .052 .072 .107 .004 .005 .000 .000 .000 .000 .000 .000 .000

 512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
 .000 .000 .001 .020 .365 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
 57 active, 4039 inactive, 418030 added
 10157020 ager polls, 0 flow alloc failures
 Active flows timeout in 1 minutes
 Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 34056 bytes
 57 active, 967 inactive, 418030 added, 418030 added to flow
 0 alloc failures, 0 force free
 1 chunk, 1 chunk added
 last clearing of statistics never
Protocol     Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------     Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-WWW       6563      0.0       186  1319      1.2       4.7       1.4
TCP-other    16163      0.0         1    47      0.0       0.0      15.4
UDP-DNS         12      0.0         1    67      0.0       0.0      15.6
UDP-NTP       1010      0.0         1    76      0.0       0.0      15.0
UDP-Frag         2      0.0         6   710      0.0       0.2      15.3
UDP-other   316602      0.3         2   156      0.8       0.6      15.4
ICMP         31165      0.0         6    63      0.2      53.4       2.2
IP-other     46438      0.0        21   125      1.0      58.0       2.1
Total:      417955      0.4         7   574      3.3      11.0      12.7

In absence of a graphical Netflow analyser, the Packets/Sec counter is a good barometer of what’s “using up all the bandwidth”. To clear the stats so that you can establish a baseline, you can use the command “clear ip flow stats”.

After the stats comes a listing of all the flows currently being tracked by the router:

SrcIf     SrcIPaddress    DstIf     DstIPaddress    Pr SrcP DstP  Pkts
Fa4       xxx.xxx.xxx.xxx Local     yyy.yyy.yyy.yyy 32 3FAF 037C    16
Tu100     10.7.1.250      BV3       10.4.1.3        06 0051 C07A   663
Tu100     10.7.1.250      BV3       10.4.1.3        06 0050 C0AC   120
BV3       10.4.1.3        Tu100     10.7.1.250      06 C0AC 0050   116
Tu100     192.168.88.20   Local     172.16.7.10     01 0000 0800     5
BV3       10.4.1.3        Fa4       zzz.zzz.zzz.zzz 06 C0A2 0050   429
BV3       10.4.1.3        Tu100     10.7.1.250      06 C07A 0051   366
Fa4       bbb.bbb.bbb.bbb BV3       yyy.yyy.yyy.yyy 06 0050 C0A0     1
BV3       10.4.1.3        Fa4       ddd.ddd.ddd.ddd 06 C07E 0050     1
Tu100     192.168.88.56   Local     172.16.7.10     06 8081 0016     7
Fa4       zzz.zzz.zzz.zzz BV3       yyy.yyy.yyy.yyy 06 0050 C0A2   763
Tu100     192.168.88.28   Local     172.16.7.10     11 04AC 00A1     1
Tu100     192.168.88.28   Local     172.16.7.10     11 04A6 00A1     1
Fa4       aaa.aaa.aaa.aaa Local     yyy.yyy.yyy.yyy 32 275F BD8A     5
Fa4       ccc.ccc.ccc.ccc Local     yyy.yyy.yyy.yyy 32 97F1 E9BE     5
Tu100     10.7.1.242      Local     172.16.7.10     01 0000 0000     3
Fa4       ddd.ddd.ddd.ddd BV3       yyy.yyy.yyy.yyy 06 0050 C07E     1

The tempting simplicity of the table above hides a plethora of gotchas for the unwary:

  • The Pr (IP protocol number),  SrcP (source port) and DstP columns are in hex, but we can all do the conversion in our heads, right? ;)
  • Netflow is a unidirectional technology. That means that if hosts A and B are talking to one another via a single TCP connection, two flows will be logged – one for A->B and one for B->A. For example, these two rows in the table above are talking about the same TCP session (the four-tuple of addresses and ports is the same for both rows):
Tu100     10.7.1.250      BV3       10.4.1.3        06 0051 C07A   663
BV3       10.4.1.3        Tu100     10.7.1.250      06 C07A 0051   366
  • Unless you configure it otherwise, Netflow is an ingress technology. This means that flows are accounted for as they enter the router, not as they leave. You can determine what happens on the egress side of things because when a flow is accounted for the output interface is determined by a FIB lookup and placed in the DstIf column; in this way, you can track a flow’s path through the router. I mention this explicitly because…
  • Netflow does not sit well with NAT. Take a look at these two rows, which represent an HTTP download (port 0x0050 is 80 in decimal) requested of non-local server zzz.zzz.zzz.zzz by client 10.4.1.3:
BV3       10.4.1.3        Fa4       zzz.zzz.zzz.zzz 06 C0A2 0050   429
Fa4       zzz.zzz.zzz.zzz BV3       yyy.yyy.yyy.yyy 06 0050 C0A2   763

So what’s yyy.yyy.yyy.yyy, then? It’s the NAT inside global address representing 10.4.1.3. As Netflow is unidirectional and is recorded as it enters an interface, the returning traffic from zzz.zzz.zzz.zzz will have the post-NAT yyy.yyy.yyy.yyy as its destination address, and will be recorded as such.

Provided that you keep that lot in mind, the flow cache is a powerful tool to explore the traffic your router is handling.

NAT translations

A typical border router may well perform NAT/PAT tasks. If so, you can use the NAT database as a source of session information. On a router, the command is “show ip nat translations [verbose]”; on a PIX/ASA, it’s “show xlate [debug]”:

router#show ip nat translations
Pro Inside global         Inside local   Outside local    Outside global
tcp yyy.yyy.yyy.yyy:49314 10.4.1.3:49314 94.42.37.14:80   94.42.37.14:80
tcp yyy.yyy.yyy.yyy:49316 10.4.1.3:49316 92.123.68.49:80  92.123.68.49:80

If you’ve got a worm on your network that’s desperately trying to spread, chances are you’ll see a ton of NAT translations (which could overwhelm a small router). Rather than paging through thousands of lines of output, you can just ask the device for some NAT statistics. On a router, it’s “show ip nat statistics”; on a PIX/ASA, it’s “show xlate count”.

Keeping tabs on the number of active NAT translations is a worthwhile thing to do. I wrote a story for Security Monkey’s blog a while back which tells the tale of a worm exhausting a router’s memory with NAT translations; you can even graph the number of translations to look for anomalies over time.

Firewall sessions

Another way of extracting session information is to ask the router or PIX about the sessions it is currently tracking for firewall purposes. On a router it’s “show ip inspect sessions [detail]”; on the PIX/ASA, it’s “show conn [detail]”.

router#show ip inspect sessions detail
Established Sessions
 Session 842064A4 (10.4.1.3:49446)=>(92.123.68.81:80) http SIS_OPEN
  Created 00:00:59, Last heard 00:00:58
  Bytes sent (initiator:responder) [440:4269]
  In  SID 92.123.68.81[80:80]=>y.y.y.y[49446:49446] on ACL outside-fw (6 matches)
 Session 84206FC4 (10.4.1.3:49443)=>(92.123.68.81:80) http SIS_OPEN
  Created 00:00:59, Last heard 00:00:59
  Bytes sent (initiator:responder) [440:2121]
  In  SID 92.123.68.81[80:80]=>y.y.y.y[49443:49443] on ACL outside-fw (4 matches)
 Session 8420728C (10.4.1.3:49436)=>(92.123.68.81:80) http SIS_OPEN
  Created 00:01:01, Last heard 00:00:50
  Bytes sent (initiator:responder) [1343:48649]
  In  SID 92.123.68.81[80:80]=>y.y.y.y[49436:49436] on ACL outside-fw (44 matches)

This has the advantage of not being complicated by NAT, but still showing useful bytecounts and session durations.

Last resorts

If none of the above can help you out, there are a couple of last resort options open to you. The first of these is the “ip accounting” interface configuration command on IOS routers. To quote Cisco:

The ip accounting command records the number of bytes (IP header and data) and packets switched through the system on a source and destination IP address basis. Only transit IP traffic is measured and only on an outbound basis; traffic generated by the router access server or terminating in this device is not included in the accounting statistics. Traffic coming from a remote site and transiting through a router is also recorded.

Also note that this command will likely have a performance impact on the router. You may end up causing more problems than you solve by using this! The output of “show ip accounting” will look something like this:

router# show ip accounting
 Source          Destination            Packets      Bytes
 172.16.19.40    192.168.67.20          7            306
 172.16.13.55    192.168.67.20          67           2749
 172.16.2.50     192.168.33.51          17           1111
 172.16.2.50     172.31.2.1             5            319
 172.16.2.50     172.31.1.2             463          30991
 172.16.19.40    172.16.2.1             4            262

If “ip accounting” was a last resort, “debug ip packet” is what you’d use as an even lasterer resort, so much so that I leave it as an exercise for the reader to find out all about it. Don’t blame me when your router chokes to the extent that you can’t even enter “undebug all”…!


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

Follow

Get every new post delivered to your Inbox.

Join 35 other followers