Archive for October, 2010

The Cisco Kid and the Great Packet Roundup, part two – session data

Posted in Cisco, General Security, NSM on 26 October, 2010 by Alec Waters

In part one, I covered how to use Cisco routers and firewalls to perform full packet capture. This exciting installment will cover how to get network session data out of these devices.

Network session data can be likened to a real-world itemised telephone bill. It tells you who “called” who, at what times, for how long, and how much was said (but not what was said). It’s an excellent lightweight way to see what’s going on armed only with a command prompt.

There are several ways to extract such information from Cisco kit; we’ll look at each in turn, following Part One’s support/troubleshooting/IR scenario of accessing remote devices where you’re not able to make topological changes or install any extra software or hardware.

Netflow

The richest source of session information on Cisco devices is Netflow (I’ll leave it to Cisco to explain how to turn it on). If you’re able to set up a Netflow collector/analyser (like this one (free for two-interface routers), or many others) you can drill down into your session info as far as you like. If you haven’t got an analyser or you can’t install one in time of need, it’s still worth switching on Netflow because you can interrogate the flow cache from the command line.

The command is “show ip cache flow”, and the output is split into two parts. The first shows some statistical information about the flows that the router has observed:

router#sh ip cache flow
IP packet size distribution (3279685 total packets):
 1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
 .000 .184 .182 .052 .072 .107 .004 .005 .000 .000 .000 .000 .000 .000 .000

 512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
 .000 .000 .001 .020 .365 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
 57 active, 4039 inactive, 418030 added
 10157020 ager polls, 0 flow alloc failures
 Active flows timeout in 1 minutes
 Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 34056 bytes
 57 active, 967 inactive, 418030 added, 418030 added to flow
 0 alloc failures, 0 force free
 1 chunk, 1 chunk added
 last clearing of statistics never
Protocol     Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------     Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-WWW       6563      0.0       186  1319      1.2       4.7       1.4
TCP-other    16163      0.0         1    47      0.0       0.0      15.4
UDP-DNS         12      0.0         1    67      0.0       0.0      15.6
UDP-NTP       1010      0.0         1    76      0.0       0.0      15.0
UDP-Frag         2      0.0         6   710      0.0       0.2      15.3
UDP-other   316602      0.3         2   156      0.8       0.6      15.4
ICMP         31165      0.0         6    63      0.2      53.4       2.2
IP-other     46438      0.0        21   125      1.0      58.0       2.1
Total:      417955      0.4         7   574      3.3      11.0      12.7

In absence of a graphical Netflow analyser, the Packets/Sec counter is a good barometer of what’s “using up all the bandwidth”. To clear the stats so that you can establish a baseline, you can use the command “clear ip flow stats”.

After the stats comes a listing of all the flows currently being tracked by the router:

SrcIf     SrcIPaddress    DstIf     DstIPaddress    Pr SrcP DstP  Pkts
Fa4       xxx.xxx.xxx.xxx Local     yyy.yyy.yyy.yyy 32 3FAF 037C    16
Tu100     10.7.1.250      BV3       10.4.1.3        06 0051 C07A   663
Tu100     10.7.1.250      BV3       10.4.1.3        06 0050 C0AC   120
BV3       10.4.1.3        Tu100     10.7.1.250      06 C0AC 0050   116
Tu100     192.168.88.20   Local     172.16.7.10     01 0000 0800     5
BV3       10.4.1.3        Fa4       zzz.zzz.zzz.zzz 06 C0A2 0050   429
BV3       10.4.1.3        Tu100     10.7.1.250      06 C07A 0051   366
Fa4       bbb.bbb.bbb.bbb BV3       yyy.yyy.yyy.yyy 06 0050 C0A0     1
BV3       10.4.1.3        Fa4       ddd.ddd.ddd.ddd 06 C07E 0050     1
Tu100     192.168.88.56   Local     172.16.7.10     06 8081 0016     7
Fa4       zzz.zzz.zzz.zzz BV3       yyy.yyy.yyy.yyy 06 0050 C0A2   763
Tu100     192.168.88.28   Local     172.16.7.10     11 04AC 00A1     1
Tu100     192.168.88.28   Local     172.16.7.10     11 04A6 00A1     1
Fa4       aaa.aaa.aaa.aaa Local     yyy.yyy.yyy.yyy 32 275F BD8A     5
Fa4       ccc.ccc.ccc.ccc Local     yyy.yyy.yyy.yyy 32 97F1 E9BE     5
Tu100     10.7.1.242      Local     172.16.7.10     01 0000 0000     3
Fa4       ddd.ddd.ddd.ddd BV3       yyy.yyy.yyy.yyy 06 0050 C07E     1

The tempting simplicity of the table above hides a plethora of gotchas for the unwary:

  • The Pr (IP protocol number),  SrcP (source port) and DstP columns are in hex, but we can all do the conversion in our heads, right? ;)
  • Netflow is a unidirectional technology. That means that if hosts A and B are talking to one another via a single TCP connection, two flows will be logged – one for A->B and one for B->A. For example, these two rows in the table above are talking about the same TCP session (the four-tuple of addresses and ports is the same for both rows):
Tu100     10.7.1.250      BV3       10.4.1.3        06 0051 C07A   663
BV3       10.4.1.3        Tu100     10.7.1.250      06 C07A 0051   366
  • Unless you configure it otherwise, Netflow is an ingress technology. This means that flows are accounted for as they enter the router, not as they leave. You can determine what happens on the egress side of things because when a flow is accounted for the output interface is determined by a FIB lookup and placed in the DstIf column; in this way, you can track a flow’s path through the router. I mention this explicitly because…
  • Netflow does not sit well with NAT. Take a look at these two rows, which represent an HTTP download (port 0×0050 is 80 in decimal) requested of non-local server zzz.zzz.zzz.zzz by client 10.4.1.3:
BV3       10.4.1.3        Fa4       zzz.zzz.zzz.zzz 06 C0A2 0050   429
Fa4       zzz.zzz.zzz.zzz BV3       yyy.yyy.yyy.yyy 06 0050 C0A2   763

So what’s yyy.yyy.yyy.yyy, then? It’s the NAT inside global address representing 10.4.1.3. As Netflow is unidirectional and is recorded as it enters an interface, the returning traffic from zzz.zzz.zzz.zzz will have the post-NAT yyy.yyy.yyy.yyy as its destination address, and will be recorded as such.

Provided that you keep that lot in mind, the flow cache is a powerful tool to explore the traffic your router is handling.

NAT translations

A typical border router may well perform NAT/PAT tasks. If so, you can use the NAT database as a source of session information. On a router, the command is “show ip nat translations [verbose]“; on a PIX/ASA, it’s “show xlate [debug]“:

router#show ip nat translations
Pro Inside global         Inside local   Outside local    Outside global
tcp yyy.yyy.yyy.yyy:49314 10.4.1.3:49314 94.42.37.14:80   94.42.37.14:80
tcp yyy.yyy.yyy.yyy:49316 10.4.1.3:49316 92.123.68.49:80  92.123.68.49:80

If you’ve got a worm on your network that’s desperately trying to spread, chances are you’ll see a ton of NAT translations (which could overwhelm a small router). Rather than paging through thousands of lines of output, you can just ask the device for some NAT statistics. On a router, it’s “show ip nat statistics”; on a PIX/ASA, it’s “show xlate count”.

Keeping tabs on the number of active NAT translations is a worthwhile thing to do. I wrote a story for Security Monkey’s blog a while back which tells the tale of a worm exhausting a router’s memory with NAT translations; you can even graph the number of translations to look for anomalies over time.

Firewall sessions

Another way of extracting session information is to ask the router or PIX about the sessions it is currently tracking for firewall purposes. On a router it’s “show ip inspect sessions [detail]“; on the PIX/ASA, it’s “show conn [detail]“.

router#show ip inspect sessions detail
Established Sessions
 Session 842064A4 (10.4.1.3:49446)=>(92.123.68.81:80) http SIS_OPEN
  Created 00:00:59, Last heard 00:00:58
  Bytes sent (initiator:responder) [440:4269]
  In  SID 92.123.68.81[80:80]=>y.y.y.y[49446:49446] on ACL outside-fw (6 matches)
 Session 84206FC4 (10.4.1.3:49443)=>(92.123.68.81:80) http SIS_OPEN
  Created 00:00:59, Last heard 00:00:59
  Bytes sent (initiator:responder) [440:2121]
  In  SID 92.123.68.81[80:80]=>y.y.y.y[49443:49443] on ACL outside-fw (4 matches)
 Session 8420728C (10.4.1.3:49436)=>(92.123.68.81:80) http SIS_OPEN
  Created 00:01:01, Last heard 00:00:50
  Bytes sent (initiator:responder) [1343:48649]
  In  SID 92.123.68.81[80:80]=>y.y.y.y[49436:49436] on ACL outside-fw (44 matches)

This has the advantage of not being complicated by NAT, but still showing useful bytecounts and session durations.

Last resorts

If none of the above can help you out, there are a couple of last resort options open to you. The first of these is the “ip accounting” interface configuration command on IOS routers. To quote Cisco:

The ip accounting command records the number of bytes (IP header and data) and packets switched through the system on a source and destination IP address basis. Only transit IP traffic is measured and only on an outbound basis; traffic generated by the router access server or terminating in this device is not included in the accounting statistics. Traffic coming from a remote site and transiting through a router is also recorded.

Also note that this command will likely have a performance impact on the router. You may end up causing more problems than you solve by using this! The output of “show ip accounting” will look something like this:

router# show ip accounting
 Source          Destination            Packets      Bytes
 172.16.19.40    192.168.67.20          7            306
 172.16.13.55    192.168.67.20          67           2749
 172.16.2.50     192.168.33.51          17           1111
 172.16.2.50     172.31.2.1             5            319
 172.16.2.50     172.31.1.2             463          30991
 172.16.19.40    172.16.2.1             4            262

If “ip accounting” was a last resort, “debug ip packet” is what you’d use as an even lasterer resort, so much so that I leave it as an exercise for the reader to find out all about it. Don’t blame me when your router chokes to the extent that you can’t even enter “undebug all”…!


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk

Follow

Get every new post delivered to your Inbox.

Join 28 other followers