In part one, I covered how to use Cisco routers and firewalls to perform full packet capture. This exciting installment will cover how to get network session data out of these devices.
Network session data can be likened to a real-world itemised telephone bill. It tells you who “called” who, at what times, for how long, and how much was said (but not what was said). It’s an excellent lightweight way to see what’s going on armed only with a command prompt.
There are several ways to extract such information from Cisco kit; we’ll look at each in turn, following Part One’s support/troubleshooting/IR scenario of accessing remote devices where you’re not able to make topological changes or install any extra software or hardware.
Netflow
The richest source of session information on Cisco devices is Netflow (I’ll leave it to Cisco to explain how to turn it on). If you’re able to set up a Netflow collector/analyser (like this one (free for two-interface routers), or many others) you can drill down into your session info as far as you like. If you haven’t got an analyser or you can’t install one in time of need, it’s still worth switching on Netflow because you can interrogate the flow cache from the command line.
The command is “show ip cache flow”, and the output is split into two parts. The first shows some statistical information about the flows that the router has observed:
router#sh ip cache flow IP packet size distribution (3279685 total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .184 .182 .052 .072 .107 .004 .005 .000 .000 .000 .000 .000 .000 .000 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .000 .000 .001 .020 .365 .000 .000 .000 .000 .000 .000 IP Flow Switching Cache, 278544 bytes 57 active, 4039 inactive, 418030 added 10157020 ager polls, 0 flow alloc failures Active flows timeout in 1 minutes Inactive flows timeout in 15 seconds IP Sub Flow Cache, 34056 bytes 57 active, 967 inactive, 418030 added, 418030 added to flow 0 alloc failures, 0 force free 1 chunk, 1 chunk added last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) -------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow TCP-WWW 6563 0.0 186 1319 1.2 4.7 1.4 TCP-other 16163 0.0 1 47 0.0 0.0 15.4 UDP-DNS 12 0.0 1 67 0.0 0.0 15.6 UDP-NTP 1010 0.0 1 76 0.0 0.0 15.0 UDP-Frag 2 0.0 6 710 0.0 0.2 15.3 UDP-other 316602 0.3 2 156 0.8 0.6 15.4 ICMP 31165 0.0 6 63 0.2 53.4 2.2 IP-other 46438 0.0 21 125 1.0 58.0 2.1 Total: 417955 0.4 7 574 3.3 11.0 12.7
In absence of a graphical Netflow analyser, the Packets/Sec counter is a good barometer of what’s “using up all the bandwidth”. To clear the stats so that you can establish a baseline, you can use the command “clear ip flow stats”.
After the stats comes a listing of all the flows currently being tracked by the router:
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Fa4 xxx.xxx.xxx.xxx Local yyy.yyy.yyy.yyy 32 3FAF 037C 16 Tu100 10.7.1.250 BV3 10.4.1.3 06 0051 C07A 663 Tu100 10.7.1.250 BV3 10.4.1.3 06 0050 C0AC 120 BV3 10.4.1.3 Tu100 10.7.1.250 06 C0AC 0050 116 Tu100 192.168.88.20 Local 172.16.7.10 01 0000 0800 5 BV3 10.4.1.3 Fa4 zzz.zzz.zzz.zzz 06 C0A2 0050 429 BV3 10.4.1.3 Tu100 10.7.1.250 06 C07A 0051 366 Fa4 bbb.bbb.bbb.bbb BV3 yyy.yyy.yyy.yyy 06 0050 C0A0 1 BV3 10.4.1.3 Fa4 ddd.ddd.ddd.ddd 06 C07E 0050 1 Tu100 192.168.88.56 Local 172.16.7.10 06 8081 0016 7 Fa4 zzz.zzz.zzz.zzz BV3 yyy.yyy.yyy.yyy 06 0050 C0A2 763 Tu100 192.168.88.28 Local 172.16.7.10 11 04AC 00A1 1 Tu100 192.168.88.28 Local 172.16.7.10 11 04A6 00A1 1 Fa4 aaa.aaa.aaa.aaa Local yyy.yyy.yyy.yyy 32 275F BD8A 5 Fa4 ccc.ccc.ccc.ccc Local yyy.yyy.yyy.yyy 32 97F1 E9BE 5 Tu100 10.7.1.242 Local 172.16.7.10 01 0000 0000 3 Fa4 ddd.ddd.ddd.ddd BV3 yyy.yyy.yyy.yyy 06 0050 C07E 1
The tempting simplicity of the table above hides a plethora of gotchas for the unwary:
- The Pr (IP protocol number), SrcP (source port) and DstP columns are in hex, but we can all do the conversion in our heads, right? 😉
- Netflow is a unidirectional technology. That means that if hosts A and B are talking to one another via a single TCP connection, two flows will be logged – one for A->B and one for B->A. For example, these two rows in the table above are talking about the same TCP session (the four-tuple of addresses and ports is the same for both rows):
Tu100 10.7.1.250 BV3 10.4.1.3 06 0051 C07A 663 BV3 10.4.1.3 Tu100 10.7.1.250 06 C07A 0051 366
- Unless you configure it otherwise, Netflow is an ingress technology. This means that flows are accounted for as they enter the router, not as they leave. You can determine what happens on the egress side of things because when a flow is accounted for the output interface is determined by a FIB lookup and placed in the DstIf column; in this way, you can track a flow’s path through the router. I mention this explicitly because…
- Netflow does not sit well with NAT. Take a look at these two rows, which represent an HTTP download (port 0x0050 is 80 in decimal) requested of non-local server zzz.zzz.zzz.zzz by client 10.4.1.3:
BV3 10.4.1.3 Fa4 zzz.zzz.zzz.zzz 06 C0A2 0050 429 Fa4 zzz.zzz.zzz.zzz BV3 yyy.yyy.yyy.yyy 06 0050 C0A2 763
So what’s yyy.yyy.yyy.yyy, then? It’s the NAT inside global address representing 10.4.1.3. As Netflow is unidirectional and is recorded as it enters an interface, the returning traffic from zzz.zzz.zzz.zzz will have the post-NAT yyy.yyy.yyy.yyy as its destination address, and will be recorded as such.
Provided that you keep that lot in mind, the flow cache is a powerful tool to explore the traffic your router is handling.
NAT translations
A typical border router may well perform NAT/PAT tasks. If so, you can use the NAT database as a source of session information. On a router, the command is “show ip nat translations [verbose]”; on a PIX/ASA, it’s “show xlate [debug]”:
router#show ip nat translations Pro Inside global Inside local Outside local Outside global tcp yyy.yyy.yyy.yyy:49314 10.4.1.3:49314 94.42.37.14:80 94.42.37.14:80 tcp yyy.yyy.yyy.yyy:49316 10.4.1.3:49316 92.123.68.49:80 92.123.68.49:80
If you’ve got a worm on your network that’s desperately trying to spread, chances are you’ll see a ton of NAT translations (which could overwhelm a small router). Rather than paging through thousands of lines of output, you can just ask the device for some NAT statistics. On a router, it’s “show ip nat statistics”; on a PIX/ASA, it’s “show xlate count”.
Keeping tabs on the number of active NAT translations is a worthwhile thing to do. I wrote a story for Security Monkey’s blog a while back which tells the tale of a worm exhausting a router’s memory with NAT translations; you can even graph the number of translations to look for anomalies over time.
Firewall sessions
Another way of extracting session information is to ask the router or PIX about the sessions it is currently tracking for firewall purposes. On a router it’s “show ip inspect sessions [detail]”; on the PIX/ASA, it’s “show conn [detail]”.
router#show ip inspect sessions detail Established Sessions Session 842064A4 (10.4.1.3:49446)=>(92.123.68.81:80) http SIS_OPEN Created 00:00:59, Last heard 00:00:58 Bytes sent (initiator:responder) [440:4269] In SID 92.123.68.81[80:80]=>y.y.y.y[49446:49446] on ACL outside-fw (6 matches) Session 84206FC4 (10.4.1.3:49443)=>(92.123.68.81:80) http SIS_OPEN Created 00:00:59, Last heard 00:00:59 Bytes sent (initiator:responder) [440:2121] In SID 92.123.68.81[80:80]=>y.y.y.y[49443:49443] on ACL outside-fw (4 matches) Session 8420728C (10.4.1.3:49436)=>(92.123.68.81:80) http SIS_OPEN Created 00:01:01, Last heard 00:00:50 Bytes sent (initiator:responder) [1343:48649] In SID 92.123.68.81[80:80]=>y.y.y.y[49436:49436] on ACL outside-fw (44 matches)
This has the advantage of not being complicated by NAT, but still showing useful bytecounts and session durations.
Last resorts
If none of the above can help you out, there are a couple of last resort options open to you. The first of these is the “ip accounting” interface configuration command on IOS routers. To quote Cisco:
The ip accounting command records the number of bytes (IP header and data) and packets switched through the system on a source and destination IP address basis. Only transit IP traffic is measured and only on an outbound basis; traffic generated by the router access server or terminating in this device is not included in the accounting statistics. Traffic coming from a remote site and transiting through a router is also recorded.
Also note that this command will likely have a performance impact on the router. You may end up causing more problems than you solve by using this! The output of “show ip accounting” will look something like this:
router# show ip accounting
Source Destination Packets Bytes
172.16.19.40 192.168.67.20 7 306 172.16.13.55 192.168.67.20 67 2749 172.16.2.50 192.168.33.51 17 1111 172.16.2.50 172.31.2.1 5 319 172.16.2.50 172.31.1.2 463 30991 172.16.19.40 172.16.2.1 4 262
If “ip accounting” was a last resort, “debug ip packet” is what you’d use as an even lasterer resort, so much so that I leave it as an exercise for the reader to find out all about it. Don’t blame me when your router chokes to the extent that you can’t even enter “undebug all”…!
Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters@dataline.co.uk
wirewatcher 