This chapter is from the book
This chapter is from the book
This chapter is from the book
Troubleshooting NAT
To troubleshoot NAT, you should first verify that each necessary step has been performed. This means:
Validate that an ARP entry exists for the translated IP (or that the translated IP is somehow being routed to the firewall).
Validate that a static host route exists on the firewall to route the translated IP address to either the untranslated address or the next hop address if the real system is more than one hop away from the firewall.
Validate anti-spoofing. Make sure that the destination IP address is being translated, and verify that the translated IP address will pass anti-spoofing checks.
Validate that the rules are set up correctly. Set any security policy rule that applies to a NAT host to track long, and ensure that address translation is happening as you expect.
Wherever a verification of the configuration fails, a packet sniffer can be your friend. The remainder of this section shows you what you should see in a packet sniffer, what you shouldn't, and how to fix it.
Although there are plenty of external packet-sniffing devices, they can be expensive and inconvenient to use. Fortunately, some operating systems come with their own. Solaris comes with a tool called snoop. IPSO, Linux, and AIX come with tcpdump. Both of these tools will be discussed briefly in this chapter. Windows NT/2000 machines come with a limited packet sniffer in Network Monitor, but you can obtain a free copy of Ethereal from http://www.ethereal.com, which works far better.
Since version 4.0, FireWall-1 has also come with its own packet-sniffing utility called fw monitor. Because it works at the same level as FireWall-1 (i.e., just after the MAC layer and before the network layer), its use in troubleshooting NAT issues is limited. fw monitor relies on INSPECT code and is discussed in Chapter 13.
Consider the network and configuration that was used in the earlier step-by-step example (see Figure 9-3). Let's assume that host 192.168.42.69 is attempting to connect to 192.168.0.13, the Intranet Web Server, which really resides at 10.0.10.80.
){kind=link}
With a successful connection, a tcpdump of the external interface should show you the following (-i is what you use to specify an interface to listen to):
# tcpdump -i eth-s1p1
tcpdump: listening on eth-s1p1
18:51:20.806020 arp who-has 192.168.0.13 tell 192.168.0.2
18:51:20.806020 arp reply 192.168.0.13 is-at 0:11:22:33:44:55
18:51:54.135062 192.168.42.69.1777 > 192.168.0.13.80: S
1184222758:1184222758(0) win 16384 <mss 1460,nop,wscale
0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
18:51:54.135062 192.168.0.13.80 > 192.168.42.69.1777: S
1332216451:1332216451(0) ack 1184222759 win 32120 <mss
1460,nop,nop,timestamp 2739310[|tcp]> (DF)
18:51:54.415021 192.168.42.69.1777 > 192.168.0.13.80: . ack 1 win
17376 <nop,nop,timestamp 11362405 2739310> (DF) [tos 0x10]
If you were to use snoop, you would see the following (-d on snoop allows you to specify an interface to listen to):
# snoop -d hme0
Using device /dev/hme (promiscuous mode)
192.168.0.2 -> (broadcast) ARP C Who is 192.168.0.13 ?
192.168.0.1 -> 192.168.0.2 ARP R 192.168.0.13 is 0:11:22:33:44:55
192.168.42.69 -> 192.168.0.13 HTTP C port=1777
192.168.0.13 -> 192.168.42.69 HTTP R port=1777
192.168.42.69 -> 192.168.0.13 HTTP C port=1777
Note that you may not necessarily see the ARP packets, especially if the originator of the packet already has the MAC address in its ARP cache. If you see SYN, SYN/ACK, and ACK packets, the connection should be established.
ARPs
The first part of the communication you should see is the request for MAC addresses via an ARP packet. When everything is working correctly, you will see an exchange like the following on the external interface with tcpdump:
18:13:20.806020 arp who-has 192.168.0.13 tell 192.168.0.2
18:13:20.806020 arp reply 192.168.0.13 is-at 0:11:22:33:44:55
With snoop, it looks like this:
192.168.0.2 -> (broadcast) ARP C Who is 192.168.0.13 ?
192.168.0.1 -> 192.168.0.2 ARP R 192.168.0.13 is 0:11:22:33:44:55
If you do only see the first packet over and over again (e.g., the ARP who is), this means that nobody owns or is proving a proxy-ARP for the translated address. Add a proxy-ARP as described previously. In some cases (especially when Windows NT is the firewall), you may need to add a static host route on the external router.
SYN Packets with No Response
You should then see the SYN packet, which looks something like this with tcpdump:
18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S
3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale
0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
With snoop, it looks like this:
192.168.42.69 -> 192.168.0.13 HTTP C port=1777
If this packet repeats over and over again, one of four things may be wrong:
The security policy is dropping the packet. Check your logs for drops.
The packet is being sent to the wrong MAC address.
The packet is not being routed properly.
The packet isn't actually getting translated, thus it is getting ignored.
Verify that the MAC address it is being sent to is correct (in this case, it should be 0:11:22:33:44:55). In tcpdump, you do this with the –e flag, which adds the MAC address to the output. In snoop, the only way to do this is with the –v flag, which unfortunately is extremely verbose.
Also in this example, you are only going to show packets coming from host 192.168.0.13 by means of adding host 192.168.0.13 to the end of your tcpdump or snoop command line. This will only show packets going to or from 192.168.0.13.
# tcpdump -e -i eth-s1p1 host 192.168.0.13
tcpdump: listening on eth-s1p1
18:21:49.201680 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82:
192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win
16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[|tcp]> (DF) [tos
0x10]
18:21:54.240965 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82:
192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win
16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[|tcp]> (DF) [tos
0x10]
18:22:07.209125 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82:
192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win
16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[|tcp]> (DF) [tos
0x10]
# snoop –v –d hme0 host 192.168.0.13
Using device /dev/hme (promiscuous mode)
ETHER: ——- Ether Header ——-
ETHER:
ETHER: Packet 27 arrived at 16:47:50.83
ETHER: Packet size = 58 bytes
ETHER: Destination = 0:55:44:33:22:11,
ETHER: Source = 0:aa:bb:cc:dd:ee,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ——- IP Header ——-
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: Total length = 44 bytes
IP: Identification = 47535
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 245 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = f23f
IP: Source address = 192.168.42.69
IP: Destination address = 192.168.0.13
IP: No options
IP:
TCP: ——- TCP Header ——-
TCP:
TCP: Source port = 2000
TCP: Destination port = 80 (HTTP)
TCP: Sequence number = 90360382
TCP: Acknowledgement number = 0
TCP: Data offset = 24 bytes
TCP: Flags = 0x02
TCP: ..0. .... = No urgent pointer
TCP: ...0 .... = No acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..1. = Syn
TCP: .... ...0 = No Fin
TCP: Window = 8760
TCP: Checksum = 0xda2b
TCP: Urgent pointer = 0
TCP: Options: (4 bytes)
TCP: - Maximum segment size = 1460 bytes
TCP:
In the preceding case, MAC 0:aa:bb:cc:dd:ee (which is the MAC of the external router) is trying to send to MAC 0:55:44:33:22:11, which is not the correct MAC. This problem can usually be resolved by flushing the ARP cache on the external router and retrying the test.
If the packet is not being routed properly, you could see a reset (RST) packet (see the section "SYN Followed by RST"), or you could see an ICMP Destination Unreachable packet. Verify that the static host route for 192.168.0.13 is pointing to the next hop address (10.0.0.2 as show in Figure 9-3).
If the packet is not actually being translated, you will see it very clearly in a tcpdump or snoop on the internal interface:
# tcpdump -i eth-s1p2 host 192.168.42.69
tcpdump: listening on eth-s1p2
18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S
3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale
0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
18:18:25.040168 192.168.42.69.1777 > 192.168.0.13.80: S
3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale
0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
18:40:30.040342 192.168.42.69.1777 > 192.168.0.13.80: S
3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale
0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
# snoop -d hme0
Using device /dev/hme (promiscuous mode)
192.168.42.69 -> 192.168.0.13 HTTP C port=1777
192.168.42.69 -> 192.168.0.13 HTTP C port=1777
192.168.42.69 -> 192.168.0.13 HTTP C port=1777
Normally, you should see the translated IP address on the internal interface. If you do not see translated packets, check your NAT rules.
SYN Followed by RST
If the packet that follows the SYN is an RST packet (with snoop, you need –V to see the TCP flags):
# tcpdump -i eth-s1p1 host 192.168.0.13
tcpdump: listening on le0
18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S
3911298019:3911298019(0) win 16384 <mss1460,nop,wscale
0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
18:13:22.040132 192.168.0.13.80 > 192.168.42.69.1777: R 0:0(0) ack
3911298020 win 0 [tos 0x10]
# snoop –V -d hme0
Using device /dev/hme (promiscuous mode)
________________________________
192.168.42.69 -> 192.168.0.13 ETHER Type=0800 (IP), size = 58
bytes
192.168.42.69 -> 192.168.0.13 IP D=192.168.0.13 S=192.168.42.69
LEN=44, ID=47247
192.168.42.69 -> 192.168.0.13 TCP D=80 S=1777 Syn Seq=3052932309
Len=0 Win=8760 Options=<mss 1460>
192.168.42.69 -> 192.168.0.13 HTTP C port=1777
________________________________
192.168.0.13 -> 192.168.42.69 ETHER Type=0800 (IP), size = 60
bytes
192.168.0.13 -> 192.168.42.69 IP D=192.168.42.69 S=192.168.0.13
LEN=40, ID=61295
192.168.0.13 -> 192.168.42.69 TCP D=64836 S=80 Rst Ack=3052932310
Win=0
192.168.0.13 -> 192.168.42.69 HTTP R port=1777
then one of three things is wrong:
The firewall is rejecting the packet on anti-spoofing. Verify that 192.168.0.13 is permitted by your anti-spoofing configuration on the internal interface.
The remote server is not running the service specified (in this case, port 80, HTTP).
The packet is being routed incorrectly.
If the remote server isn't actually running the service, you will see the following in a tcpdump and snoop on the internal interface:
# tcpdump -i eth-s1p1 host 10.0.10.80
tcpdump: listening on le0
18:13:22.040132 192.168.42.69.1777 > 10.0.10.80.80: S
3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale
0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
18:13:22.040132 10.0.10.80.80 > 192.168.42.69.1777: R 0:0(0) ack
3911298020 win 0 [tos 0x10]
# snoop –V -d hme0
Using device /dev/hme (promiscuous mode)
________________________________
192.168.42.69 -> 10.0.10.80 ETHER Type=0800 (IP), size = 58 bytes
192.168.42.69 -> 10.0.10.80 IP D=10.0.10.80 S=192.168.42.69
LEN=44, ID=47247
192.168.42.69 -> 10.0.10.80 TCP D=80
S=1777 Syn Seq=3052932309 Len=0 Win=8760 Options=<mss 1460>
192.168.42.69 -> 10.0.10.80 HTTP C port=1777
________________________________
192.168.0.13 -> 192.168.42.69 ETHER Type=0800 (IP), size = 60 bytes
192.168.0.13 -> 192.168.42.69 IP D=192.168.42.69 S=10.0.10.80 LEN=40, ID=61295
192.168.0.13 -> 192.168.42.69 TCP D=64836 S=80 Rst Ack=3052932310
Win=0
192.168.0.13 -> 192.168.42.69 HTTP R port=1777
The internal interface should see the untranslated packets (10.0.10.80 is the system's real IP). If the packet is being routed to the wrong interface, you will also see the same behavior as in the preceding output, but you will see a reject on Rule 0 in the Log Viewer as well. Verify that the static host route is set up correctly.
Useful tcpdump Flags
Table 9-4 contains a list of some useful flags for tcpdump, which takes commands in the following format:
tcpdump –i interface-name [other-flags] [expression]
Table 9-4 tcpdump Flags
|
Flag |
Description |
|
-e |
Displays MAC addresses with each packet. |
|
-i interface |
Required. Specify an interface to listen on. |
|
-l |
Buffer stdout, which is useful for piping tcpdump output to other programs. |
|
-n |
Disables name resolution on packets shown. |
|
-p |
Do not put interface in promiscuous mode (i.e., only show packets destined for the host). |
|
-r filename |
Read tcpdump capture from specified file. |
|
-s N |
Capture N number of bytes for each packet. The default is 68. (Useful with –x or –X.) |
|
-S |
Print absolute TCP sequence numbers instead of relative ones. |
|
-w filename |
Write captured packets to specified file. |
|
-x |
Hex dump of received packets. |
|
-X |
Hex and ASCII dump of received packets (IPSO only). |
tcpdump Expressions
All tcpdump commands can be followed by an expression that filters the displayed (or saved) packets so that only the packets that are interesting are shown. Some useful expressions are shown in Table 9-5.
Table 9-5 tcpdump Expressions
|
Expression |
Description |
|
port 80 |
Show all packets with source or destination port 80 (TCP or UDP). |
|
host 192.168.0.3 |
Show all packets coming from or going to host 192.168.0.3. |
|
host 192.168.0.3 and tcp port 80 |
Show all packets coming from or going to host 192.168.0.3 and that are TCP packets with a source or destination port of 80. |
|
proto vrrp |
Show all VRRP packets. On non-IPSO platforms, use "ip proto 112." |
|
icmp |
Show all ICMP packets. |
|
\(src host 192.168.0.1 or src host 192.168.0.2\) and proto vrrp |
Show all VRRP packets that originate from 192.168.0.1 and 192.168.0.2 (the \ before the parenthesis is to escape it for the shell). |
|
ether host aa:bb: cc:dd:ee:ff |
Show all packets that come from or go to the specified MAC address. |
|
ip proto 50 |
Show all packets of IP Proto 50 (IPSec AH in this example). |
|
ip[2:2] > 576 |
Show IP packets that are longer than 576 bytes ([2:2] refers to the specific byte location in the TCP header and its length). |
|
tcp[13] & 0x12 != 0 |
Show only TCP SYN/ACK packets. tcp[13] refers to the 13th byte in the TCP header of the packet. |
|
icmp[0] |
Show ICMP type 0 packets (i.e., echo reply). Icmp[0] refers to the 0th byte in the ICMP header. |
|
icmp[0] = 3 and icmp[1] = 4 |
Show ICMP type 3, code 4 packets. These happen to be a response to receiving a packet that is too large to process and has the Don't Fragment bit set. |
Useful snoop Flags
Table 9-6 contains a list of some useful flags for snoop, which takes commands in the following format:
snoop [flags] [expression]
snoop Expressions
All snoop commands can be followed by an expression that filters the displayed (or saved) packets so that only the packets that are interesting are shown.
Table 9-6 snoop Flags
|
Flag |
Description |
|
-d interface |
Specify an interface to listen on. |
|
-v |
Verbose mode. All packet data is displayed for each packet. |
|
-V |
A less-verbose verbose mode. A summary line is displayed for each layer in the ISO model. |
|
-n |
Disables name resolution on packets shown. |
|
-P |
Do not put interface in promiscuous mode, which means only show packets actually destined for this host (not useful on a switched segment). |
|
-i filename |
Read packets previously captured from file filename. |
|
-o filename |
Write packets to capture file named filename. |
|
-s numbytes |
Capture numbytes bytes for each packet. Normally, all bytes in the packet are captured. |
|
-p x,y |
Show packets numbered between x and y. The first packet captured is 1. |
|
-t [a|d|r] |
Timestamp format: a (absolute, i.e., wall clock time), d (delta, since capture was started), and r (relative time). |
Some useful expressions are shown in Table 9-7. Note that many of the expressions are similar to tcpdump.
Table 9-7 snoop Expressions
|
Expression |
Description |
|
port 80 |
Show all packets with source or destination port 80 (TCP or UDP). |
|
host 192.168.0.3 |
Show all packets coming from or going to host 192.168.0.3. You can also omit the "host" qualifier as well. |
|
host 192.168.0.3 |
Show all packets coming from or going to host 192.168.0.3 and that are |
|
and tcp port 80 |
TCP packets with a source or destination port of 80. |
|
Icmp |
Show all ICMP packets. |
|
from 192.168.0.1 |
Show all packets that originate from 192.168.0.1 or are destined for |
|
or to 192.168.0.2 |
192.168.0.2. |
|
ether aa:bb: |
Show all packets that come from or go to the specified MAC address. |
|
cc:dd:ee:ff |
|
|
ip proto 50 |
Show all packets of IP Proto 50 (IPSec AH in this example). |
|
greater 576 |
Show packets longer than 576 bytes. You can use the word "less" instead of "greater" to show packets smaller than 576 bytes. |
|
tcp[13] & |
Show only TCP SYN/ACK packets. tcp[13] refers to the 13th byte in the |
|
0x12 != 0 |
TCP header of the packet. |