- Protecting LAN Users from Router Loss
- Dial Backup by One Router for a Line on Another
- Protecting against LAN Segmentation
- Wrap-up
Protecting against LAN Segmentation
While using two routers at each location eliminates the problem of losing a router or connecting link, it also introduces a new vulnerability. If the LAN shared by the two routers is split into two segments, as can occur if the LAN spans multiple hubs or switches, then users on one LAN segment or the other may be cut off from effective communications, even though they remain physically connected. Consider the simplified site IP network in Figure 6-9.
){kind=link}
Figure
6-9.
Vulnerable dual router network
Under most failure conditions, the redundant components allow for continued operation. If a WAN link fails, the routers cooperate to shift all traffic transparently to the remaining link. Similarly, if one of the routers fails, the remaining router keeps all data moving. If a failure severs the link between a router and its supporting hub (or switch, as the case may be), the interface on the router fails, or the interface on the hub fails, that router will stop advertising reachability of the LAN. Again, the other router will carry all the traffic. Even if a hub fails and its users are left disconnected, the LAN interface on the router will go down and the router will send data back out across the WAN to the remaining router for those users still connected to a working hub.
Consider, however, what happens if the link between the two hubs is lost or disconnected. Both routers continue to advertise the entire subnetwork as reachable, even though each can now only reach the users on its half of the LAN. Although both routers still deliver packets from users on the LAN to the rest of the network, incoming packets may or may not be delivered to the router on the same piece of the LAN as the destination user. It depends on the routing weights on the WAN links and on the specific user.
The standard solution for this failure mode is to build the LAN using a mesh of switches that can address link failure with Spanning Tree recalculations. This allows us to support multiple physically diverse inter-switch links and eliminates the single point of failure in any one link between two different locations in the building.
While a redundant switched LAN design is necessary for high availability operation, it is not sufficient to eliminate all potential for LAN segmentation. The fundamental limitation is the single link at each router connecting it to the LAN. This means that each router must depend on a single switch for its connectivity to the rest of the LAN. Also, the physically diverse inter-switch links are no longer diverse where they come together at the switch, and there may be common paths shared inside the switch proper.
A similar failure mode also occurs when a hub, switch, or router interface fails in a way that makes the LAN port serving the router appear to the router to be alive—even though the router is no longer able to communicate with other devices on the LAN. Nor does it work to split the LAN into two independent subnetworks, each served by a single router, as then each router would again become a single point of failure. Nor is the problem limited to IP users—IPX users will suffer the same fate. Indeed, any network architecture that routes on the basis of an address containing reference to a specific physical network (as the subnetwork ID in IP and the network ID in IPX do) will fail if the physical network is split into two or more disconnected segments.
Only network architectures that do not include a level in the routing hierarchy corresponding to the physical network of attachment in the internetwork address are immune to this mode of failure. For example, the OSI Intermediate System-to-Intermediate System (IS-IS) routing protocol could theoretically recover from the failure automatically because the routing hierarchy embedded in an OSI network address skips directly from the intradomain routing area to the end system identifier.
A Workaround for IP Networks
One solution that will work with any protocol that supports dynamic routing is to connect each router to two or more shared LANs and install multihomed hosts, as described in Chapter 3, Multihomed Hosts. However, the cost of making all hosts multihomed just to counter a very low probability failure mode may be hard to justify. More powerful routers are required, the number of LAN components is doubled, and routers throughout the network are burdened with the extra networks being advertised.
If we cannot obtain all the availability we need by implementing a well-connected switched LAN from quality, high-reliability components, we can modify the network design at the router level to provide an extra layer of resistance to disruption due to LAN segmentation. Even though at this point we are assuming a well-designed, redundantly linked switched architecture, we will simplify the discussion by continuing to show two hubs connected by a single link. The actual connectivity is not relevant to the discussion as long as there is a potential for LAN segmentation that needs to be countered.
In the TCP/IP network architecture, we can stay with a single LAN and work around the problem by judicious use of variable length subnetwork masking (VLSM). To see how we can work around the LAN segmentation problem, we take the same physical configuration as in Figure 6-9 but modify the address assignments as shown in Figure 6-10. (Note that this approach will not work for IPX users, as the physical network ID in a Novell IPX network is always a fixed 32 bits, and each physical network can have only one network ID.)
){kind=link}
Figure
6-10.
Modified dual router network
At first glance, this addressing modification looks more like a step backward than an improvement. The trick is that both routers are configured to support the full range of user addresses, and each advertises the reachability of both subnetwork address ranges.
We accomplish this addressing sleight of hand by using a third, common subnetwork for routing between the two routers on the LAN (for example, 10.1.0.0/24). Each router then covers its preferred portion of the overall LAN address range using static routes pointing to the LAN interface. An additional static route on each router points to the LAN interface using a subnetwork address and mask that includes all users on the LAN (in this case 10.1.0.0/20). The static routes are then redistributed into the dynamic routing protocol with metrics that make the local LAN segment routes much more desirable than the overall route.
Normally, routers will always deliver packets to a directly connected network regardless of routing metrics. However, route specificity (prefix length) can take precedence over all other considerations. In order to deliver packets directly to the destination end-system on the LAN without first going to the router advertising that segment's IP subnetwork, we override the learned routes with a third set of static routes that have even longer prefixes. That way, routing on the LAN behaves like a single IP subnetwork rather than like multiple independent subnetworks requiring one-armed routing. Because these longer prefix routes would direct traffic from the outside to the wrong router during a LAN segmentation event, we must ensure that these more specific static routes are filtered from any routing advertisements leaving the router so that only the desired static routes are seen by the outside world.
When the LAN is operating normally, either router can send packets into the cloud from any source on the LAN and can deliver packets arriving from the cloud to any address on the LAN. Even if the two routers connect into the WAN at widely different locations or with different service quality, optimal routing is maintained. This occurs because each router advertises the other router's portion of the LAN subnetwork based on learning it via the routing protocol over the LAN, adding only a single LAN hop to the route metric. Should a router fail, the other router will continue to distribute reachability to the failed router's subnetwork, only now it will be based on the higher-cost static route for the overall LAN subnetwork pointing to the LAN interface. This route is ignored during normal operation because of its higher associated cost and lower specificity.
This means that even though each router is not "on" the other router's local subnetwork, each can still send packets to those users if one should happen to arrive. Similarly, Hot Standby Router Protocol is used to define a default gateway (or two, when load balancing) that can be reached by all user systems regardless of which routers are alive or connected to the LAN segment of the end-system.
This approach is not without its trade-offs. For example, using proxy ARP to simplify subnetwork mask independence can be problematic. Cisco routers will provide only a proxy ARP response to end-systems that have IPaddresses in the subnetwork defined on the LAN interface. Our protected end-systems are all on subnetworks that are defined only by static routes, so proxy ARP is not usable. Instead, each end-system must have a correctly configured subnetwork mask (for the overall subnetwork) and use a default gateway in the much smaller "real" subnetwork shared by the routers'LAN interfaces. Consequently, use of HSRP or VRRP is usually mandatory to prevent the default gateway from being a single point of failure.
Probably the biggest drawback to this approach is that it does nothing to alleviate the inability of users on the LAN to communicate with other users who are on the same LAN but who happen to be on the other side of the split. Following LAN segmentation, local users can continue to communicate with the outside world, but they are cut off from local users on the other side of the break. The problem remains even though the routers may still have connectivity via other LANs at the site or via the WAN. Since this limitation is present in a normally configured environment as well, it might not be considered a significant issue.
Multihomed hosts implemented using routing to a third address inside the host—as described in Chapter 3—will continue to function correctly. But local users on other LANs supported by the same routers at the site could get confused as their ability to communicate with singly connected systems on the segmented LAN will depend upon their default gateway—potentially making the ability to connect seem random.
Two other considerations are the lack of IPX support and the configuration complexity. IPX will continue to fail the same way IP does in the typical single-subnetwork configuration discussed in the introduction. The complex configuration—particularly the use of selectively redistributed routes with selective adjustment of route metrics using route maps—can be problematic by exposing defects in, and implementation differences between, various router software releases.
Configuration Example: Single LAN with Two WAN Routers
In Figure 6-11, we take the simplified configuration of Figure 6-10 and attach a more practical LAN to it using layer 2 switches. We also add a third physical switch segment to the LAN. We show how we can configure a router attached to that switch so that it can continue to communicate following a LAN segmentation event, regardless of which piece of the LAN its switch remains a part of. The approach is easily extended to allow more routers to protect additional LAN segments and switches. Similarly, the individual subnetwork masks can be adjusted to make any of the protected LAN areas capable of supporting more (or fewer) end systems.
){kind=link}
Figure
6-11.
LAN segmentation protection scenario
While this approach does provide added resilience, it also stresses the routing protocols significantly. It becomes necessary to thoroughly test all possible failure modes—not just LAN segmentation—to ensure proper operation in any given environment.
Starting with the configuration for Router 1 in Listing 6-20, only a small portion of the overall subnetwork serviced by the LAN is configured on the Ethernet interface. For EIGRP or OSPF to establish a neighbor relationship and exchange routing information across the LAN, all routers must be configured so their Ethernet interface address is on the same subnetwork using the same subnetwork mask—10.1.0.0/24 in this case. The definition of a second HSRP address on the interface is optional, but it allows us to provide some load balancing of outbound traffic.
Listing 6-20. Simple LAN segmentation resistant example, Router 1
version 11.0 ! hostname Router1 ! interface Ethernet0 ip address 10.1.0.253 255.255.255.0 no ip redirects standby 1 priority 110 standby 1 preempt standby 1 authentication mumble standby 1 ip 10.1.0.1 standby 2 priority 90 standby 2 preempt standby 2 authentication mumble standby 2 ip 10.1.0.2 ! interface Serial 0 description Link to the outside world ip address 10.255.1.6 255.255.255.252 ! router eigrp 1 redistribute static route-map Local network 10.0.0.0 no auto-summary ! ip classless ! All destinations on the local LAN ip route 10.1.0.0 255.255.240.0 Ethernet0 ! LAN subnetworks attached to this router ip route 10.1.1.0 255.255.255.0 Ethernet0 ! Routes to avoid one-armed routing ip route 10.1.2.0 255.255.255.128 Ethernet0 ip route 10.1.2.128 255.255.255.128 Ethernet0 ip route 10.1.3.0 255.255.255.128 Ethernet0 ip route 10.1.3.128 255.255.255.128 Ethernet0 ! ! Static routes denied redistribution access-list 20 permit 10.1.2.0 0.0.0.255 access-list 20 permit 10.1.3.0 0.0.0.255 ! Static routes redistributed with low metrics access-list 21 permit 10.1.1.0 0.0.0.255 ! Static routes redistributed with high metrics access-list 22 permit 10.1.0.0 0.0.15.255 ! ! Control redistribution of static routes into EIGRP route-map Local deny 25 match ip address 20 ! route-map Local permit 30 match ip address 21 set metric 10000 100 255 1 1500 ! route-map Local permit 35 match ip address 22 set metric 10000 10000 255 1 1500 ! end
The configuration under router eigrp 1 is deceptively simple, consisting of only three statements. The redistribute static route-map Local statement tells EIGRP to advertise any static routes available as specified by the route map Local. The network 10.0.0.0 statement enables EIGRP routing on all interfaces on subnetworks in the major network 10.0.0.0/8. The no auto-summary statement is required only if this router has any interfaces not in 10.0.0.0/8. It keeps all the individual routes for subnetworks inside 10.0.0.0 from being combined into a single advertisement for the 10.0.0.0 major network when leaving the 10.0.0.0 network for another major network.
The static routes that follow are what allow the router to reach all destinations on the LAN. The first route tells the router that all systems in the overall subnetwork can be reached directly through interface Ethernet0. The second route then defines the subnetwork assigned to our local switch's LAN segment, the subnetwork that this router is protecting. The next four routes split each subnetwork protected by another router on this LAN into two smaller subnetworks. This allows this router to send packets arriving for end-systems on other routers' switches directly to the addressee rather than routing them to that segment's router first.
Redistribution of the static routes into EIGRP is controlled by the route map Local, which in turn is controlled by access lists 20, 21 and 22. Access list 20 in configured to permit all static routes that we do not want to advertise that might otherwise match any tests for routes that we do want to redistribute. Access list 21 is configured to pass only our local LAN segment subnetwork, the subnetwork we are trying to protect. Finally, access list 22 needs to match the overall LAN subnetwork. By the time we get to access list 22, all other routes in the overall subnetwork must have been matched by access list 20 or 21.
The route map Local is where the critical work is done. Route map Local first checks for the long prefix static routes used to prevent one-armed routing. All static routes that are permitted by access list 20 are denied under the route-map Local deny 25 section.
Route map Local then checks for any static routes defining the local subnetwork LAN segment attached to this route runder route-map Local permit 30. These are determined by matching access list 21 in the clause match ip address 21. In addition to permitting these routes to be redistributed, the clause set metric 10000 100 255 1 1500 sets the EIGRP metric applied to the route. Because we want these routes to look like a normal subnetwork on a 10 Mbps Ethernet, we set the bandwidth to 10,000 Kbps, the delay to 100 tens of microseconds, the reliability to a perfect 255 (out of 255), the loading to the minimum of 1 (out of 255), and the maximum packet size to 1500 octets. These are the same metrics that are applied by default to the real Ethernet subnetwork 10.1.0.0/24. If the interface was a Fast Ethernet interface running at 100 Mbps, we would change the line to read set metric 100000 10 255 1 1500.
The short prefix static route for the overall LAN is then permitted by matching access list 22 under route-map Local permit 35. Since we do not want this route to be used unless the router that should be advertising a shorter prefix is completely unavailable, we set the delay metric to 10,000 tens of microseconds. With Cisco routers, this is actually not required, as a longer prefix route will always be used over a less specific route. However, it does make it clear when looking at the routing tables on a remote router that this is not the preferred route. It also clearly identifies any routes missed by the previous two filters that are being improperly advertised.
The configuration of Router 2 in Listing 6-21 is a mirror image of Router 1's. The HSRP groups are set up to be the reverse of those on Router 1 to provide a degree of load balancing for locally sourced traffic. The static routes and the access lists used by the route map controlling their redistribution are adjusted to match the LAN segment that this router is protecting.
Listing 6-21. Simple LAN segmentation resistant example—Router 2
version 11.0 ! hostname Router2 ! interface Ethernet0 ip address 10.1.0.254 255.255.255.0 no ip redirects standby 1 priority 90 standby 1 preempt standby 1 authentication mumble standby 1 ip 10.1.0.1 standby 2 priority 110 standby 2 preempt standby 2 authentication mumble standby 2 ip 10.1.0.2 ! interface Serial 0 description Link to the outside world ip address 10.255.1.10 255.255.255.252 ! router eigrp 1 redistribute static route-map Local network 10.0.0.0 no auto-summary ! ip classless ! All destinations on the local LAN ip route 10.1.0.0 255.255.240.0 Ethernet0 ! LAN subnetworks attached to this router ip route 10.1.2.0 255.255.255.0 Ethernet0 ! Routes to avoid one-armed routing ip route 10.1.1.0 255.255.255.128 Ethernet0 ip route 10.1.1.128 255.255.255.128 Ethernet0 ip route 10.1.3.0 255.255.255.128 Ethernet0 ip route 10.1.3.128 255.255.255.128 Ethernet0 ! ! Static routes denied redistribution access-list 20 permit 10.1.1.0 0.0.0.255 access-list 20 permit 10.1.3.0 0.0.0.255 ! Static routes redistributed with low metrics access-list 21 permit 10.1.2.0 0.0.0.255 ! Static routes redistributed with high metrics access-list 22 permit 10.1.0.0 0.0.15.255 ! ! Control redistribution of static routes into EIGRP route-map Local deny 25 match ip address 20 ! route-map Local permit 30 match ip address 21 set metric 10000 100 255 1 1500 ! route-map Local permit 35 match ip address 22 set metric 10000 10000 255 1 1500 ! end
Unlike Routers 1 and 2, Router 3 in Listing 6-22 does not support any external connectivity. As a result, it should never process any production traffic. Router 3's only purpose is as a source of routing advertisements for its subnetwork (10.1.3.0/24) to whichever of Routers 1 and 2 has LAN connectivity to the Router 3 LAN segment. For example, if the Router 3 LAN segment should become disconnected from Router 1, traffic for 10.1.3.0/24 will be sent from outside sources to Router 2, which will continue to propagate the preferred route learned from Router 3. Router 1 will still advertise a route to the overall subnetwork but that route will be ignored because it is less specific and has a higher cost metric.
Listing 6-22. Simple LAN segmentation resistant example—Router 3
version 11.0 ! hostname Router3 ! interface Ethernet0 ip address 10.1.0.252 255.255.255.0 ! router eigrp 1 redistribute static route-map Local network 10.0.0.0 no auto-summary ! ip classless ! All destinations on the local LAN ip route 10.1.0.0 255.255.240.0 Ethernet0 ! LAN subnetworks attached to this router ip route 10.1.3.0 255.255.255.0 Ethernet0 ! ! Static routes redistributed with low metrics access-list 21 permit 10.1.3.0 0.0.0.255 ! ! Control redistribution of static routes into EIGRP route-map Local permit 30 match ip address 21 set metric 10000 100 255 1 1500 ! end
If Router 3 fails, traffic for its LAN segment will continue to be delivered to the site based on the redistributed static route on both Routers 1 and 2. This route covers the entire subnetwork assigned to the distributed LAN. Once at Router 1 or 2, traffic will be sent straight to the end system as long as the LAN is still fully connected.
The design of Router 3 can be replicated to support additional LAN segments. Since these isolated routers never carry production traffic, they will never introduce a performance bottleneck, and the cheapest, slowest router capable of handling the routing protocol can be used. Just remember to add the appropriate static routes to Routers 1 and 2 to avoid one-armed routing, and extend access list 20 so they are not advertised.
For maximum robustness in a highly connected switched network, we can dispense with the local segment routes on the routers with external connectivity. Instead, we use a separate Router 3-like router on each switch. This protects the users on the "end" switches connecting to the external routers from being isolated because the external router on that switch developed a LAN connectivity problem.
Configuration Example: Putting It All Together
The following router configurations from a production network show how all the techniques discussed in this chapter have been applied in the real world. This example also demonstrates how network design can be influenced by history and user requirements.
The site is a critical spoke in the hub-and-spokes configuration shown in Figure 6-12. All the other spokes are single router sites using Frame Relay with ISDN backup. IP, Novell IPX, and IBM SNA communications are supported from the site to the data center at the hub site. IP is routed using EIGRP; IPX is routed using Novell IPXRIP; and IBM SNA is carried over TCP/IP using Data Link Switching (DLSw) on the site routers. The critical communications need driving the network availability requirements is SNA from controllers at the site to a central IBM system in the data center at the hub site. However, we will only look at the IP and IPX parts at this time. The IBM communications are supported by DLSw on the routers; the IBM SNA availability requirement therefore translates to a mandate that TCP/IP connections between routers at the data center and any router at the site that has an active LAN connection never go down. For now, we will focus on meeting this IP connectivity requirement and postpone further discussion of IBM SNA support until Chapter 10.
){kind=link}
Figure
6-12.
Putting it all together—overview
Other IP and IPX communications requirements were much less severe. The only Novell server accessed is at the data center, and that is required only for electronic mail support and Intel LANdesk for the help desk, neither of which was considered a critical function. Local personal computers were fully configured for stand-alone use of all personal productivity applications (other than e-mail, of course), minimizing the impact of network outages. Similarly, aside from DLSw support of SNA, the only IP usage was network management and limited access to the site's UNIX server for processing time and attendance records. Since site time clocks did not depend upon the LAN for keeping real-time records, a network outage less than a few hours in duration would be merely an inconvenience rather than a major loss.
At the site, the two routers were about 500 m apart, connected as shown in Figure 6-13. Since they were at opposite ends of a large building, the probability of LAN segmentation was significant, despite the diversity provided by the extra connection via the shipping dock. The routers themselves were Cisco 2524 modular routers that support only a single LAN interface—which in turn supports only a single HSRP address. However, since normally only one of the routers would have a WAN connection to the data center, there was no benefit to defining multiple HSRP addresses for load balancing. Indeed, rather than being an impediment, the single HSRP address served as an effective way to automatically route LAN users to the router with the best external connectivity.
){kind=link}
Figure
6-13.
Putting it all together—extended site detail
To minimize communications costs, only one of the two routers had a permanent connection to the data center. Instead, we used multiple levels of dial backup with the primary path a T1 Frame Relay link from Router WarehouseA, with PVCs defined to two independent core routers. The preferred backup path was an ISDN BRI dialed from Router WarehouseB, and analog dial over a POTS line at Router WarehouseA is the final fallback. A second physical Frame Relay link to Router WarehouseB was considered, but the idea was rejected because the cost of providing adequate physical diversity exceeded the losses associated with network failure. In other words, it was cheaper to pay overtime or transport the data entry people to an alternate location than to buy the physical diversity required to reduce the potential for an extended outage.
Due to routing anomalies found during testing, there are also configuration items that may be unique to the specific IOS versions in use. For example, we found an interaction between IOS 11.0(11) running at the data center and IOS 11.2(15) at this site that prevented lost routes from being properly deleted from an adjacent router. The workaround for the problem was to limit EIGRP advertisements on each WAN link to only carry target addresses unique to the link. While this is appropriate for a spoke in a hub-and-spokes topology—where the only impact is the need to define some extra default routes—a different workaround (or better still, elimination of the defect causing the problem) would be required for a general mesh network, in which general routing must be performed at every site.
Note in the configuration of Router WarehouseA in Listing 6-23 that EIGRP is not used to route IPX traffic even though that is a supported feature. This, too, is historical. When the network was originally implemented, EIGRP did not reliably support IPX, and NLSP was yet to be announced. So the network was designed with EIGRP routing of IP and IPXRIP routing of IPX. This did slow down recovery of IPX sessions when network failures occured, since (depending upon the exact mode of failure) it could take several minutes for IPXRIP to identify the failure and switch over to a new route. The client had plans to phase out IPX usage altogether, so this unnecessary delay was not considered a problem. However, if the design were being done today, we would use NLSP or IPX EIGRP for the IPX routing protocol.
Listing 6-23. Multiple protocol complete configuration, Router WarehouseA
version 11.0 ! hostname WarehouseA ! enable secret TopSecret ! ! Data Center router name and password for Async dial backup username AsyncServer password AsyncSecret no ip domain-lookup ! Configure IPX routing on this router ipx routing 0000.0ADC.1901 ipx internal-network ADC1901 chat-script hayes ABORT ERROR ABORT "NO " ABORT BUSY"" "at" "" "at&f" OK "atl1m1&c1&d2&k3S0=2" OK "at dt \T"
Also note that the Auxiliary port is used for dual purposes. Not only does it support the second level of dial backup, it also supports dial-in access to allow network management to directly access the router if necessary. In many environments, this could be considered an unacceptable security risk, as it exposes the routers to dial in by crackers as well as by network management personnel. Any outside access to routers must be closely monitored and controlled.
Turning our attention to the configuration of Router WarehouseB in Listing 6-24, we discover that this router actually goes beyond the stated goal of two levels of backup communication to implement a third. Even though there is no modem attached to the Auxiliary port, it is still configured to support both DDR dial out and management dial in. In an emergency, a modem could be dispatched to the site and any functional phone line in the area could be used to restore communications until the data networking lines were repaired. Enabling dial in on the modem line is not a significant security risk here. There is no modem on-line to respond to an unfriendly dial access, except while emergency restoral or access is in active use.
Listing 6-24. Multiple protocol complete configuration, Router WarehouseB
version 11.0 ! hostname WarehouseB ! enable secret TopSecret ! username ISDNserver password ISDNsecret username AsyncServer password AsyncSecret no ip domain-lookup ipx routing 0000.0ADC.1902 ipx internal-network ADC1902 isdn switch-type basic-ni1 chat-script hayes ABORT ERROR ABORT "NO " ABORT BUSY