- Fault Tolerance Principles
- RAID
- Ethernet Bonding
- Clusters
This chapter is from the book
This chapter is from the book
This chapter is from the book
Ethernet Bonding
As you develop fault-tolerant systems, you quickly realize that after disk failures, network issues are probably the second area that requires redundancy. After all, switches need maintenance from time to time and they do fail, as do networking cards and even Ethernet cables. If you want to be able to survive switch failure or maintenance, you need a system with multiple Ethernet ports connected to redundant switches. Most servers these days come with at least two Ethernet ports if not more, so it makes sense to set up Ethernet bonding, especially when you see how easy it is.
Ethernet bonding is a feature built into the Linux kernel as a module. With Ethernet bonding you can have multiple Ethernet ports that answer to the same IP address. Depending on the bonding mode you choose, you could have an active/passive scenario where one port activates only if the other appears off-line, or you could have an active/active scenario where you accept traffic across all ports.
Full documentation of all of the Ethernet bonding modes is available in the Documentation/networking/bonding.txt file included with any Linux kernel source. Below is an excerpt from that documentation that describes each bond mode:
balance-rr or 0
Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.
active-backup or 1
Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond’s MAC address is externally visible on only one port (network adapter) to avoid confusing the switch.
balance-xor or 2
XOR policy: Transmit based on the selected transmit hash policy. The default policy is a simple [(source MAC address XOR’d with destination MAC address) modulo slave count]. Alternate transmit policies may be selected via the xmit_hash_policy option, described below. This mode provides load balancing and fault tolerance.
broadcast or 3
Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance.
802.3ad or 4
IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification.
balance-tlb or 5
Adaptive transmit load balancing: channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.
balance-alb or 6
Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond such that different peers use different hardware addresses for the server.
So which bonding mode should you use? This can be a difficult question to answer as different networks support certain modes better than others. My recommendation is to try out some of the different bonding modes and test their fail-over by unplugging a cable while pinging the server. Different modes handle port failure differently, especially in the case where a cable is reconnected (or a switch is rebooted) and the port takes 30 seconds or so to come up. On some bonding modes pings will continue with no interruption, while on others you might have a 30-second outage while the port comes up. Note that because the bonding mode is set in the bonding module when it is loaded, if you change the bonding mode you will likely need to reboot (or at least take down the bond0 interface and unload and reload the module). For this example I will choose bonding mode 1 since it has only one port active at a time, so it is relatively safe on any switch.
The first step to configure bonding is to install the ifenslave package:
$ sudo apt-get install ifenslave
This package includes the ifenslave utility which the system will use to bond two interfaces together. The next step is to open /etc/modprobe.d/ aliases, scroll to the bottom of the file, and add
alias bond0 bonding options bonding mode=1 miimon=100
The options line is what you can use to change your bonding mode. The miimon option tells the kernel how often to check the link state of the interface in milliseconds. In this case it is checked every 100 milliseconds.
The next step is to open your /etc/network/interfaces file and comment out any configuration lines for the network interfaces you will bond (you will probably have a configuration only for eth0). Also, if you have any references to auto eth0, comment those out as well. Then create a new configuration for the bond0 interface that mimics the settings you had for eth0. At the very end of the bond0 configuration you add an extra line called slaves that lists the different interfaces you want to bond together. Here’s an example interfaces file for my server:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
#auto eth0
#iface eth0 inet static
# address 192.168.0.5
# netmask 255.255.255.0
# gateway 192.168.0.1
auto bond0
iface bond0 inet static
address 192.168.0.5
netmask 255.255.255.0
gateway 192.168.0.1
slaves eth0 eth1
Save your changes and then run sudo service networking restart or sudo /etc/init.d/networking restart. Once you run ifconfig, you should see the new bond0 device:
$ sudo ifconfig
bond0 Link encap:Ethernet HWaddr 00:0c:29:28:13:3b
inet addr:192.168.0.5 Bcast:192.168.0.255
Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe28:133b/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:38 errors:0 dropped:0 overruns:0 frame:0
TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:16644 (16.2 KB) TX bytes:3282 (3.2 KB)
eth0 Link encap:Ethernet HWaddr 00:0c:29:28:13:3b
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:37 errors:0 dropped:0 overruns:0 frame:0
TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:16584 (16.1 KB) TX bytes:3282 (3.2 KB)
Interrupt:17 Base address:0x1400
eth1 Link encap:Ethernet HWaddr 00:0c:29:28:13:3b
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:60 (60.0 B) TX bytes:0 (0.0 B)
Interrupt:18 Base address:0x1480
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Now you can test fail-over by unplugging eth0 while pinging the IP. Whenever a particular interface is down, the kernel will log both to dmesg and to /var/log/syslog. Here’s an example log entry like one you would see if you unplugged eth0:
Feb 14 16:43:28 kickseed kernel: [ 2901.700054] eth0: link down
Feb 14 16:43:29 kickseed kernel: [ 2901.731190] bonding: bond0:
link status definitely down for interface eth0, disabling it
Feb 14 16:43:29 kickseed kernel: [ 2901.731300] bonding: bond0:
making interface eth1 the new active one.