Disaster had struck when both of the Arrowpoint Content Switches died with power supply faults. We needed to move them in the racks, they powered off but they didn’t power back on.

I looked at off the shelf solutions and was forced to rethink when I discovered that they come in with a price tag of between £2000 and £9000 each device. Whilst visiting on a good friend and colleague Scott the subject load balancers came up. His advise was to check out the Linux Virtual Server Project and in particular to look at ipvsadm.

We were looking for a simple solution to load balance both secure and non-secure web servers. After hunting around the internet I came across Keepalived and I am impressed to say that with a small amount of iptables it pretty much just works out of the box.

The solution consists of two load balancers  lb1 and lb2 sitting between the Internet and the web server farm. VRRP is the redundancy protocol used to allow multiple boxes to work together.

The Internet facing IP addresses are managed by keepalived and associated with vrrp 1. An IP address is allocated as the default gateway for the web server farm and this is associated with vrrp 2. Keepalived ensures that only the active lb has these ip addresses active.

We run VMWare ESX Servers which aids fast deloyment of new services. It’s also quite useful as you can create the build you want the copy it as many times as you need … I digress.

From the list of Linux Distro’s I chose Ubuntu 8.10 Server as it was one I had to hand.  Once installed the packages were patched using apt-get and then keepalived was installed

apt-get install keepalived

This creates a folder /etc/keepalived which contains the control file keepalived.conf an document example as follows.

global_defs {
	notification_email {
		mick@mydomain.com
	}
	notification_email_from keepalived@mydomain.com
	smtp_server localhost
	smtp_connect_timeout 30
# name associated with this load balancer LVS_BACKUP is my backup
	router_id LVS_MASTER
}

vrrp_sync_group VG1 {
	group {
		VI_PUBLIC
		VI_GATEWAY
	}
}

vrrp_instance VI_PUBLIC {
# this denotes the default state - the backup is state BACKUP
	state MASTER
# interface connected to the public lan
	interface eth0
# vrrp 1
	virtual_router_id 1
	lvs_sync_daemon_interface eth0
# sets who is primary and backup (backup priority is 150)
	priority 200
	authentication {
		auth_type PASS
		auth_pass xxx
	}
	virtual_ipaddress {
# the public ip address associated with the web farm
		200.200.200.200/24
	}
}

vrrp_instance VI_GATEWAY {
# this denotes the default state - the backup is state BACKUP
	state MASTER
# interface connected to the web farm lan
	interface eth1
	lvs_sync_daemon_interface eth2
# vrrp 2
	virtual_router_id 2
# sets who is primary and backup (backup priority is 150)
	priority 200
	advert_int 1
	smtp_alert
	authentication {
		auth_type PASS
		auth_pass ThisPassword
	}
	virtual_ipaddress {
# the default gateway for the web farm
		192.168.1.1/24
	}
}
# public ip address associated with the webserver
virtual_server 200.200.200.200 80 {
# HTTP 1.1 hostname
	virtualhost www.mydomain.com
	delay_loop 30
	lb_algo wlc
# we are NAT'ing the addresses
	lb_kind NAT
	nat_mask 255.255.255.0
	persistence_timeout 50
	protocol TCP
# ip address of real web server 1 from farm
	real_server 192.168.1.100 80 {
		weight 1
		HTTP_GET {
			url {
# look for a small image that you know will always exist
				path /icons/unknown.gif
# if web server okay you will get a status code 200
				status_code 200
			}
# you'll need to tinker around with these to get them just right
			connect_timeout 20
			nb_get_retry 10
			delay_before_retry 30
# web server runs of port 80
			connect_port 80
		}
	}
# ip address of real web server 2 from farm
	real_server 192.168.1.101 80 {
		weight 1
		HTTP_GET {
			url {
				path /icons/unknown.gif
				status_code 200
			}
			connect_timeout 20
			nb_get_retry 10
			delay_before_retry 30
			connect_port 80
		}
	}
}

Because the load balancers will be the default gateways for the web server farm. We need to configure iptables nat masqurade so that if the web server needs to talk to the internet it will have a public IP address. The following very basic /etc/iptables.rules is used to nat the replies.

:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [4980:5513190]
:OUTPUT ACCEPT [3522:318353]
-A INPUT -i l0 -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i eth1 -j ACCEPT
-A FORWARD -i eth1 -j ACCEPT
COMMIT
*nat
  REROUTING ACCEPT [4036:3275594]
  OSTROUTING ACCEPT [118:7080]
:OUTPUT ACCEPT [142:8825]

-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT

To bind this in to ubunu a script load_tables containing

#!/bin/sh
iptables-restore < /etc/iptables.rules

Is placed in/etc/network/if-pre-up.d and then it will be called on boot.