LVS

Revision as of 03:47, 17 October 2005

Apache pool

Director setup

Dalembert is functioning as an LVS-DR director. Installing a new LVS director is just a matter of

yum install ipvsadm
ip addr add 10.0.5.3 dev eth0
scp dalembert:/usr/local/bin/lvsmon /usr/local/bin/
screen
lvsmon
^AD
run-icpagent.sh

Apache setup

When installing new apaches, one has to be very careful of the so-called "ARP problem". Eventually we might be able to configure the switch so this can't happen, but currently, if you add the LVS virtual IP to an interface of something other the director without setting arp_announce and arp_ignore on that interface, the apache may steal the IP from the director. It would then suddenly find itself with 100% of the site load.

Procedure is as follows:

cat /home/config/others/etc/sysctl.conf.local >> /etc/sysctl.conf
sysctl -w net.ipv4.conf.eth0.arp_ignore=1
sysctl -w net.ipv4.conf.eth0.arp_announce=2
sysctl -w net.ipv4.conf.eth1.arp_ignore=1
sysctl -w net.ipv4.conf.eth1.arp_announce=2

The last two commands will probably give you an error since eth1 usually doesn't exist, but you may as well run them anyway just in case. Now, I haven't tried this myself yet, but I think it would be sensible to run a test to make sure ARP is configured correctly. 10.0.5.4 is an IP address that should not be used anywhere.

ip addr add 10.0.5.4 dev lo
ssh zwinger ping 10.0.5.4

This should give "destination host unreachable". If you get a response, fix it before continuing to the next step. This is the scary step.

ip addr del 10.0.5.4 dev lo
ip addr add 10.0.5.3 dev lo

Then add it to the apaches node group and restart lvsmon on the director.

lvsmon

Lvsmon is 80 lines of PHP code written by Tim to monitor apaches and configure ipvsadm accordingly. It should be run in a screen, with no arguments. It uses curl to request http://en.wikipedia.org/w/health-check.php . Because it's so short, I'd recommend you read the code if you want to know the details. But here's an important point: it gets a list of apaches from the dsh node group, and then tests them with their unique 10/8 address, not with the VIP. So if you have apache running on a machine but you don't have it set up for LVS rotation, it's important to remove it from the apaches node group, or else intermittent "connection refused" errors will be returned to the user.

Removing apaches

Apaches can be removed from the pool temporarily by simply shutting down apache. Because lvsmon runs in a single thread, checking apaches in turn, it's probably better to remove permanently dead apaches from the apache nodelist.

Hashar's proposed squid setup

This text is a DRAFT, do not use it, base any work on it or anything else.

Needed:

Linux kernel 2.6.10+ (for IPVS patch) heartbeat ipvsadm keepalived ldirectord

We can use IP, 207.142.131.213 which is free. All clients will be sent on this IP which is assigned to a director box on a virtual interface (eth0:1). In case of failure (detected by heartbeat, the ip is switched on the other director box). That IP is know as the virtual ip (VIP).

So bascily we handle everything on our side, no more dns load balancing.

Hearbeat

Configuration file (/etc/heartbeat/ha.cf) sample in ha.cf.

We will use it on the two ldirector box, if the main one fail, the other one will detect the failure and takeover the IP address automaticly. It is probably much better than the actual way of doing things: manually reassigning ip between squids.

Squid failure is handled directly by ldirectord.

ldirectord

Configuration file (/etc/ha.d/ldirectord.cf) sample in ldirectord.cf.

ldirectord is a daemon to that will set up LVS for you, much better to edit the configuration file than editing the table with ipvsadm :o)

When one of the real server die, it will detect the failure automaticly and either gracefully discontinue it ( quiescent = yes) or remove it from the pool ( quiescent = no). If no more real server are available, a fallback can be set with an emergency page.

ipvsadm

ipvsadm is the tool to configure and check the virtual table. The table itself is set up by ldirectord so you probably do not want to mess things.

IP assignment

VIP       207.142.131.213
LVS       192.168.0.1
LVS_back  192.168.0.2

# squids 
will      10.0.0.243
browne    10.0.0.229
srv10     10.0.0.194
srv9      10.0.0.195
srv8      10.0.0.196
srv7      10.0.0.197
srv6      10.0.0.198

Network configuration

ldirector server(s) and squids need to be in the SAME broadcast network. They all share the VIP so the squids should not honour arp requests directed to the virtual VIP.

The only server that should answer on the VIP is the ldirector box, so you have to stop squids from answering arp requests that does not match their real ip. This is done by editing /etc/sysctl.conf:

net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

Now add VIP on the squid loopback:

/sbin/ip addr add 207.142.131.213/32 dev lo brd + scope host

Eventually apply /etc/sysctl changes:

echo '2' > /proc/sys/net/ipv4/conf/lo/arp_announce
echo '1' > /proc/sys/net/ipv4/conf/lo/arp_ignore

scope host directive makes the address only valid on this host.

The squid daemon is then set to listen on VIP (wich is on lo:0).