PyBal

Revision as of 23:12, 7 September 2011

PyBal is a LVS monitoring script quite similar to lvsmon. It's written in Python using the Twisted framework.

At this moment, just a few features distinguish it from lvsmon:

• It's using asynchronous communication, and thus runs all checks in parallel instead of sequentially
• It has an extra monitoring method called IdleConnection, which keeps an idle connection open to all squids, and therefore notices immediately when the Squid processes are shut down / crashing
• It can fetch server lists over HTTP as well as from the local filesystems

...but I intend to polish it more, and extend it with useful things.

The script is in MediaWiki CVS SVN repo mediawiki, subdirectory pybal.

-- Mark

pybal uses AS64600 for it's BGP stuff

Setup

PyBal is currently installed on our LVS hosts, in directory /usr/sbin. Start or stop it by

 /etc/init.d/pybal start|stop

NOTE that recently pybal seems to not always stop when receiving a TERM signal (i.e. via this script). So you need to check the process start time and see if it *really* restarted, if not kill -9 manually, then start via script.

Configuration is in /etc/pybal/. pybal.conf defines the LVS service parameters, text-squids defines the list of text squid servers, upload-squids contains the list of upload squids, and >tt>apaches</tt> contains the list of all apaches. Attributes for servers:

• weight: a larger number means that more requests get sent to this server in comparison with others
• enabled: either True or False, depnding on whether you want requests to be sent to this server

The format should be fairly self explanatory; the files more or less use Python assignment / dictionary syntax.

PyBal supports multiple LVS services through a single instance and configuration file pybal.conf, e.g.:

[text]
protocol = tcp
ip = 145.97.39.155
port = 80
scheduler = wlc
config = file:///etc/pybal/text-squids

[images]
protocol = tcp
ip = 145.97.39.156
port = 80
scheduler = wlc
config = file:///etc/pybal/upload-squids

Beware, the code as checked out from Subversion has DryRun = True set in ipvs.py, meaning that it will not modify any actual IPVS state but only show the commands for debugging. This should be changed to a commandline option, but for now edit that file to DryRun = False.

The configuration files are (manually) backed up in /home/wikipedia/conf/pybal.

Howto

Depool a server

Either:

• Set 'enabled': False, in which case PyBal keeps monitoring it, but keeps it unpooled, or
• Comment the entire line in the config file, in which case PyBal doesn't even know about it

PyBal rereads its realserver config files every minute, so there's no need to restart it.