Squids

Revision as of 15:34, 22 August 2006

Architecture

There are 5 squid farms at the moment:

• PMTPA text squids, serving all text pages. Uses IP/DNS-based round robin, no load balancer.
• PMTPA image squids, serving upload.wikimedia.org. Uses LVS load balancing on avicenna.
• YASEO text squids, serving all text pages for Asian users. Uses LVS load balancing.
• KNAMS text squids, serving all text pages for European users. Uses LVS load balancing on pascal.
• KNAMS image squids, serving upload.wikimedia.org. Uses LVS load balancing on pascal.
• LOPAR text squids, unused at the moment. decomissioned

Emergency operations

After a PMTPA power outage

The xx.wikipedia.org switches don't have a load balancer. IP addresses have to be assigned manually. Check /usr/local/dsh/node_groups/squid for available hosts and use nslookup to identify which IPs to assign to them. See below for details.

To switch away from foreign squids

• Need to be root
• On zwinger:
  • Edit /usr/local/etc/powerdns/geomaps/rr.wikimedia.org. Each non comment line is a country code mapping to a cluster name.(0 is default route, to go to Florida, must be left in).
  • Load root ssh key
  • Run /usr/local/etc/powerdns/update to deploy.
  • Wait for DNS propagation time (600s, in /usr/local/etc/pdns.conf geo-ttl=600 . Might reduce it to give faster switch back after problem is over, reducing it can't make the emregency fix work faster though - it's cache time for other DNS servers)
• Reverse these changes to switch back.

Squid builds

Currently running:

• squid-2.5.STABLE13-7wm.src.rpm
• squid-2.6.STABLE3-1wm.src.rpm

Multiple binary builds exist for different arches and/or distributions. Can be found under /home/wikipedia/rpms/squid and Subversion.

IPs of virtual ethernet interfaces on old PMTPA text Squids

• to find out the IPs, use nslookup rr.pmtpa.wikimedia.org
• Used to be assigned at boot time, but this can lead to problems with duplicated IP addresses.
• Can use script takeip in /home/wikipedia/bin to take over an IP if a squid goes down.
• To take down a virtual eth interface, /sbin/ip addr del xxx.xxx.xxx.xxx dev eth0
• According to dammit, should use 255.255.255.255 for netmask for the IPs of virtual eth interfaces - otherwise there can be routing confusion. takeip does this automatically

TODO

set up LVS

Common operations

Starting

The Squid RPM has a SysV init script /etc/init.d/squid like any proper RPM, start it using

# /sbin/service squid start

It's automatically started at boot time, alter this using /sbin/chkconfig.

Reloading

# /sbin/service squid restart

cachemgr.cgi

this needs to be updated

There is a cachemgr.cgi available at http://noc.wikimedia.org/~mark/cgi-bin/cachemgr.cgi. The password (for at least the French squids at this moment) can be found in /home/wikipedia/doc/fr-cachemgr-pw.

See also

• MediaWiki caching -- some cache headers explained
• Multicast HTCP purging -- new method of cache purging
• French squids -- for a documentation on the french cluster

New squid setup

• Install the Squid RPM
• Adapt LVS

Squid 2.6

Squid 2.6 has been released, with some relevant changes to us, mostly concerning performance and accelerator features. Nearly all Wikimedia specific patches entered the distribution, which is helpful to maintain the RPM.

However, the configuration file changed in some incompatible ways.

Configuration file changes

The following lines, to set Squid up as an HTTP accelerator, no longer exist:

httpd_accel_port 80
httpd_accel_host virtual
httpd_accel_with_proxy on
httpd_accel_uses_host_header on

Instead, this is now indicated with the vhost option of http_port:

http_port 80 vhost

There is a new htcp-oldsquid option for cache_peer, which should be used for Squids still running version 2.5. The HTCP implementation used is incompatible with the 2.5 implementation.

HTCP queries and HTCP CLR (purge) packets are now guarded by an access list. To allow these queries for Wikimedia servers, use:

# HTCP and HTCP CLR access
htcp_access allow all

htcp_clr_access allow tiertwo
htcp_clr_access deny all