Squids

This page desperately needs to be filled out or brought up to date. If you're familiar with the operations of this part of the site, please help!

FIXME: How to update squid error pages

Architecture

There are 5 squid farms at the moment:

• PMTPA text squids, serving all text pages. Uses IP/DNS-based round robin, no load balancer.
• PMTPA image squids, serving upload.wikimedia.org. Uses LVS load balancing on avicenna.
• YASEO text squids, serving all text pages for Asian users. Uses LVS load balancing.
• KNAMS text squids, serving all text pages for European users. Uses LVS load balancing on pascal.
• LOPAR text squids, unused at the moment. decomissioned

Emergency operations

After a PMTPA power outage

The xx.wikipedia.org switches don't have a load balancer. IP addresses have to be assigned manually. Check /usr/local/dsh/node_groups/squid for available hosts and use nslookup to identify which IPs to assign to them. See below for details.

To switch away from foreign squids

• Need to be root
• On zwinger:
  • Edit /usr/local/etc/powerdns/geomaps/rr.wikimedia.org. Each non comment line is a country code mapping to a cluster name.(0 is default route, to go to Florida, must be left in).
  • Load root ssh key
  • Run /usr/local/etc/powerdns/update to deploy.
  • Wait for DNS propagation time (600s, in /usr/local/etc/pdns.conf geo-ttl=600 . Might reduce it to give faster switch back after problem is over, reducing it can't make the emregency fix work faster though - it's cache time for other DNS servers)
• Reverse these changes to switch back.

Squid builds

Currently running:

• squid-2.5.STABLE13-7wm.src.rpm - Multiple binary builds exist for different arches and/or distributions.

IPs of virtual ethernet interfaces on old PMTPA text Squids

• to find out the IPs, use nslookup rr.pmtpa.wikimedia.org
• Used to be assigned at boot time, but this can lead to problems with duplicated IP addresses.
• Can use script takeip in /home/wikipedia/bin to take over an IP if a squid goes down.
• To take down a virtual eth interface, /sbin/ip addr del xxx.xxx.xxx.xxx dev eth0
• According to dammit, should use 255.255.255.255 for netmask for the IPs of virtual eth interfaces - otherwise there can be routing confusion. takeip does this automatically

TODO

set up LVS

Common operations

Starting

The Squid RPM has a SysV init script /etc/init.d/squid like any proper RPM, start it using

# /sbin/service squid start

It's automatically started at boot time, alter this using /sbin/chkconfig.

Reloading

# /sbin/service squid restart

cachemgr.cgi

this needs to be updated

There is a cachemgr.cgi available at http://noc.wikimedia.org/~mark/cgi-bin/cachemgr.cgi. The password (for at least the French squids at this moment) can be found in /home/wikipedia/doc/fr-cachemgr-pw.

See also

• MediaWiki caching -- some cache headers explained
• Multicast HTCP purging -- new method of cache purging
• French squids -- for a documentation on the french cluster

New squid setup

• Install the Squid RPM
• Adapt LVS