Squids

Revision as of 04:55, 20 April 2006

Emergency operations

After a PMTPA power outage

The xx.wikipedia.org switches don't have a load balancer. IP addresses have to be assigned manually. Check /usr/local/dsh/node_groups/squid for available hosts and use nslookup to identify which IPs to assign to them. See below for details.

To switch away from Paris squids

• Need to be root
• On zwinger:
  • Edit /usr/local/etc/powerdns/geomaps/rr.wikimedia.org. Each non comment line is a country code mapping to a cluster name.(0 is default route, to go to Florida, must be left in).
  • Load root ssh key
  • Run /usr/local/etc/powerdns/update to deploy.
  • Wait for DNS propagation time (600s, in /usr/local/etc/pdns.conf geo-ttl=600 . Might reduce it to give faster switch back after problem is over, reducing it can't make the emregency fix work faster though - it's cache time for other DNS servers)
• Reverse these changes to switch back.

General

• to add new IPs for squid:
  • add the IPs in /var/named/master/wikipedia.zone in the obvious places (on zwinger)
  • make sure to update the serial number in the zone file!
  • run rndc reload to make it update - no need to restart named

• some useful commands:
  • dig ANY en.wikipedia.org @zwinger.wikipedia.org
  • dig +short ANY en.wikipedia.org

• editing squid.conf for all: /h/w/conf/squid, read the README

Squid builds

Currently running:

• stable9 + all stable 9 patches except broken 2GB patch + gwicke + nortt + htcpclr (identifies as: 2.5.STABLE9.wp20050410.S9plus.no2GB[icpfix,nortt,htcpclr]) builddir: /home/wikipedia/src/squid/squid-2.5.STABLE9-kate-no2GB
• squid-2.5.STABLE12RC1-1wm.src.rpm - Multiple binary builds exist for different arches and/or distributions.

IPs of virtual ethernet interfaces

• to find out the IPs, use nslookup rr.pmtpa.wikimedia.org
• Used to be assigned at boot time, but this can lead to problems with duplicated IP addresses.
• Can use script takeip in /home/wikipedia/bin to take over an IP if a squid goes down.
• To take down a virtual eth interface, /sbin/ip addr del xxx.xxx.xxx.xxx dev eth0
• According to dammit, should use 255.255.255.255 for netmask for the IPs of virtual eth interfaces - otherwise there can be routing confusion. takeip does this automatically

TODO

set up heartbeat or wackamole for automatic IP takeover.

Starting

Squid RPM

The Squid RPM has a SysV init script /etc/init.d/squid like any proper RPM, start it using

# /sbin/service squid start

It's automatically started at boot time, alter this using /sbin/chkconfig.

Old squid builds

/usr/local/bin/ squid is a symlink to /usr/local/squid/RunCache which is a loop that starts squid with the right args and sets the ulimit (squid needs more than 1024 fd's, hence it does ulimit -n 8192).

Reloading

After config changes, call squidhup (in /usr/local/bin) which is short for /usr/local/squid/sbin/squid -k reconfigure.

Logs

Logs are rotated every 10 minutes, and deleted shortly after. The logrotate config is in /etc/logrotate.d/squid.

cachemgr.cgi

this needs to be updated

There is a cachemgr.cgi available at http://noc.wikimedia.org/~mark/cgi-bin/cachemgr.cgi. The password (for at least the French squids at this moment) can be found in /home/wikipedia/doc/fr-cachemgr-pw.

to investigate

• Selectively remove Squid-2 cache contents: http://www.wa.apana.org.au/~dean/squidpurge/

Diagnostic aid

innocence modified the squid error pages to identify which squid was erroring; it was mostly benet on 14 Jan 04; Steps Were Taken

New Donors

As, and even before, we get our protocols entirely worked out for remote squid sites, people are going to volunteer. That link will provide such sites with a place to put their contact and other information, and for us to ask them questions.

See also

• MediaWiki caching -- some cache headers explained
• Multicast HTCP purging -- new method of cache purging
• French squids -- for a documentation on the french cluster

New squid setup

memcheck

Squid has demanding memory access patterns, these memtest options might help to catch more problems before they are real ones: "Also, go to the options, and turn on caching, as well as all memory addresses and tests ... (keys pressed if I can remember, is:

c->1->2->2->3->3->3

should turn on above options for memtest)." (from [1])

Setup

This is out of date information, adapt to the new Squid RPM

• Add the machine to /home/wikipedia/conf/squid/Makefile, run make
• On the new machine run /home/wikipedia/bin/squidsetup as root after doing a ssh-add. If there are errors fetching the squid stuff, adjust the server to fetch from in the script.
• follow the instructions re crontab
• test the server using telnet or nc
• take ips
• add /usr/local/bin/squid to /etc/rc.local