Squids

There are 6 clusters of squid servers, one upload and one text at each of our three locations: yaseo, knams and pmpta. Each server runs two instances of squid: a frontend squid listening on port 80, and a cache squid listening on port 3128. The purpose of the frontend squid is to distribute load to the cache squids based on URL hash, using the CARP algorithm.

LVS is used to balance incoming requests between the CARP frontends.

Squid cluster distribution

	text	upload
pmtpa	sq16-sq40 66.230.200.100	sq1-sq15, sq41-sq50 66.230.200.228
knams	knsq1-knsq7 91.198.174.2	knsq8 - knsq15 91.198.174.3
yaseo	yf1000-yf1002 203.212.189.253	yf1002 - yf1005 203.212.189.254

Installation

All Wikimedia squids are running ubuntu. All necessary packages for squid can be installed by installing the wikimedia-task-squid package from the Wikimedia APT repository.

To reinstall a previously existing Squid server:

1. Save the SSH hostkeys
2. Reinstall the server using PXE
3. After boot, copy the old SSH hostkey back using scp -o StrictHostKeyChecking=no files hostname:/etc/ssh/
4. Log in, and set the root password
5. # apt-get install wikimedia-task-squid (Answer the question about the LVS service IP very carefully)
6. From zwinger, do: cd /home/w/conf/squid/ && make && ./deploy hostname
7. (upload squids only) Run # setup-aufs-cachedirs to set up the AUFS cache partition. This will wipe any previous AUFS partition/data!
8. If the Squid had not been running for a long time, clean the cache with /etc/init.d/squid clean
   1. Anything under ~2 hours and the cache does not need to be cleaned.
9. # /etc/init.d/squid start
   1. Wait a few minutes after starting the backend here to start the frontend.
   2. You can watch TOP and see when the squid process slows down, or just read /var/log/squid/cache.log
10. # /etc/init.d/squid-frontend start
11. Install ganglia gmond

Configuration

Configuration is done by editing the master files in /home/wikipedia/conf/squid, then running make to rebuild the configuration files, and ./deploy to deploy them to the remote servers. The configuration files are:

squid.conf.php 

Template file for the cache (backend) instances

frontend.conf.php 

Template file for the frontend instances

text-settings.php 

A settings array which applies to text squids. All elements in this array will become available as variables during execution of squid.conf.php and frontend.conf.php. The settings array can be used to give server-specific configuration.

upload-settings.php 

Same as text-settings.php but for upload squids

common-acls.conf 

ACL directives used by both text and upload frontends. Use this to block clients from all access.

upload-acls.conf 

ACL directives used by upload frontends. Use this for e.g. image referrer regex blocks.

text-acls.conf 

ACL directives used by text frontends. Use this for e.g. remote loader IP blocks.

Configuration.php 

Contains most of the generator code

generate.php 

the script that the makefile runs

Feel free to check in your changes to RCS.

The deployment script has lots of options. Run it with no arguments to get a summary.

Current problems

AIO livelock

Occasionally Squid runs into a livelock, doing no system calls but using 100% user CPU. A gdb backtrace showed:

#1  0x000000000048ae99 in squidaio_sync () at aufs/aiops.c:936
#2  0x000000000048b123 in squidaio_queue_request (request=0x2aab938f4d00) at aufs/aiops.c:570
#3  0x000000000048b78c in squidaio_read (fd=13, bufp=0x2aab93923000 "", bufs=6736, offset=3744910336, whence=0, resultp=0x2aab938f4d88)

Next time this happens, collect:

print request_queue_len
print request_queue.head
print request_queue2.head
print done_queue.head
print done_requests.head

See also

• MediaWiki caching -- some cache headers explained
• Multicast HTCP purging -- new method of cache purging
• Squid logging
• Squid log format