Squids

The squids located near Paris should not be used for traffic for clients outside France, due to the current (temporary) throttling on some network destinations imposed by the company offering free connectivity.

• Current squid machines: browne, benet, maurus, rabanus, will

• to add new IPs for squid:
  • add the IPs in /var/named/master/wikipedia.zone in the obvious places (on zwinger)
  • make sure to update the serial number in the zone file!
  • run rndc reload to make it update - no need to restart named

• some useful commands:
  • dig ANY en.wikipedia.org @zwinger.wikipedia.org
  • dig +short ANY en.wikipedia.org

• editing squid.conf for all: /h/w/conf/squid, read the README

IPs of virtual ethernet interfaces

• to find out the IPs, run host en.wikipedia.org and host cache.wikimedia.org
• Used to be assigned at boot time, but this can lead to problems with duplicated IP addresses.
• According to damyta, should use 255.255.255.255 for netmask for the IPs of virtual eth interfaces - otherwise there can be routing confusion.
  • We've been using this configuration for months now and it works fine. -- Jeronim 10:41, 19 Sep 2004 (UTC)
• You must ping through the switch via a virtual interface which you have just brought up (using ping -I), to update the switch's idea of who has what IP. When pinging with -I207.142.131.248 or similar, use suda as the ping target; pinging the broadcast address does not always work.
  • Can use script takeip in /home/wikipedia/bin to take over an IP if a squid goes down.
  • To take down a virtual eth interface, /sbin/ifconfig eth0:n down
  • TODO: set up heartbeat or wackamole for automatic IP takeover.

Things to check when it goes crazy

• check to see if access.log (probably in /var/log/squid) has reached 2GB in size. If it has, logrotate it or at least rename it.

Starting

• (/usr/local/bin/) squid is a symlink to /usr/local/squid/RunCache which is a loop that starts squid with the right args and sets the ulimit (squid needs more than 1024 fd's, hence it does ulimit -n 8192).

Reloading

After config changes, call squidhup (in /usr/local/bin) which is short for /usr/local/squid/sbin/squid -k reconfigure.

Logs

Currently on yongle in /var/backup/archiv, and/or on zwinger in /home/wikipedia/logs/archiv/

French squids

See caching servers out of Florida for rationale and performance.

• Hardware
  • 3 600MHz Celeron 1U machines with 20GB HDD – HP Web Hosting Server Appliance sa1100 - specs: [1]
  • 2 serial ports, 2 NICs, no graphics card
  • Debian
  • use PC133 ECC unregistered DIMMs, 256 MiB max per DIMM, 4 slots per machine (possibility of using registered RAM? probably only if all DIMMs are registered)
  • ennael: 768 MiB RAM (original 128 MiB DIMM faulty and removed on January 12, 2005; 256 MiB DIMM installed on Sunday 16, 2005); chloe, bleuenn: 640 MiB RAM (128 in original machine + 2 × 256 upgrade)

• Network connectivity
  • Dexlan 5-port 100base-TX Ethernet switch (port 1=uplink, ports 2-4=squids, port 5=free); all cables straight;
  • names are chloe (212.85.150.132), bleuenn (212.85.150.133), and ennael (212.85.150.131);
  • addresses 212.85.150.130 and 212.85.150.134 may also be used to connect other machines (laptop for maintenance...);
  • IP block : 212.85.150.128/29
  • Network : 212.85.150.128
  • Netmask : 255.255.255.248
  • Broadcast : 212.85.150.135
  • Gateway : 212.85.150.129
• null-modem serial cables run from ttyS1 on one machine to ttyS0 on another; setting is 19200 bps;
  • getty running on ttyS0 on each machine, to allow communication using minicom;
  • order (caller to listener) is chloe → ennael → bleuenn → chloe;
  • BIOS, grub and kernel all configured for serial console; BIOS and grub accessible through serial console;
  • as a consequence, any remote rebooting should be made by logging into the preceding machine in the ring and running minicom onto the machine to be rebooted;
• rack space provided by Lost Oasis inside the Telecity colocation in Aubervilliers near the northern city limits of Paris.
• Contact: Med

Performance note: from broadband connections in France, at off-peak times, the download speed for large files is around 80 kiB/s when not in cache and 235 kiB/s when in cache.

New squid setup

#!/bin/sh
echo 'getting squid from browne'
scp -r  root@browne.wikimedia.org:/usr/local/squid /usr/local
echo 'getting config'
scp -r  root@zwinger.wikimedia.org:/home/gwicke/squid/newsquid.conf /usr/local/squid/etc/squid.conf
echo 'create a shortcut to /usr/local/bin/squid'
ln -s /usr/local/squid/bin/RunCache /usr/local/bin/squid
#squid user/group is already existing on all servers
#echo 'setting up the squid user'
#adduser --system --no-create-home --disabled-login squid
#echo 'setting up the squid group'
#addgroup --system --no-create-home squid

echo 'creating /var/spool/squid and /var/log/squid'
mkdir /var/spool/squid
chown -R squid:squid /var/spool/squid
mkdir /var/log/squid
chown -R squid:squid /var/log/squid
mkdir /var/log/squid/outgoing
chown -R squid:squid /var/log/squid/outgoing
echo 'edit the configuration'
vim /usr/local/squid/etc/squid.conf
echo 'initializing cache dirs'
/usr/local/squid/sbin/squid -z

echo 'getting log rotation scripts'
scp root@zwinger.wikimedia.org:/home/gwicke/squid/crontab /root
scp root@browne.wikimedia.org:/usr/local/bin/preplogs_for_move /usr/local/bin/preplogs_for_move
scp root@browne.wikimedia.org:/etc/logrotate.d/squid /etc/logrotate.d/squid
# don't overwrite root's crontab, as there maybe something important in there
# crontab -u root /root/crontab

• log transfer from /var/log/squid/outgoing/* to /home/wikipedia/logs/incoming/{hostname}/

cachemgr.cgi

There is a cachemgr.cgi available at http://noc.wikimedia.org/~mark/cgi-bin/cachemgr.cgi. The password (for at least the French squids at this moment) can be found in /home/wikipedia/doc/fr-cachemgr-pw.

to investigate

• Selectively remove Squid-2 cache contents: http://www.wa.apana.org.au/~dean/squidpurge/

Sibling problems

The Paris squids (and possibly the Florida squids) do not fetch content from each other properly. Experimentally, any of the three will answer with its own cached content if asked for a page that it has in its cache, but will request content from Florida even if one of the other two has the content.

Diagnostic aid

innocence modified the squid error pages to identify which squid was erroring; it was mostly benet on 14 Jan 04; Steps Were Taken

New Donors

As, and even before, we get our protocols entirely worked out for remote squid sites, people are going to volunteer. That link will provide such sites with a place to put their contact and other information, and for us to ask them questions.

See also

• MediaWiki caching -- some cache headers explained