Squids

Revision as of 17:32, 14 April 2005

Emergency operations

To switch away from Paris squids

• Need to be root
• On zwinger and larousse:
  • vim /usr/local/etc/pdns.geomap and comment out all lines beginning with a number other than 0 (0 is default route, to go to Florida, must be left in).
  • pdns_control rediscover to tell PowerDNS to reload configuration.
  • Wait for DNS propagation time (600s, in /usr/local/etc/pdns.conf geo-ttl=600 . Might reduce it to give faster switch back after problem is over, reducing it can't make the emregency fix work faster though - it's cache time for other DNS servers)
• Reverse these changes to switch back.

Big trouble with Florida squids

• check to see if access.log (probably in /var/log/squid) has reached 2GB in size. If it has, logrotate it or at least rename it.

General

• Current squid machines: browne, benet, maurus, rabanus, will

• to add new IPs for squid:
  • add the IPs in /var/named/master/wikipedia.zone in the obvious places (on zwinger)
  • make sure to update the serial number in the zone file!
  • run rndc reload to make it update - no need to restart named

• some useful commands:
  • dig ANY en.wikipedia.org @zwinger.wikipedia.org
  • dig +short ANY en.wikipedia.org

• editing squid.conf for all: /h/w/conf/squid, read the README

Squid builds

Currently running:

• stable9 + all stable 9 patches except broken 2GB patch + gwicke + nortt + htcpclr (identifies as: 2.5.STABLE9.wp20050410.S9plus.no2GB[icpfix,nortt,htcpclr])
• builddir: /home/wikipedia/src/squid/squid-2.5.STABLE9-kate-no2GB

IPs of virtual ethernet interfaces

• to find out the IPs, run host en.wikipedia.org and host cache.wikimedia.org
• Used to be assigned at boot time, but this can lead to problems with duplicated IP addresses.
• According to damyta, should use 255.255.255.255 for netmask for the IPs of virtual eth interfaces - otherwise there can be routing confusion.
  • We've been using this configuration for months now and it works fine. -- Jeronim 10:41, 19 Sep 2004 (UTC)
• You must ping through the switch via a virtual interface which you have just brought up (using ping -I), to update the switch's idea of who has what IP. When pinging with -I207.142.131.248 or similar, use suda as the ping target; pinging the broadcast address does not always work.
  • Can use script takeip in /home/wikipedia/bin to take over an IP if a squid goes down.
  • To take down a virtual eth interface, /sbin/ifconfig eth0:n down
  • TODO: set up heartbeat or wackamole for automatic IP takeover.

Starting

• (/usr/local/bin/) squid is a symlink to /usr/local/squid/RunCache which is a loop that starts squid with the right args and sets the ulimit (squid needs more than 1024 fd's, hence it does ulimit -n 8192).

Reloading

After config changes, call squidhup (in /usr/local/bin) which is short for /usr/local/squid/sbin/squid -k reconfigure.

Logs

Currently on yongle in /var/backup/archiv, and/or on zwinger in /home/wikipedia/logs/archiv/

French squids

See caching servers out of Florida for rationale and performance.

Setup

• Hardware
  • 3 600MHz Celeron 1U machines with 20GB HDD – HP Web Hosting Server Appliance sa1100 - specs: [1]
  • 2 serial ports, 2 NICs, no graphics card
  • Debian
  • use PC133 ECC unregistered DIMMs, 256 MiB max per DIMM, 4 slots per machine (possibility of using registered RAM? probably only if all DIMMs are registered)
  • ennael: 768 MiB RAM (original 128 MiB DIMM faulty and removed on January 12, 2005; 256 MiB DIMM installed on Sunday 16, 2005); chloe, bleuenn: 640 MiB RAM (128 in original machine + 2 × 256 upgrade)
  • On the way, waiting to be installed: sophie and florence (same specs, 128 MiB RAM, waiting for 768 MiB extension each).

• Network connectivity
  • Dexlan 5-port 100base-TX Ethernet switch (port 1=uplink, ports 2-4=squids, port 5=free); all cables straight;
  • names are chloe (212.85.150.132), bleuenn (212.85.150.133), and ennael (212.85.150.131);
  • addresses 212.85.150.130 and 212.85.150.134 may also be used to connect other machines (laptop for maintenance...);
  • IP block : 212.85.150.128/29
    • they now grant us /28 but we haven't yet redone the setup
  • Network : 212.85.150.128
  • Netmask : 255.255.255.248
  • Broadcast : 212.85.150.135
  • Gateway : 212.85.150.129
• null-modem serial cables run from ttyS1 on one machine to ttyS0 on another; setting is 19200 bps;
  • getty running on ttyS0 on each machine, to allow communication using minicom;
  • order (caller to listener) is chloe → ennael → bleuenn → chloe;
  • BIOS, grub and kernel all configured for serial console; BIOS and grub accessible through serial console;
  • as a consequence, any remote rebooting should be made by logging into the preceding machine in the ring and running minicom onto the machine to be rebooted;
• rack space provided by Lost Oasis inside the Telecity colocation in Aubervilliers near the northern city limits of Paris.
• Contact: Med or Submarine.

Network specificities

Obsolete information

The network provider that gives us rack space and bandwidth for free pays a lot for transit to certain destinations. Because of this, they throttle those destinations with a maximum traffic going out of our host, at the level of the last router.

The symptoms for this are long ping times from the affected networks.

It especially looks like anything going through OpenTransit.net is throttled to approximatively 5 Mbits/s.

Normally, this throttling does not concern any network in France. However, on Sunday, January 16 evening, networking problems caused ALL traffic, including French traffic, to go through OpenTransit and thus to be throttled.

As of January 20, all traffic restrictions were lifted. The provider will warn us if we use too much transit.

Status

See Ielo's page for network status. ->traffic graph

From broadband connections in France, at off-peak times, the download speed for large files is around 80 kiB/s when not in cache and 235 kiB/s when in cache.

Countries and content concerned

fr:, en:, commons and upload (all wikis) are cached for Belgium, France, Germany, Luxembourg, Switzerland and the United Kingdom.

See the statistics page.

cachemgr.cgi

There is a cachemgr.cgi available at http://noc.wikimedia.org/~mark/cgi-bin/cachemgr.cgi. The password (for at least the French squids at this moment) can be found in /home/wikipedia/doc/fr-cachemgr-pw.

to investigate

• Selectively remove Squid-2 cache contents: http://www.wa.apana.org.au/~dean/squidpurge/
• change fd limit to 8192 in /usr/local/squid/bin/RunCache, restart squids. Fl squids are sometimes close to their current limit of 4096.

Diagnostic aid

innocence modified the squid error pages to identify which squid was erroring; it was mostly benet on 14 Jan 04; Steps Were Taken

New Donors

As, and even before, we get our protocols entirely worked out for remote squid sites, people are going to volunteer. That link will provide such sites with a place to put their contact and other information, and for us to ask them questions.

See also

• MediaWiki caching -- some cache headers explained
• Multicast HTCP purging -- new method of cache purging

New squid setup

memcheck

Squid has demanding memory access patterns, these memtest options might help to catch more problems before they are real ones: "Also, go to the options, and turn on caching, as well as all memory addresses and tests ... (keys pressed if I can remember, is:

c->1->2->2->3->3->3

should turn on above options for memtest)." (from [2])

Setup

• Add the machine to /home/wikipedia/conf/squid/Makefile, run make
• On the new machine run /home/wikipedia/bin/squidsetup as root after doing a ssh-add. If there are errors fetching the squid stuff, adjust the server to fetch from in the script.
• follow the instructions re crontab
• test the server using telnet or nc
• take ips
• add /usr/local/bin/squid to /etc/rc.local