Squids

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(not so out of date anymore)
(udpated)
Line 1: Line 1:
−
{{fixme|How to update [[squid error pages]]}}
+
There are 6 clusters of squid servers, one upload and one text at each of our three locations: yaseo, knams and pmpta. Each server runs two instances of squid: a frontend squid listening on port 80, and a cache squid listening on port 3128. The purpose of the frontend squid is to distribute load to the cache squids based on URL hash, using the [http://icp.ircache.net/carp.txt CARP] algorithm.
  
−
__TOC__
+
[[LVS]] is used to balance incoming requests between the CARP frontends.
  
−
==Architecture==
+
== Installation ==
−
There are 5 squid farms at the moment:
+
−
* PMTPA text squids, serving all text pages. Uses IP/DNS-based round robin, no load balancer.
+
−
* PMTPA image squids, serving upload.wikimedia.org. Uses [[LVS]] load balancing on avicenna.
+
−
* YASEO text squids, serving all text pages for Asian users. Uses [[LVS]] load balancing on [[yf1018]]
+
−
* KNAMS text squids, serving all text pages for European users. Uses [[LVS]] load balancing on [[pascal]].
+
−
* KNAMS image squids, serving upload.wikimedia.org. Uses [[LVS]] load balancing on [[pascal]].
+
−
* <s>LOPAR text squids, unused at the moment.</s> decomissioned
+
  
−
==Emergency operations==
+
All Wikimedia squids are running ubuntu. All necessary packages for squid can be installed by installing the wikimedia-task-squid package from the Wikimedia [[APT repository]].
−
===After a PMTPA power outage===
+
−
The xx.wikipedia.org switches don't have a load balancer. IP addresses have to be assigned manually. Check /usr/local/dsh/node_groups/squid for available hosts and use nslookup to identify which IPs to assign to them. See [[#IPs of virtual ethernet interfaces|below for details]].
+
  
−
===To switch away from foreign squids===
+
== Configuration ==
−
*Need to be root
+
−
*On zwinger:
+
−
** Edit <tt>/usr/local/etc/powerdns/geomaps/rr.wikimedia.org</tt>. Each non comment line is a country code mapping to a cluster name.(0 is default route, to go to Florida, must be left in).
+
−
** Load root ssh key
+
−
** Run <tt>/usr/local/etc/powerdns/update</tt> to deploy.
+
−
** Wait for DNS propagation time (600s, in /usr/local/etc/pdns.conf geo-ttl=600 . Might reduce it to give faster switch back after problem is over, reducing it can't make the emregency fix work faster though - it's cache time for other DNS servers)
+
−
*Reverse these changes to switch back.
+
  
−
== Squid builds ==
+
Configuration is done by editing the master files in <tt>/home/wikipedia/conf/squid</tt>, then running <tt>make</tt> to rebuild the configuration files, and <tt>./deploy</tt> to deploy them to the remote servers. The configuration files are:
  
−
Currently running:
+
; squid.conf.php : Template file for the cache (backend) instances
−
* squid-2.5.STABLE13-7wm.src.rpm
+
; frontend.conf.php : Template file for the frontend instances
−
* squid-2.6.STABLE3-1wm.src.rpm
+
; text-settings.php : A settings array which applies to text squids. All elements in this array will become available as variables during execution of squid.conf.php and frontend.conf.php. The <tt>settings</tt> array can be used to give server-specific configuration.
−
Multiple binary builds exist for different arches and/or distributions. Can be found under <tt>/home/wikipedia/rpms/squid</tt> and [[Subversion]].
+
; upload-settings.php : Same as text-settings.php but for upload squids
 +
; common-acls.conf : ACL directives used by both text and upload frontends. Use this to block clients from all access.
 +
; upload-acls.conf : ACL directives used by upload frontends. Use this for e.g. image referrer regex blocks.
 +
; text-acls.conf : ACL directives used by text frontends. Use this for e.g. remote loader IP blocks.
 +
; Configuration.php : Contains most of the generator code
 +
; generate.php : the script that the makefile runs
  
−
==IPs of virtual ethernet interfaces on old PMTPA text Squids==
+
Feel free to check in your changes to RCS.
−
* to find out the IPs, use <tt>nslookup rr.pmtpa.wikimedia.org</tt>
+
−
* Used to be assigned at boot time, but this can lead to problems with duplicated IP addresses.
+
−
* Can use script ''takeip'' in /home/wikipedia/bin to take over an IP if a squid goes down.
+
−
* To take down a virtual eth interface, <tt>/sbin/ip addr del ''xxx.xxx.xxx.xxx'' dev eth0</tt>
+
−
* According to dammit, should use 255.255.255.255 for netmask for the IPs of virtual eth interfaces - otherwise there can be routing confusion. takeip does this automatically
+
  
−
;TODO: set up LVS
+
The deployment script has lots of options. Run it with no arguments to get a summary.
−
 
+
−
== Common operations ==
+
−
=== Starting ===
+
−
The Squid RPM has a SysV init script <tt>/etc/init.d/squid</tt> like any proper RPM, start it using
+
−
# /sbin/service squid start
+
−
 
+
−
It's automatically started at boot time, alter this using <tt>/sbin/chkconfig</tt>.
+
−
 
+
−
=== Reloading ===
+
−
# /sbin/service squid restart
+
−
 
+
−
===cachemgr.cgi===
+
−
:''this needs to be updated''
+
−
There is a cachemgr.cgi available at http://noc.wikimedia.org/~mark/cgi-bin/cachemgr.cgi. The password (for at least the French squids at this moment) can be found in <tt>/home/wikipedia/doc/fr-cachemgr-pw</tt>.
+
  
 
== See also ==
 
== See also ==
 
* [[MediaWiki caching]] -- some cache headers explained
 
* [[MediaWiki caching]] -- some cache headers explained
 
* [[Multicast HTCP purging]] -- new method of cache purging
 
* [[Multicast HTCP purging]] -- new method of cache purging
−
* [[French squids]] -- for a documentation on the french cluster
+
* [[Squid logging]]
−
 
+
* [[Squid log format]]
−
== New squid setup ==
+
−
* Install the Squid RPM
+
−
* Adapt LVS
+
−
[[Category:How-To]] [[Category:Software]]
+
−
 
+
−
== Squid 2.6 ==
+
−
Squid 2.6 has been released, with some relevant changes to us, mostly concerning performance and accelerator features. Nearly all Wikimedia specific patches entered the distribution, which is helpful to maintain the RPM.
+
−
 
+
−
However, the configuration file changed in some incompatible ways.
+
−
 
+
−
=== Configuration file changes ===
+
−
The following lines, to set Squid up as an HTTP accelerator, no longer exist:
+
−
httpd_accel_port 80
+
−
httpd_accel_host virtual
+
−
httpd_accel_with_proxy on
+
−
httpd_accel_uses_host_header on
+
−
Instead, this is now indicated with the <tt>vhost</tt> option of <tt>http_port</tt>:
+
−
http_port 80 vhost
+
−
 
+
−
There is a new <tt>htcp-oldsquid</tt> option for <tt>cache_peer</tt>, which should be used for Squids still running version 2.5. The HTCP implementation used is incompatible with the 2.5 implementation.
+
−
 
+
−
HTCP queries and HTCP CLR (purge) packets are now guarded by an access list. To allow these queries for Wikimedia servers, use:
+
−
 
+
−
# HTCP and HTCP CLR access
+
−
htcp_access allow all
+
−
+
−
htcp_clr_access allow tiertwo
+
−
htcp_clr_access deny all
+

Revision as of 08:38, 11 March 2007

There are 6 clusters of squid servers, one upload and one text at each of our three locations: yaseo, knams and pmpta. Each server runs two instances of squid: a frontend squid listening on port 80, and a cache squid listening on port 3128. The purpose of the frontend squid is to distribute load to the cache squids based on URL hash, using the CARP algorithm.

LVS is used to balance incoming requests between the CARP frontends.

Installation

All Wikimedia squids are running ubuntu. All necessary packages for squid can be installed by installing the wikimedia-task-squid package from the Wikimedia APT repository.

Configuration

Configuration is done by editing the master files in /home/wikipedia/conf/squid, then running make to rebuild the configuration files, and ./deploy to deploy them to the remote servers. The configuration files are:

squid.conf.php 
Template file for the cache (backend) instances
frontend.conf.php 
Template file for the frontend instances
text-settings.php 
A settings array which applies to text squids. All elements in this array will become available as variables during execution of squid.conf.php and frontend.conf.php. The settings array can be used to give server-specific configuration.
upload-settings.php 
Same as text-settings.php but for upload squids
common-acls.conf 
ACL directives used by both text and upload frontends. Use this to block clients from all access.
upload-acls.conf 
ACL directives used by upload frontends. Use this for e.g. image referrer regex blocks.
text-acls.conf 
ACL directives used by text frontends. Use this for e.g. remote loader IP blocks.
Configuration.php 
Contains most of the generator code
generate.php 
the script that the makefile runs

Feel free to check in your changes to RCS.

The deployment script has lots of options. Run it with no arguments to get a summary.

See also

Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Squids&oldid=11653"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox