Varnish

Varnish is a fast caching proxy, and can be used as an alternative to Squid in a reverse caching accelerator setup.

We currently use Varnish for serving bits.wikimedia.org.

HOWTO

See Varnish statistics

Run

# varnishstat

Fred has also written a Ganglia plugin in Python for varnish, which is automatically installed by Puppet. All varnishstat metrics are therefore visible on Ganglia.

Set runtime parameters

Run

# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082

See request logs

As explained below, there are no access logs. However, you can see NCSA style log entries for current requests using:

# varnishncsa

Configuration

Deployment of Varnish is done using Puppet, using class varnish in file manifests/varnish.pp.

We use the varnish package of Ubuntu/Debian, minimum version 2.1.2. Puppet installs this, and replaces its /etc/default/varnish file to set some startup parameters (discussed below). Varnish uses a VCL file (Varnish Configuration Language), a DSL where Varnish behaviour is controlled using subroutines that are compiled into C and executed during each request. The Wikimedia VCL file is

/etc/varnish/wikimedia.vcl

Like with Squid, special sysctl settings are installed by Puppet to tune the system for high HTTP traffic performance.

Varnish does not use log files, but instead writes detailed information about its operations to a SHM ring buffer of a fixed size. Any interested programs can just read along and produce statistics or log output without it slowing down the Varnish daemon itself. The SHM file is mlocked in memory, but Linux insists on writing its buffers to disk anyway - therefore Puppet mounts the directory /var/lib/ganglia into a 150M sized tmpfs filesystem to avoid this.

Startup parameters

The following parameters have been changed from the defaults. Most of these are set in the file /etc/default/varnish.

• NFILES=500000

For ulimit -n, the number of files/sockets/file descriptors Varnish can open

• MEMLOCK=90000

For ulimit -l, so Varnish can lock the entire SHM log buffer (default 80M) into memory.

• ulimit -s 128

To reduce the VSIZE with many Varnish threads

• VARNISH_VCL_CONF=/etc/varnish/wikimedia.vcl

Points to the Wikimedia specific VCL file

• VARNISH_LISTEN_ADDRESS=
• VARNISH_LISTEN_PORT=80

Bind to TCP port 80 on all IPs

• VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
• VARNISH_ADMIN_LISTEN_PORT=6082

Listening socket for the administrative interface

• VARNISH_MIN_THREADS=500
• VARNISH_MAX_THREADS=8000

The minimum and maximum amounts of threads Varnish will keep around for requests, per thread pool.

• VARNISH_STORAGE="malloc,1G"

For bits, which has a small content set, we want to keep everything in memory.

• EXTRA_OPTS="-p thread_pools=8 -p thread_pool_add_delay=1 -p send_timeout=30 -p listen_depth=4096"

Extra runtime parameters, explained below:

• thread_pools=8

One thread pool per CPU core; this reduces mutex contention.

• thread_pool_add_delay

Create more threads quickly when needed

• send_timeout=3

Keep the amount of open connections low, and close idle connections quickly.

• listen_depth=4096

Allow many new connections in the accept() queue, before Varnish can open them.

You also may want to read Bits varnish testing instead, for Domas his findings during a pilot project.

Things that need special consideration

• HTCP purging
• Immediate purging of cache objects (nuke?)
• Header normalization (Host, Accept-Encoding...)
• Two-layer setup (CARP style)
• Compatible logging
• Request stats

Would be nice

• SSL
• IPv6

Many of these are probably already taken care of by our friends at Wikia, and therefore possibly also within Varnish itself...

External links

• Varnish main web site
• OSCON presentation about Varnish at Wikia by Artur Bergman, containing useful information about performance tuning and some neat features as well.