DNS

Revision as of 15:11, 2 January 2007

This page describes Wikimedia's DNS setup. Wikimedia use two separate kinds of DNS servers, authoritative nameservers (that respond to queries from third party nameservers for our domains) and resolvers (that resolve DNS queries for our own servers)

Authoritative nameservers

In the new DNS setup, Wikimedia have 3 authoritative DNS servers, all running PowerDNS. The three authoritative servers are:

• ns0.wikimedia.org - 66.230.200.16 (browne)
• ns1.wikimedia.org - 211.115.107.190 (amaryllis)
• ns2.wikimedia.org - 145.97.39.158 (pascal)

In addition, 207.142.131.208 is also assigned to browne. This IP address was previously ns1, and is still registered as ns1 in Verio's DNS servers. Since most clients come via Verio, amaryllis is effectively unused. The unambiguous hostname for 207.142.131.208 is ns1-old.wikimedia.org, and the unambiguous hostname for 211.115.107.190 is ns3.wikimedia.org.

The servers are running with two PowerDNS backends each: the Bind backend (which emulates/reads Bind style zonefiles) and Geobackend (which is responsible for geographic DNS). The two backends are overlapping, meaning that for a given query the Bind backend is asked first, and if that declines to answer (because it doesn't have the requested record), the next backend, geobackend will be asked. Therefor, the usual contents of the zones are in regular Bind style zonefiles, and the geodns record rr.wikimedia.org. is added by geobackend.

Browne is the master nameserver, sort of, but the slaves are not using AXFR! Zonefiles and other configuration are replicated through the use of rsync in an update script.

All configuration files can be found in

/usr/local/etc/powerdns/

on all three hosts.

The main PowerDNS configuration file is /usr/local/etc/pdns.conf. Its configuration is documented on PowerDNS.

Additionally, there's a Bind backend configuration file, /usr/local/etc/bind.conf. It's compatible with Bind's own configuration format, but is only used to list the domains which the Bind backend has to serve. (Almost) all other options are ignored. In our setup, bind.conf is autogenerated from the domain templates.

Important differences from the previous DNS setup

• PowerDNS is used exclusively, instead of a mixed Bind/PowerDNS setup
• The wildcard records have been removed. This means that the zonefiles and/or /home/wikipedia/conf/langlist will need to be kept up to date!
• Geographic DNS and static DNS have been integrated into the same nameservers and wikimedia.org zone, which improves query latency because only a single query/response is needed.
• Zonefiles are generated from zone templates.
• No AXFR is being used; zones are replicated through rsync, and SOA serials are purely cosmetic.

Domain templates

Because Wikimedia have a lot of zones that essentially contain the same records (aliases for wikipedia.org and other projects), the old DNS setup used a single zonefile for multiple zones. That has the advantage that just a single change in a zonefile affects many zones. Unfortunately, it doesn't permit the use of $ORIGIN lines in the zonefile. In the new DNS setup, each zone gets its own zonefile, but multiple zonefiles can be generated from a single zone template.

The zone templates are (regular) files in

/usr/local/etc/powerdns/templates/

Each regular file in this directory corresponds to a zone with the same name. Each symbolic link to a regular file in this directory corresponds to a domain alias. So, in this example:

# ls -l templates/mediawiki*
lrwxrwxrwx    1 root     root           13 Jun 19 15:52 templates/mediawiki.com -> mediawiki.org
lrwxrwxrwx    1 root     root           13 Jun 19 15:52 templates/mediawiki.net -> mediawiki.org
-rw-r--r--    1 root     root         1500 Jun 19 15:12 templates/mediawiki.org

...one zone mediawiki.org is listed, with two alias zones, mediawiki.com and mediawiki.net.

Substitution variables

Within the zone template, a few predefined variables can be used, that will be substituted when the actual zonefiles are generated from the template. These variables include:

$zonename 

The actual zone qname (FQDN) of the zonefile to be generated

$serial 

The SOA serial number, derived from the current date and hour in YYYYMMDDHH format

$langlist 

A list of language subdomain CNAMEs, i.e. a list of all language abbreviations for all languages any Wikimedia project has, generated from /home/wikipedia/common/langlist.

gen-zones

The actual zonefiles are generated from the zone templates by a Python script, gen-zones. It simply reads all zone templates from the template directory, applies string substitutions, and writes the result to the

/usr/local/etc/powerdns/zones

directory, where PowerDNS can read them as regular zonefiles.

gen-bind.conf

gen-bind.conf is a Python script, that generates bind.conf by looking at the structure of the files and symlinks in /usr/local/etc/powerdns/templates/. For each regular file in that directory, it creates a corresponding block of zone statements for that zone and the zone aliases. For the example above, that would give rise to:

# mediawiki.org aliases
zone "mediawiki.com"    { type master;  file "mediawiki.com";   };
zone "mediawiki.net"    { type master;  file "mediawiki.net";   };
zone "mediawiki.org"    { type master;  file "mediawiki.org";   };

update

update is a simple shellscript, that automates the invocations of the scripts above. It goes through the following steps:

1. generation of a (new) list of language subdomain CNAMEs from /home/wikipedia/common/langlist
2. generation of the zonefiles from the zone templates
3. generation of bind.conf
4. reload of the local powerdns daemon (on browne)
5. synchronizing the slaves. for each slave:
   1. copying the langlist-cnames, zone templates and geomaps to the slave (using rsync)
   2. generation of the zonefiles on the slave
   3. generation of bind.conf on the slave
   4. reload of the remote powerdns daemon

Basically, update takes care of everything after you've edited the zonefiles.

Geographic DNS

Geographic DNS makes sure that clients end up using the Wikimedia cluster closest to them, by varying DNS responses based on the (country of the) resolver IP querying. Its configuration is still mostly the same as described on PowerDNS.

Geomaps are to be found in

/usr/local/etc/powerdns/geomaps

The IP->Country RBLDNS zonefile is located in

/usr/local/etc/powerdns/zz.countries.nerd.dk.rbldnsd

HOWTO

This section briefly explains how to do the most common DNS changes.

Changing records in a zonefile

1. Edit the template file /usr/local/etc/powerdns/templates/zonename on browne
2. Run /usr/local/etc/powerdns/update

Adding a new zone

1. First, decide if this new zone will use a new, independent zonefile, or will be an alias of another zone

   independent zonefile 

   Create the new zone template as /usr/local/etc/powerdns/templates/zonename (Copy an existing, relatively clean zonefile like wiktionary.org to start with).

   zone alias 

   Make a symbolic link /usr/local/etc/powerdns/templates/aliasname for the alias to the zone being aliased.

2. Run /usr/local/etc/powerdns/update

Removing a zone

1. Remove the corresponding file or symlink /usr/local/etc/powerdns/templates/zonename
2. Run /usr/local/etc/powerdns/update

Adding a new (language) wiki

1. Add the language code to </tt>/home/wikipedia/common/langlist</tt>
2. Run /usr/local/etc/powerdns/update

Resolvers

Each cluster has its own set of recursive resolvers:

pmtpa 

bayle (66.230.200.17) khaldun (66.230.200.18)

knams 

pascal

yaseo 

amaryllis

(expand)

Each resolver runs the PowerDNS recursor, using package pdns-recursor in the Wikimedia APT repository (universe). The configuration file is:

/etc/powerdns/recursor.conf

Some runtime control is available through rec_control, see http://docs.powerdns.com/rec-control.html

The following settings have been modified from the default:

allow-from

Lists the IP ranges that are allowed to query this recursor. 127/8 and internal and external Wikimedia IP ranges are listed.

forward-zones

Forwards queries for the internal zones to the authoritative nameserver(s):

forward-zones= wmnet=66.230.200.16, 10.in-addr.arpa=66.230.200.16

local-address

Comma separated list of IPs on which the recursor should listen for queries. Lists 127.0.0.1 and the (external) service IP, e.g. 66.230.200.17.

setgid, setuid

Change uid/gid to pdns. Unfortunately this account is not created by the Debian package, so use:

# adduser --system --no-create-home --group --disabled-password pdns

Statistics

To setup statistics of the recursor, use the following steps:

1. install rrdtool
2. Copy the directory /usr/local/powerdnsstats off one of the other recursors (bayle, khaldun)
3. mkdir /var/www/pdns as root
4. Run /usr/local/powerdnsstats/create as root
5. Install lighttpd or apache if not already present
6. Set up the following cron job, in /etc/cron.d/pdns-recursor:

*/5 * * * *     root    cd /var/www/pdns/ && /usr/local/powerdnsstats/update && /usr/local/powerdnsstats/makegraphs >/dev/null

External links

• PowerDNS homepage
• PowerDNS manual
• PowerDNS Trac Wiki