DNS

Latest revision as of 15:41, 2 January 2013

This page describes Wikimedia's DNS setup. Wikimedia use two separate kinds of DNS servers, authoritative nameservers (that respond to queries from third party nameservers for our domains) and resolvers (that resolve DNS queries for our own servers)

Need to make changes to Wikimedia zones? See HOWTO in this page's TOC.

[edit] Authoritative nameservers

In the new even newer DNS setup, Wikimedia have 3 authoritative DNS servers, all running PowerDNS. The three authoritative servers are:

• ns0.wikimedia.org - 208.80.152.130 (dobson)
• ns1.wikimedia.org - 208.80.152.142 (linne)
• ns2.wikimedia.org - 91.198.174.4 (nescio)

The servers are running with two PowerDNS backends each: the Bind backend (which emulates/reads Bind style zonefiles) and Geobackend (which is responsible for geographic DNS). The two backends are overlapping, meaning that for a given query the Bind backend is asked first, and if that declines to answer (because it doesn't have the requested record), the next backend, geobackend will be asked. Therefor, the usual contents of the zones are in regular Bind style zonefiles, and the geodns record rr.wikimedia.org. is added by geobackend.

Dobson is the master nameserver, sort of, but the slaves are not using AXFR! Zonefiles and other configuration are replicated through the use of rsync in an update script. In case of emergency the servers can be synced from any other as well.

All configuration files can be found in

/etc/powerdns/

on all three hosts.

The main PowerDNS configuration file is /etc/powerdns/pdns.conf. Its configuration is documented on PowerDNS.

Additionally, there's a Bind backend configuration file, /etc/powerdns/bind.conf. It's compatible with Bind's own configuration format, but is only used to list the domains which the Bind backend has to serve. (Almost) all other options are ignored. In our setup, bind.conf is autogenerated from the domain templates.

[edit] Installation steps for a Wikimedia authoritative DNS server

This is now (mostly) managed by Puppet using the dns::auth-server class in dns.pp. However, lily is not (yet) puppet managed so needs to be done manually!

Almost everything is done by the wikimedia-task-dns-auth package. Installing this will pull in PowerDNS and setup the directory structure.

Make sure PowerDNS is not running - we don't want to serve out outdated/missing info.

Then, change the relevant settings in /etc/powerdns/pdns.conf - notably the following settings usually need changing:

• local-address
• query-local-address

These need to be set to the DNS service IP of the host (ns0.wikimedia.org etc.)

• default-soa-name
• geo-soa-values

Need to be set to the correct name server name

And run authdns-update on the master server

Then, bind the service IP in /etc/network/interfaces:

up ip addr add 211.115.107.190/32 dev $IFACE

Then, start PowerDNS.

[edit] Important differences from the original Bind DNS setup

• PowerDNS is used exclusively, instead of a mixed Bind/PowerDNS setup
• The wildcard records have been removed. This means that the zonefiles and/or /home/wikipedia/conf/langlist will need to be kept up to date!
• Geographic DNS and static DNS have been integrated into the same nameservers and wikimedia.org zone, which improves query latency because only a single query/response is needed.
• Zonefiles are generated from zone templates.
• No AXFR is being used; zones are replicated through rsync, and SOA serials are purely cosmetic.

[edit] Domain templates

Because Wikimedia have a lot of zones that essentially contain the same records (aliases for wikipedia.org and other projects), the old DNS setup used a single zonefile for multiple zones. That has the advantage that just a single change in a zonefile affects many zones. Unfortunately, it doesn't permit the use of $ORIGIN lines in the zonefile. In the new DNS setup, each zone gets its own zonefile, but multiple zonefiles can be generated from a single zone template.

The zone templates are (regular) files in

/etc/powerdns/templates/

Each regular file in this directory corresponds to a zone with the same name. Each symbolic link to a regular file in this directory corresponds to a domain alias. So, in this example:

# ls -l templates/mediawiki*
lrwxrwxrwx    1 root     root           13 Jun 19 15:52 templates/mediawiki.com -> mediawiki.org
lrwxrwxrwx    1 root     root           13 Jun 19 15:52 templates/mediawiki.net -> mediawiki.org
-rw-r--r--    1 root     root         1500 Jun 19 15:12 templates/mediawiki.org

...one zone mediawiki.org is listed, with two alias zones, mediawiki.com and mediawiki.net.

[edit] Revision Control

The templates are actaully stored in svn on sockpuppet. If you want to edit them, check out the pdns-templates repo. They are checked out to ns0 (dobson) by the authdns-update script when you need to push changes live.

[edit] Substitution variables

Within the zone template, a few predefined variables can be used, that will be substituted when the actual zonefiles are generated from the template. These variables include:

$zonename 

The actual zone qname (FQDN) of the zonefile to be generated

$serial 

The SOA serial number, derived from the current date and hour in YYYYMMDDHH format

$langlist 

A list of language subdomain CNAMEs, i.e. a list of all language abbreviations for all languages any Wikimedia project has, generated from /home/wikipedia/common/langlist.

[edit] gen-zones

The actual zonefiles are generated from the zone templates by a Python script, gen-zones. It simply reads all zone templates from the template directory, applies string substitutions, and writes the result to the

/etc/powerdns/zones

directory, where PowerDNS can read them as regular zonefiles.

[edit] gen-bind.conf

gen-bind.conf is a Python script, that generates bind.conf by looking at the structure of the files and symlinks in /usr/local/etc/powerdns/templates/. For each regular file in that directory, it creates a corresponding block of zone statements for that zone and the zone aliases. For the example above, that would give rise to:

# mediawiki.org aliases
zone "mediawiki.com"    { type master;  file "mediawiki.com";   };
zone "mediawiki.net"    { type master;  file "mediawiki.net";   };
zone "mediawiki.org"    { type master;  file "mediawiki.org";   };

[edit] gen-langlist-cnames

gen-langlist-cnames is a shell script that takes a list with language codes from /etc/powerdns/langlist and generates appropriate CNAMES out of it. It's called by authdns-update.

[edit] gen-dblist-cnames

gen-dblist-cnames generates CNAMEs for toolserver usage.

[edit] authdns-update

/usr/sbin/authdns-update is a simple shell script, that automates the invocations of the scripts above. It goes through the following steps:

1. export the templates from svn on sockpuppet to /etc/powerdns/templates/
2. generation of a (new) list of language subdomain CNAMEs from (rsynced) /home/wikipedia/common/langlist using gen-langlist-cnames.
3. generation of the zonefiles from the zone templates
4. generation of bind.conf
5. reload of the local powerdns daemon (on ns0)
6. synchronizing the slaves. for each slave:
   1. copying the langlist-cnames, zone templates and geomaps to the slave (using rsync)
   2. generation of the zonefiles on the slave
   3. generation of bind.conf on the slave
   4. reload of the remote powerdns daemon

Basically, authdns-update takes care of everything after you've edited and checked in the zonefiles.

[edit] authdns-local-update

/usr/sbin/authdns-local-update is meant to be used on one of the slaves, for pulling in updates from the master. It can be used to bring a slave back up to date after e.g. downtime or a software install/update. It is also used in this way by puppet, for pulling in updates rather than authdns-update's pull process.

[edit] Geographic DNS

Geographic DNS makes sure that clients end up using the Wikimedia cluster closest to them, by varying DNS responses based on the (country of the) resolver IP querying. Its configuration is still mostly the same as described on PowerDNS.

Geomaps are to be found in

/etc/powerdns/geomaps

However, this is now a symlink to one of several directories containing geomaps, called "scenarios". Several scenarios are provided for certain common cases, like "esams-down".

To see what scenario is currently used, and the available scenarios, use:

# authdns-scenario

To select a new scenario, use:

# authdns-scenario scenario

This changes the symlink /etc/powerdns/geomaps.

The IP->Country RBLDNS zonefile is located in

/etc/powerdns/zz.countries.nerd.dk.rbldnsd

[edit] HOWTO

This section briefly explains how to do the most common DNS changes.

[edit] Change GeoDNS

For example, when a certain cluster is down/unreachable, and you want to move all traffic to the others.

The new setup now uses scenarios for this, which are effectively just directories with sets of geomaps for each common case.

To move all traffic away from e.g. esams when it's down, log in on ns0 (dobson) and do:

# authdns-scenario esams-down
# authdns-update -s "ns2.wikimedia.org"

The latter skips ns2.wikimedia.org (lily.esams) in the update process since it'll likely be unreachable anyway. Also see below.

To see which scenario (default: normal) is currently used, use:

# authdns-scenario

To see which scenarios exist, use:

$ ls /etc/powerdns/scenarios

[edit] Changing records in a zonefile

1. Edit the template file ~/pdns-templates/zonename on sockpuppet
2. Run svn diff > /tmp/dns.diff and get a review
3. Check in your change with a useful comment and your name
4. Run authdns-update on ns0 (dobson) to push the changes live
   • It's normal to see a whole pile of "A /etc/powerdns/templates/<filename>"; that's the svn export.
   • Make sure to forward your ssh key when connecting to dobson; it's required to fetch the zone files from svn.
5. query all three DNS servers to make sure your change has been correctly deployed
   • for example: for i in 0 1 2 ; do dig @ns${i}.wikimedia.org -t any my-changed-record.wikimedia.org ; done

[edit] Adding a new zone

1. First, decide if this new zone will use a new, independent zonefile, or will be an alias of another zone

   independent zonefile 

   Create the new zone template as !/pdns-templates/zonename on sockpuppet (Copy an existing, relatively clean zonefile like wiktionary.org to start with).

   zone alias 

   Make a symbolic link ~/pdns-templates/aliasname for the alias to the zone being aliased.

2. svn add the file to the repository and check in your changes.
3. Run authdns-update
4. query all three ns servers to verify that your change took correctly.

[edit] Removing a zone

1. svn rm the corresponding file or symlink ~/pdns-templates/zonename
2. log into ns0 (dobson) and delete /etc/powerdns/templates/zonename
   • files are deployed from svn with svn export, which won't delete files removed from the repository from the exported location.
3. Run authdns-update

[edit] Adding a new (language) wiki

1. Add the language code to /home/wikipedia/common/langlist
2. Run authdns-update

[edit] If a certain nameserver is unreachable

When a certain nameserver is unreachable, the others can still be updated from any of the other servers, by running authdns-update there. To skip the unreachable server in the update process, use:

# authdns-update -s "server list"

where server list is a space separated list of FQDNs. Do not forget the quotes, the script will only accept one argument behind -s.

If you only need to change the geomaps, e.g. when a certain cluster is unreachable, then it's also possible to run authdns-scenario on each server and restart PowerDNS manually.

[edit] Resolvers

Each cluster has its own set of recursive resolvers:

pmtpa 

dobson (208.80.152.131), mchenry (208.80.152.132)

esams 

lily (91.198.174.6), dobson (208.80.152.131)

Each resolver runs the PowerDNS recursor, using package pdns-recursor in the Wikimedia APT repository (universe). The configuration file is:

/etc/powerdns/recursor.conf

Some runtime control is available through rec_control, see http://docs.powerdns.com/rec-control.html

With puppet, setting variable $dns_recursor_ipaddress to the recursor service ip, and including class dns::recursor suffices to make a server into a PowerDNS recursor.

The following settings have been modified from the default:

[edit] allow-from

Lists the IP ranges that are allowed to query this recursor. 127/8 and internal and external Wikimedia IP ranges are listed.

[edit] forward-zones

Forwards queries for the internal zones to the authoritative nameserver(s):

forward-zones= wmnet=208.80.152.130;208.80.152.174;91.198.174.4, 10.in-addr.arpa=208.80.152.130;208.80.152.174;91.198.174.4

[edit] local-address

Comma separated list of IPs on which the recursor should listen for queries. List the (external) service IP, e.g. 208.80.152.131.

[edit] setgid, setuid

Change uid/gid to pdns. Unfortunately this account is not created by the Debian package, so use:

# adduser --system --no-create-home --group --disabled-password pdns

[edit] Statistics

This is handled by the puppet class dns::recursor::statistics

To setup statistics of the recursor, use the following steps:

1. install rrdtool
2. Copy the directory /usr/local/powerdnsstats off one of the other recursors (dobson, mchenry)
3. Install lighttpd or apache if not already present
4. mkdir /var/www/pdns as root
5. Run cd /var/www/pdns && /usr/local/powerdnsstats/create && wget http://dobson.wikimedia.org/pdns/index.html as root
6. Set up the following cron job, in /etc/cron.d/pdns-recursor:

*/5 * * * *     root    cd /var/www/pdns/ && /usr/local/powerdnsstats/update && /usr/local/powerdnsstats/makegraphs >/dev/null

[edit] IPv6

Besides looking at the IPv6 page, note that requests for upload.knams.wikimedia.org are handled by /usr/local/lib/selective-answer.py on the dns hosts (see trunk/tools/selective-answer/), and that at the moment you have to be listed in /etc/powerdns/participants in order to get the AAAA record back.

[edit] HOWTO

[edit] Remove a record from the DNS resolver caches

If you have just added or updated a DNS record on the authoritative nameservers, it may still be cached on the (unrelated) DNS resolvers used by our servers. To clear a record from the cache, use:

# rec_control wipe-cache hostname

on all the DNS resolvers. This will also clear any negative cache records.

[edit] External links

• PowerDNS homepage
• PowerDNS manual
• PowerDNS Trac Wiki