Puppet

Revision as of 16:05, 18 January 2010

puppet is the main configuration management tool to be used on the Wikimedia clusters.

puppetd is the client daemon that runs on all servers, and manages machines with configuration information gathered from puppetmasterd, running on machine sockpuppet.pmtpa.wmnet.

puppetd

To install puppet on a single machine, simply run

# apt-get install puppet

On Solaris, the installation instructions for the Blastwave packages seem to work.

Communication with the puppetmaster server is over encrypted SSL and with signed certificates. To sign the certificate of the newly installed machine on the puppetmaster server, log in on sockpuppet.pmtpa.wmnet and run:

# puppetca -s clienthostname

To check the list of outstanding, unsigned certificates, use:

# puppetca -l

Reinstalls

When a server gets reinstalled, the existing certs/keys on the puppetmaster will not match the freshly generated keys on the client, and puppet will not work.

Before a server runs puppet for the first time (again), on the puppetmaster host, the following command should be run to erase all history of a server:

# puppetca --clean clienthostname

However, if this is done after puppetd has already run and therefore has already generated new keys, this is not sufficient. To fix this situation on the client, use the following command to erase the newly generated keys/certificates:

# find /var/lib/puppet -name "$(hostname -f)*" -exec rm -f {} \;

Puppetmaster

The puppetmaster server in pmtpa is sockpuppet.pmtpa.wmnet.

Installation

Simply use the (backported) puppetmaster Ubuntu package:

# apt-get install puppetmaster

Configuration

The default configuration is very usable, but we've made some tweaks here and there.

See /etc/puppet/site.pp for the basics. Puppet currently pushes out crontabs for the image scalers, ganglia binaries and conf files on on hosts, and syncs user information including ssh keys on all hosts. It will reread its conf instantly. Changes to any given host get pushed out every 30 minutes, but puppet is continually updating some host or other. See syslog on sockpuppet for details.

MD5 is broken, use SHA1 for signing certificates:

ca_md=sha1

We use storeconfigs so hosts can exchange configuration (e.g. SSH host keys). To enable this, configure:

storeconfigs=true
dbadapter=sqlite3
dblocation=$vardir/clientconfigs/clientconfigs.sqlite3

Packages rails, sqlite3, libsqlite3-ruby need to be installed. The directory /var/lib/puppet/clientconfigs should be created and owned by user/group puppet.

Making changes

We have a private svn for changes. Here's what you do (on sockpuppet):

cd /root/puppet
svn up
edit
svn commit
( cd /etc/puppet/files; svn up; cd /var/lib/puppet/templates; svn up; cd /etc/puppet/manifests/; svn up; touch /etc/puppet/manifests/site.pp )

Or Else!

You can syntax check your changes by

# puppet --parseonly filename-here

Noop test run

You can do a dry run of your changes using:

# puppetd --noop --test --debug

This will give you (among other things) a list of all the changes it would make.

Trigger a run

Just run:

# puppetd --test

Todo

• More secure certificate signing
• Better, more automated version control
• Better tools for adding/maintaining node definitions

More information

• language description and tutorial: [1]
• general puppet documentation: [2]