Svn.wikimedia.org

Revision as of 18:54, 29 December 2010

Hardware

• formey.esams.wikimedia.org
  • SVN+SSH developer access
  • SVN over HTTP public read-only access
  • ViewVC

Related services

• svn-private - houses internal scripts and configurations that can't be publicly viewable

Dependencies:

• lists.wikimedia.org - MediaWiki-CVS commit notification list
• Offsite -- CIA update notifications

Used by:

• codereview-proxy.wikimedia.org - indirection tool to access SVN logs from MediaWiki for CodeReview extension

Server setup

Packages

apt-get install some packages:

 subversion          # SVN core bits
 libsvn-notify-perl  # For commit notification mails
 viewvc              # For repository view
 python-subversion   # Needed for viewvc
 python-pygments     # Used by viewvc for syntax highlighting
 php5-cli            # Used by svnusers.php
 doxygen             # For doc generation
 apache2             # Web server!
 libapache2-svn      # HTTP SVN module for apache
 libapache2-mod-php5 # For user.php

Custom software

 /usr/local/bin/sillyshell     # restricted shell; in tools/sillyshell in SVN
 /usr/local/bin/ciabot_svn.py  # IRC notifications for SVN, available at cia.cx
 /usr/local/viewvc/*           # ViewVC - viewvc.org

Doc generation

Handled by puppet.

# Set up an account to run the docs under
adduser --system mwdocs
su -s /bin/bash mwdocs
cd && svn co file:///svnroot/mediawiki/trunk/phase3

# make a log file for the cronjob
cd /var/log
touch mwdocs.log
chown mwdocs mwdocs.log 

# fix path in apache config
# add a cron job:
crontab -e
30 0 * * * (cd /home/mwdocs/phase3 && svn up && php maintenance/mwdocgen.php --all) >> /var/log/mwdocs.log 2>&1

SVN users setup

Note: By default most LDAP commands try to run as the "scriptuser", which requires root privileges. In general most of these should run as root anyway. You'll need to use sudo to run these commands if you are not root.

Add users

Run add-ldap-user on formey:

Usage: add-ldap-user [options] <username> <key(s)>
example: add-ldap-user laner http://ryandlane.com/static/pubkey.key

Options:
  -h, --help            show this help message and exit
  -s, --self            Use your credentials, rather than the proxyagent's
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials, rather than
                        your own
  --shell=LOGINSHELL    The user's shell (default: /usr/bin/sillyshell)
  --gid=GIDNUMBER       The user's gid (default: 500)
  --uid=UIDNUMBER       The user's uid (default: next available uid)
  --home=HOMEDIRECTORY  The user's home directory (default /home/username)
  --cn=CN               The user's CN (default: firstname + lastname, or
                        username, if first/last not set)
  --firstname=GIVENNAME
                        The user's first name (default: username)
  --lastname=SN         The user's last name (default: username)

Modify users

Run modify-ldap-user on formey:

Usage: modify-ldap-user [options] <username> [--rename <newusername>]
example: modify-ldap-user --replacekeys=http://ryandlane.com/static/pubkey.key laner

Options:
  -h, --help            show this help message and exit
  -s, --self            Use your credentials, rather than the proxyagent's
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials, rather than
                        your own
  --shell=LOGINSHELL    Set the user's shell
  --gid=GIDNUMBER       Set the user's gid
  --uid=UIDNUMBER       Set the user's uid
  --cn=CN               Set the user's CN
  --firstname=GIVENNAME
                        Set the user's first name
  --lastname=SN         Set the user's last name
  --replacekeys=REPLACEKEYLOCATION
                        Replaces all of the user's keys
  --addkeys=ADDKEYLOCATION
                        Adds keys to the user's entry
  --rename              Rename the user

Checking LDAP information

You can check entries by using ldaplist, which is slightly easier to use than ldapsearch:

Usage: ldaplist [options] [database] [object-name]

example: ldaplist -l passwd ldap_user

Options:
  --help                show this help message and exit
  -s, --self            Use your credentials
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials
  --scriptuser          Use the scriptusers' credentials
  -v, --verbose         Show the database and search filter used for this
                        search
  -l, --longlisting     List all the attributes for each  entry  matching  the
                        search  criteria.  By default, ldaplist lists only the
                        Distiguished Name of the entries found.
  -h                    Show available databases to search
  -d, --showdatabase    Show the base dn being used for this database
  -a SHOWATTRIBUTES, --showattributes=SHOWATTRIBUTES
                        Show the given attributes
  -r, --recursive       Recurse netgroups
  --like                Search for objects that equal or sound like [object-
                        name]

For instance, to check a user's account:

ldaplist -l passwd <username>

Dumps

Dumps are handled daily at 18:00 UTC and offloaded to Tridge via Amanda. Cron runs daily /usr/local/bin/svndump.php. Sunday performs a full dump, with daily incrementals. The previous day's dump is cleared before starting each day's dump.

Post-commit hooks

SVN Post commit hooks are used for commit e-mail notification, CIA and CodeReview, and can be found in /svnroot/svn tree/hooks/

CodeReview

# Hit CodeReview
wget -q -O /dev/null --post-data="" \
  "http://www.mediawiki.org/w/api.php?action=codeupdate&format=xml&repo=MediaWiki&rev=$REV" &