Svn.wikimedia.org

Latest revision as of 00:46, 10 August 2012

svn.wikimedia.org.

[edit] Hardware

• formey
  • SVN+SSH developer access
  • SVN over HTTP public read-only access
  • ViewVC

[edit] Related services

• svn-private - houses internal scripts and configurations that can't be publicly viewable
• codereview-proxy.wikimedia.org

Dependencies:

• lists.wikimedia.org - MediaWiki-CVS commit notification list
• Offsite -- CIA update notifications

Used by:

• codereview-proxy.wikimedia.org - indirection tool to access SVN logs from MediaWiki for CodeReview extension

[edit] Server setup

[edit] Packages

apt-get install some packages:

 subversion          # SVN core bits
 libsvn-notify-perl  # For commit notification mails
 viewvc              # For repository view
 python-subversion   # Needed for viewvc
 python-pygments     # Used by viewvc for syntax highlighting
 php5-cli            # Used by svnusers.php
 doxygen             # For doc generation
 apache2             # Web server!
 libapache2-svn      # HTTP SVN module for apache
 libapache2-mod-php5 # For user.php

[edit] Custom software

 /usr/local/bin/sillyshell     # restricted shell; in tools/sillyshell in SVN
 /usr/local/bin/ciabot_svn.py  # IRC notifications for SVN, available at cia.cx
 /usr/local/viewvc/*           # ViewVC - viewvc.org

[edit] Doc generation

Handled by puppet.

# Set up an account to run the docs under
adduser --system mwdocs
su -s /bin/bash mwdocs
cd && svn co file:///svnroot/mediawiki/trunk/phase3

# make a log file for the cronjob
cd /var/log
touch mwdocs.log
chown mwdocs mwdocs.log 

# fix path in apache config
# add a cron job:
crontab -e
30 0 * * * (cd /home/mwdocs/phase3 && svn up && php maintenance/mwdocgen.php --all) >> /var/log/mwdocs.log 2>&1

[edit] SVN users setup

Note: By default most LDAP commands try to run as the "scriptuser", which requires root privileges. In general most of these should run as root anyway. You'll need to use sudo to run these commands if you are not root.

[edit] Add users

Run add-ldap-user on formey:

Usage: add-ldap-user [options] <username> <key(s)>
example: add-ldap-user laner http://ryandlane.com/static/pubkey.key

Options:
  -h, --help            show this help message and exit
  -s, --self            Use your credentials, rather than the proxyagent's
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials, rather than
                        your own
  --shell=LOGINSHELL    The user's shell (default: /usr/bin/sillyshell)
  --gid=GIDNUMBER       The user's gid (default: 500)
  --uid=UIDNUMBER       The user's uid (default: next available uid)
  --home=HOMEDIRECTORY  The user's home directory (default /home/username)
  --cn=CN               The user's CN (default: firstname + lastname, or
                        username, if first/last not set)
  --firstname=GIVENNAME
                        The user's first name (default: username)
  --lastname=SN         The user's last name (default: username)

The <key(s)> should be either a static URL or a local file.

[edit] Modify users

Run modify-ldap-user on formey:

Usage: modify-ldap-user [options] <username> [--rename <newusername>]
example: modify-ldap-user --replacekeys=http://ryandlane.com/static/pubkey.key laner

Options:
  -h, --help            show this help message and exit
  -s, --self            Use your credentials, rather than the proxyagent's
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials, rather than
                        your own
  --shell=LOGINSHELL    Set the user's shell
  --gid=GIDNUMBER       Set the user's gid
  --uid=UIDNUMBER       Set the user's uid
  --cn=CN               Set the user's CN
  --firstname=GIVENNAME
                        Set the user's first name
  --lastname=SN         Set the user's last name
  --replacekeys=REPLACEKEYLOCATION
                        Replaces all of the user's keys
  --addkeys=ADDKEYLOCATION
                        Adds keys to the user's entry
  --rename              Rename the user

[edit] Delete users

It is likely this should never be done. There are probably better ways to deny access. Removing a user's SSH keys should deny access everywhere. However, if you really want to do this, you can use delete-ldap-user:

Usage: delete-ldap-user [options] <username>

Options:
  -h, --help            show this help message and exit
  -s, --self            Use your credentials
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  --scriptuser          Use the scriptusers' credentials
  -m, --directorymanager
                        Use the Directory Manager's credentials, rather than
                        your own
  --no-delete-home      Don't delete the user's home directory

[edit] Checking LDAP information

You can check entries by using ldaplist, which is slightly easier to use than ldapsearch:

Usage: ldaplist [options] [database] [object-name]

example: ldaplist -l passwd ldap_user

Options:
  --help                show this help message and exit
  -s, --self            Use your credentials
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials
  --scriptuser          Use the scriptusers' credentials
  -v, --verbose         Show the database and search filter used for this
                        search
  -l, --longlisting     List all the attributes for each  entry  matching  the
                        search  criteria.  By default, ldaplist lists only the
                        Distiguished Name of the entries found.
  -h                    Show available databases to search
  -d, --showdatabase    Show the base dn being used for this database
  -a SHOWATTRIBUTES, --showattributes=SHOWATTRIBUTES
                        Show the given attributes
  -r, --recursive       Recurse netgroups
  --like                Search for objects that equal or sound like [object-
                        name]

For instance, to check a user's account:

ldaplist -l passwd <username>

To find a group and list its membership:

ldaplist -l group <some group name>

[edit] Access Control

By default, users have no access in /svnroot/mediawiki/conf/authz. Users can be added to the core, and mediawiki groups using svn-group; mediawiki is for extensions and tools access, whereas core is for all other access. Most people with core will also need mediawiki.

Using svn-group:

Usage: /usr/local/bin/svn-group {add|delete} <user> <group>
Example: /usr/local/bin/svn-group add laner core

Groups:

• mediawiki - only access to extensions, tools and branches
• core - access to core mediawiki
• wmf-deployment - access to deployment branch

When you give someone commit access, link their mediawiki.org username to their commit username by going to https://www.mediawiki.org/wiki/Special:Code/MediaWiki/author/commitname and using the "link this user" link. Then, give the mediawiki.org username "coder" rights via https://www.mediawiki.org/wiki/Special:UserRights/username. Send an email to wikitech-l announcing the new committer (ok to do this in 1 daily batch). Also add the committer to the mw:Developers page.

To see which SVN groups a user is currently in, look in /svnroot/mediawiki/conf/authz.

All SVN users have access to the pywikipediabot project, even those who aren't in /svnroot/mediawiki/conf/authz.

[edit] Troubleshooting

[edit] Bad ssh keys

[12:53]  <Ryan_Lane> I think the way I'm parsing keys is occasionally screwed up
[12:55]  <Ryan_Lane> ^demon: if it imported the keys correctly, there should be a sshPublicKey attribute for every key
[12:56]  <Ryan_Lane> when I added rfaulk, his keys were all jacked up
[12:56]  <Ryan_Lane> with like 20 sshPublicKey attributes, each having one letter :)
[12:57]  <Ryan_Lane> ^demon: if you ever run into that problem, download the key from the URL, and use modify-ldap-user --replacekeys=<filename>
[12:57]  <Ryan_Lane> it always seems to work from a file

[edit] Dumps

Dumps are handled daily at 18:00 UTC and offloaded to Tridge via Amanda. Cron runs daily /usr/local/bin/svndump.php. Sunday performs a full dump, with daily incrementals. The previous day's dump is cleared before starting each day's dump.

[edit] Post-commit hooks

SVN Post commit hooks are used for commit e-mail notification, CIA and CodeReview, and can be found in /svnroot/svn tree/hooks/

[edit] CodeReview

# Hit CodeReview
wget -q -O /dev/null --post-data="" \
  "http://www.mediawiki.org/w/api.php?action=codeupdate&format=xml&repo=MediaWiki&rev=$REV" &

[edit] See also

• http://svn.wikimedia.org
• codereview-proxy.wikimedia.org
• svn-private