Svn.wikimedia.org

Revision as of 12:42, 26 April 2010

Hardware

• mayflower.esams.wikimedia.org
  • SVN+SSH developer access
  • SVN over HTTP public read-only access
  • ViewVC

Related services

• svn-private - houses internal scripts and configurations that can't be publicly viewable

Dependencies:

• lists.wikimedia.org - MediaWiki-CVS commit notification list
• Offsite -- CIA update notifications

Used by:

• codereview-proxy.wikimedia.org - indirection tool to access SVN logs from MediaWiki for CodeReview extension

Server setup

Packages

apt-get install some packages:

 subversion          # SVN core bits
 libsvn-notify-perl  # For commit notification mails
 python-subversion   # Needed for viewvc
 php5-cli            # Used by viewvc to do syntax highlighting
 doxygen             # For doc generation
 apache2             # Web server!
 libapache2-svn      # HTTP SVN module for apache
 libapache2-mod-php5 # For user.php

Custom software

 /usr/local/bin/sillyshell     # restricted shell; in tools/sillyshell in SVN
 /usr/local/bin/ciabot_svn.py  # IRC notifications for SVN, available at cia.cx
 /usr/local/viewvc/*           # ViewVC - viewvc.org

Viewvc

Using 1.0.x branch from their SVN; install it and set up the .conf file.

TODO: clean up the conf file and put it in SVN or this wiki.

# Install viewvc!
svn co http://viewvc.tigris.org/svn/viewvc/branches/1.0.x/ viewvc-1.0
./viewvc-install
# Installation path [/usr/local/viewvc]: 
# update viewvc path in apache config file

Patch for PHP view

To get view links instead of download links for PHP files in the repo:

Index: lib/viewvc.py
===================================================================
--- lib/viewvc.py       (revision 1698)
+++ lib/viewvc.py       (working copy)
@@ -904,7 +904,7 @@
   return mime_type and mime_type in ('image/gif', 'image/jpeg', 'image/png')
 
 def is_text(mime_type):
-  return not mime_type or mime_type[:5] == 'text/'
+  return not mime_type or mime_type[:5] == 'text/' or mime_type in ('application/x-httpd-php')
 
 def is_plain_text(mime_type):
   return not mime_type or mime_type == 'text/plain'

Apache

Stick conf file in /etc/apache2/sites-available and link it into sites-enabled, replacing the default config.

TODO: clean up the conf file and put it in SVN or this wiki.

Doc generation

# Set up an account to run the docs under
adduser --system mwdocs
su -s /bin/bash mwdocs
cd && svn co file:///svnroot/mediawiki/trunk/phase3

# make a log file for the cronjob
cd /var/log
touch mwdocs.log
chown mwdocs mwdocs.log 

# fix path in apache config
# add a cron job:
crontab -e
30 0 * * * (cd /home/mwdocs/phase3 && svn up && php maintenance/mwdocgen.php --all) >> /var/log/mwdocs.log 2>&1

SVN users setup

# The Easy way: 
run add-svn-user on mayflower: 
root@mayflower:~# add-svn-user 
Usage: add-svn-user <username> <keyfile>
The keyfile should contain the user's SSH public key, and will be copied
to their ~/.ssh/authorized_keys.

# Setup...
addgroup svn

# When adding users, put in restricted setup:
/usr/sbin/useradd -G svn -s /usr/local/bin/sillyshell "$name"

# Make cache directory for users.php
mkdir /var/cache/svnusers
chown www-data /var/cache/svnusers

# Change path in users.php from /usr/local/bin/svn to /usr/bin/svn
# Check out thingy
su www-data
svn co file:///svnroot/mediawiki/USERINFO /var/cache/svnusers

Dumps

Currently handled ad-hoc by demon. Stored at /home/demon/svn/* (mounted from storage2 by Tomasz). Can do next incremental dump by running /home/demon/svn-dump-update (this should be crontab'd probably). If you need to do a full dump (we do incremental by default) this takes about an hour or so to dump ~65000 revs.

• Needs more permanent home
• Needs web access (put it on dumps.wikimedia.org or as a link on svn.wikimedia.org?)

Post-commit hooks

SVN Post commit hooks are used for commit e-mail notification, CIA and CodeReview, and can be found in /svnroot/svn tree/hooks/

CodeReview

# Hit CodeReview
wget -q -O /dev/null --post-data="" \
  "http://www.mediawiki.org/w/api.php?action=codeupdate&format=xml&repo=MediaWiki&rev=$REV" &