Svn.wikimedia.org

Latest revision as of 00:46, 10 August 2012

svn.wikimedia.org.

[edit] Hardware

formey
- SVN+SSH developer access
- SVN over HTTP public read-only access
- ViewVC

[edit] Related services

svn-private - houses internal scripts and configurations that can't be publicly viewable
codereview-proxy.wikimedia.org

Dependencies:

lists.wikimedia.org - MediaWiki-CVS commit notification list
Offsite -- CIA update notifications

Used by:

codereview-proxy.wikimedia.org - indirection tool to access SVN logs from MediaWiki for CodeReview extension

[edit] Server setup

[edit] Packages

apt-get install some packages:

 subversion          # SVN core bits
 libsvn-notify-perl  # For commit notification mails
 viewvc              # For repository view
 python-subversion   # Needed for viewvc
 python-pygments     # Used by viewvc for syntax highlighting
 php5-cli            # Used by svnusers.php
 doxygen             # For doc generation
 apache2             # Web server!
 libapache2-svn      # HTTP SVN module for apache
 libapache2-mod-php5 # For user.php

[edit] Custom software

 /usr/local/bin/sillyshell     # restricted shell; in tools/sillyshell in SVN
 /usr/local/bin/ciabot_svn.py  # IRC notifications for SVN, available at cia.cx
 /usr/local/viewvc/*           # ViewVC - viewvc.org

[edit] Doc generation

Handled by puppet.

# Set up an account to run the docs under
adduser --system mwdocs
su -s /bin/bash mwdocs
cd && svn co file:///svnroot/mediawiki/trunk/phase3

# make a log file for the cronjob
cd /var/log
touch mwdocs.log
chown mwdocs mwdocs.log

# fix path in apache config
# add a cron job:
crontab -e
30 0 * * * (cd /home/mwdocs/phase3 && svn up && php maintenance/mwdocgen.php --all) >> /var/log/mwdocs.log 2>&1

[edit] SVN users setup

Note: By default most LDAP commands try to run as the "scriptuser", which requires root privileges. In general most of these should run as root anyway. You'll need to use sudo to run these commands if you are not root.

[edit] Add users

Run add-ldap-user on formey:

Usage: add-ldap-user [options] <username> <key(s)>
example: add-ldap-user laner http://ryandlane.com/static/pubkey.key

Options:
  -h, --help            show this help message and exit
  -s, --self            Use your credentials, rather than the proxyagent's
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials, rather than
                        your own
  --shell=LOGINSHELL    The user's shell (default: /usr/bin/sillyshell)
  --gid=GIDNUMBER       The user's gid (default: 500)
  --uid=UIDNUMBER       The user's uid (default: next available uid)
  --home=HOMEDIRECTORY  The user's home directory (default /home/username)
  --cn=CN               The user's CN (default: firstname + lastname, or
                        username, if first/last not set)
  --firstname=GIVENNAME
                        The user's first name (default: username)
  --lastname=SN         The user's last name (default: username)

The <key(s)> should be either a static URL or a local file.

[edit] Modify users

Run modify-ldap-user on formey:

Usage: modify-ldap-user [options] <username> [--rename <newusername>]
example: modify-ldap-user --replacekeys=http://ryandlane.com/static/pubkey.key laner

Options:
  -h, --help            show this help message and exit
  -s, --self            Use your credentials, rather than the proxyagent's
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials, rather than
                        your own
  --shell=LOGINSHELL    Set the user's shell
  --gid=GIDNUMBER       Set the user's gid
  --uid=UIDNUMBER       Set the user's uid
  --cn=CN               Set the user's CN
  --firstname=GIVENNAME
                        Set the user's first name
  --lastname=SN         Set the user's last name
  --replacekeys=REPLACEKEYLOCATION
                        Replaces all of the user's keys
  --addkeys=ADDKEYLOCATION
                        Adds keys to the user's entry
  --rename              Rename the user

[edit] Delete users

It is likely this should never be done. There are probably better ways to deny access. Removing a user's SSH keys should deny access everywhere. However, if you really want to do this, you can use delete-ldap-user:

Usage: delete-ldap-user [options] <username>

Options:
  -h, --help            show this help message and exit
  -s, --self            Use your credentials
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  --scriptuser          Use the scriptusers' credentials
  -m, --directorymanager
                        Use the Directory Manager's credentials, rather than
                        your own
  --no-delete-home      Don't delete the user's home directory

[edit] Checking LDAP information

You can check entries by using ldaplist, which is slightly easier to use than ldapsearch:

Usage: ldaplist [options] [database] [object-name]

example: ldaplist -l passwd ldap_user

Options:
  --help                show this help message and exit
  -s, --self            Use your credentials
  -D BINDAS, --bindas=BINDAS
                        Specify user to bind as
  -m, --directorymanager
                        Use the Directory Manager's credentials
  --scriptuser          Use the scriptusers' credentials
  -v, --verbose         Show the database and search filter used for this
                        search
  -l, --longlisting     List all the attributes for each  entry  matching  the
                        search  criteria.  By default, ldaplist lists only the
                        Distiguished Name of the entries found.
  -h                    Show available databases to search
  -d, --showdatabase    Show the base dn being used for this database
  -a SHOWATTRIBUTES, --showattributes=SHOWATTRIBUTES
                        Show the given attributes
  -r, --recursive       Recurse netgroups
  --like                Search for objects that equal or sound like [object-
                        name]

For instance, to check a user's account:

ldaplist -l passwd <username>

To find a group and list its membership:

ldaplist -l group <some group name>

[edit] Access Control

By default, users have no access in /svnroot/mediawiki/conf/authz. Users can be added to the core, and mediawiki groups using svn-group; mediawiki is for extensions and tools access, whereas core is for all other access. Most people with core will also need mediawiki.

Using svn-group:

Usage: /usr/local/bin/svn-group {add|delete} <user> <group>
Example: /usr/local/bin/svn-group add laner core

Groups:

mediawiki - only access to extensions, tools and branches
core - access to core mediawiki
wmf-deployment - access to deployment branch

When you give someone commit access, link their mediawiki.org username to their commit username by going to https://www.mediawiki.org/wiki/Special:Code/MediaWiki/author/commitname and using the "link this user" link. Then, give the mediawiki.org username "coder" rights via https://www.mediawiki.org/wiki/Special:UserRights/username. Send an email to wikitech-l announcing the new committer (ok to do this in 1 daily batch). Also add the committer to the mw:Developers page.

To see which SVN groups a user is currently in, look in /svnroot/mediawiki/conf/authz.

All SVN users have access to the pywikipediabot project, even those who aren't in /svnroot/mediawiki/conf/authz.

[edit] Troubleshooting

[edit] Bad ssh keys

[12:53]  <Ryan_Lane> I think the way I'm parsing keys is occasionally screwed up
[12:55]  <Ryan_Lane> ^demon: if it imported the keys correctly, there should be a sshPublicKey attribute for every key
[12:56]  <Ryan_Lane> when I added rfaulk, his keys were all jacked up
[12:56]  <Ryan_Lane> with like 20 sshPublicKey attributes, each having one letter :)
[12:57]  <Ryan_Lane> ^demon: if you ever run into that problem, download the key from the URL, and use modify-ldap-user --replacekeys=<filename>
[12:57]  <Ryan_Lane> it always seems to work from a file

[edit] Dumps

Dumps are handled daily at 18:00 UTC and offloaded to Tridge via Amanda. Cron runs daily /usr/local/bin/svndump.php. Sunday performs a full dump, with daily incrementals. The previous day's dump is cleared before starting each day's dump.

[edit] Post-commit hooks

SVN Post commit hooks are used for commit e-mail notification, CIA and CodeReview, and can be found in /svnroot/svn tree/hooks/

[edit] CodeReview

# Hit CodeReview
wget -q -O /dev/null --post-data="" \
  "http://www.mediawiki.org/w/api.php?action=codeupdate&format=xml&repo=MediaWiki&rev=$REV" &

[edit] See also

Svn.wikimedia.org

Latest revision as of 00:46, 10 August 2012

Contents

[edit] Hardware

[edit] Related services

[edit] Server setup

[edit] Packages

[edit] Custom software

[edit] Doc generation

[edit] SVN users setup

[edit] Add users

[edit] Modify users

[edit] Delete users

[edit] Checking LDAP information

[edit] Access Control

[edit] Troubleshooting

[edit] Bad ssh keys

[edit] Dumps

[edit] Post-commit hooks

[edit] CodeReview

[edit] See also

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Ops documentation

Wiki

Toolbox

@@ Line 1: / Line 1: @@
+'''svn.wikimedia.org'''.
 == Hardware ==
-* [[mayflower]].esams.wikimedia.org
+* [[formey]]
 ** SVN+SSH developer access
 ** SVN over HTTP public read-only access
@@ Line 7: / Line 9: @@
 == Related services ==
 * [[svn-private]] - houses internal scripts and configurations that can't be publicly viewable
+* [[codereview-proxy.wikimedia.org]]
 Dependencies:
 * [[lists.wikimedia.org]] - MediaWiki-CVS commit notification list
-* Offsite -- [[CIA]] update notifications
+* Offsite -- [http://cia.vc CIA] update notifications
 Used by:
@@ Line 25: / Line 27: @@
    subversion          # SVN core bits
    libsvn-notify-perl  # For commit notification mails
+  viewvc              # For repository view
    python-subversion   # Needed for viewvc
-   php5-cli            # Used by viewvc to do syntax highlighting
+   python-pygments     # Used by viewvc for syntax highlighting
+  php5-cli            # Used by svnusers.php
    doxygen             # For doc generation
    apache2             # Web server!
@@ Line 37: / Line 41: @@
    /usr/local/bin/ciabot_svn.py  # IRC notifications for SVN, available at cia.cx
    /usr/local/viewvc/*           # ViewVC - viewvc.org
-=== Viewvc ===
-Using 1.0.x branch from their SVN; install it and set up the .conf file.
-TODO: clean up the conf file and put it in SVN or this wiki.
- # Install viewvc!
- svn co http://viewvc.tigris.org/svn/viewvc/branches/1.0.x/ viewvc-1.0
- ./viewvc-install
- # Installation path [/usr/local/viewvc]:
- # update viewvc path in apache config file
-==== Patch for PHP view ====
-To get view links instead of download links for PHP files in the repo:
-<pre>
-Index: lib/viewvc.py
-===================================================================
---- lib/viewvc.py       (revision 1698)
-+++ lib/viewvc.py       (working copy)
-@@ -904,7 +904,7 @@
-   return mime_type and mime_type in ('image/gif', 'image/jpeg', 'image/png')
- def is_text(mime_type):
--  return not mime_type or mime_type[:5] == 'text/'
-+  return not mime_type or mime_type[:5] == 'text/' or mime_type in ('application/x-httpd-php')
- def is_plain_text(mime_type):
-   return not mime_type or mime_type == 'text/plain'
-</pre>
-=== Apache ===
-Stick conf file in /etc/apache2/sites-available and link it into sites-enabled, replacing the default config.
-TODO: clean up the conf file and put it in SVN or this wiki.
 === Doc generation ===
+Handled by puppet.
   # Set up an account to run the docs under
@@ Line 97: / Line 64: @@
 === SVN users setup ===
- # The Easy way:
+'''Note:''' By default most LDAP commands try to run as the "scriptuser", which requires root privileges. In general most of these should run as root anyway. You'll need to use sudo to run these commands if you are not root.
- run add-svn-user on mayflower:
-  root@mayflower:~# add-svn-user
+==== Add users ====
-  Usage: add-svn-user <username> <keyfile>
-  The keyfile should contain the user's SSH public key, and will be copied
+Run add-ldap-user on formey:
-  to their ~/.ssh/authorized_keys.
+ Usage: add-ldap-user [options] <username> <key(s)>
+  example: add-ldap-user laner http://ryandlane.com/static/pubkey.key
+ Options:
+   -h, --help            show this help message and exit
+   -s, --self            Use your credentials, rather than the proxyagent's
+   -D BINDAS, --bindas=BINDAS
+                         Specify user to bind as
+   -m, --directorymanager
+                         Use the Directory Manager's credentials, rather than
+                         your own
+   --shell=LOGINSHELL    The user's shell (default: /usr/bin/sillyshell)
+   --gid=GIDNUMBER       The user's gid (default: 500)
+   --uid=UIDNUMBER       The user's uid (default: next available uid)
+   --home=HOMEDIRECTORY  The user's home directory (default /home/username)
+   --cn=CN               The user's CN (default: firstname + lastname, or
+                         username, if first/last not set)
+   --firstname=GIVENNAME
+                         The user's first name (default: username)
+   --lastname=SN         The user's last name (default: username)
+The <key(s)> should be either a static URL or a local file.
+==== Modify users ====
+Run modify-ldap-user on formey:
+ Usage: modify-ldap-user [options] <username> [--rename <newusername>]
+  example: modify-ldap-user --replacekeys=http://ryandlane.com/static/pubkey.key laner
+ Options:
+   -h, --help            show this help message and exit
+   -s, --self            Use your credentials, rather than the proxyagent's
+   -D BINDAS, --bindas=BINDAS
+                         Specify user to bind as
+   -m, --directorymanager
+                         Use the Directory Manager's credentials, rather than
+                         your own
+   --shell=LOGINSHELL    Set the user's shell
+   --gid=GIDNUMBER       Set the user's gid
+   --uid=UIDNUMBER       Set the user's uid
+   --cn=CN               Set the user's CN
+   --firstname=GIVENNAME
+                         Set the user's first name
+   --lastname=SN         Set the user's last name
+   --replacekeys=REPLACEKEYLOCATION
+                         Replaces all of the user's keys
+   --addkeys=ADDKEYLOCATION
+                         Adds keys to the user's entry
+   --rename              Rename the user
+==== Delete users ====
+It is likely this should never be done. There are probably better ways to deny access. Removing a user's SSH keys should deny access everywhere. However, if you really want to do this, you can use delete-ldap-user:
+ Usage: delete-ldap-user [options] <username>
+ Options:
+   -h, --help            show this help message and exit
+   -s, --self            Use your credentials
+   -D BINDAS, --bindas=BINDAS
+                         Specify user to bind as
+   --scriptuser          Use the scriptusers' credentials
+   -m, --directorymanager
+                         Use the Directory Manager's credentials, rather than
+                         your own
+   --no-delete-home      Don't delete the user's home directory
+==== Checking LDAP information ====
+You can check entries by using ldaplist, which is slightly easier to use than ldapsearch:
+{{:ldaplist}}
+==== Access Control ====
+By default, users have no access in <code>/svnroot/mediawiki/conf/authz</code>. Users can be added to the '''core''', and '''mediawiki''' groups using '''svn-group'''; '''mediawiki''' is for extensions and tools access, whereas '''core''' is for all other access. Most people with '''core''' will also need '''mediawiki.'''
+Using svn-group:
+  Usage: /usr/local/bin/svn-group {add|delete} <user> <group>
+ Example: /usr/local/bin/svn-group add laner core
+Groups:
+* mediawiki - only access to extensions, tools and branches
+* core - access to core mediawiki
+* wmf-deployment - access to deployment branch
+When you give someone commit access, link their mediawiki.org username to their commit username by going to <code>https://www.mediawiki.org/wiki/Special:Code/MediaWiki/author/commitname</code> and using the "link this user" link.  Then, give the mediawiki.org username "coder" rights via <code>https://www.mediawiki.org/wiki/Special:UserRights/username</code>.  Send an email to wikitech-l announcing the new committer (ok to do this in 1 daily batch).  Also add the committer to the [[mw:Developers]] page.
+To see which SVN groups a user is currently in, look in <code>/svnroot/mediawiki/conf/authz</code>.
- # Setup...
+All SVN users have access to the pywikipediabot project, even those who aren't in <code>/svnroot/mediawiki/conf/authz</code>.
- addgroup svn
- # When adding users, put in restricted setup:
+==== Troubleshooting ====
- /usr/sbin/useradd -G svn -s /usr/local/bin/sillyshell "$name"
- # Make cache directory for users.php
+===== Bad ssh keys =====
- mkdir /var/cache/svnusers
- chown www-data /var/cache/svnusers
-  # Change path in users.php from /usr/local/bin/svn to /usr/bin/svn
+  [12:53]  <Ryan_Lane> I think the way I'm parsing keys is occasionally screwed up
-  # Check out thingy
+  [12:55]  <Ryan_Lane> ^demon: if it imported the keys correctly, there should be a sshPublicKey attribute for every key
-  su www-data
+  [12:56]  <Ryan_Lane> when I added rfaulk, his keys were all jacked up
-  svn co file:///svnroot/mediawiki/USERINFO /var/cache/svnusers
+ [12:56]  <Ryan_Lane> with like 20 sshPublicKey attributes, each having one letter :)
+ [12:57]  <Ryan_Lane> ^demon: if you ever run into that problem, download the key from the URL, and use modify-ldap-user --replacekeys=<filename>
+  [12:57]  <Ryan_Lane> it always seems to work from a file
 === Dumps ===
-Currently handled by [[User:^demon|demon]], runs as a cron on Saturdays at 1AM. Stored at /home/demon/svn/* (mounted from storage2 by Tomasz). Can do next incremental dump by running <tt>/home/demon/svn-dump-update</tt>. If you need to do a full dump (we do incremental by default) this takes about an hour or so to dump ~65000 revs. Available on the web at [http://svn.wikimedia.org/dumps svn.wikimedia.org/dumps]
+Dumps are handled daily at 18:00 UTC and offloaded to [[Tridge]] via [[Amanda]]. Cron runs daily /usr/local/bin/svndump.php. Sunday performs a full dump, with daily incrementals. The previous day's dump is cleared before starting each day's dump.
-* Needs more permanent home
 == Post-commit hooks ==
@@ Line 133: / Line 189: @@
 </pre>
+== See also ==
+* http://svn.wikimedia.org
+* [[codereview-proxy.wikimedia.org]]
+* [[svn-private]]
+{{lowercase}}
 [[Category:Services]]