Heterogeneous deployment

Revision as of 18:58, 24 August 2012

See also: "Heterogeneous deployment" page on mediawiki.org

See also: Heterogeneous deployment/old for information about older setup

This document describes the current deployment infrastructure, which allows for different wikis to be running different MediaWiki versions as the same time.

Where things live

common/

• wikiversions.dat: a newline separated list where each row is of the format <wikiname> <version>. Versions must have a "php-" prefix, like "php-1.17". This is here for admins to edit as needed. Note that addwiki.php updates this automatically (rebuilding the cdb too).
• wikiversions.cdb: a cdb database of the wikiversions.dat file. Never edit this directly.

common/multiversion

Note: these files are all checked out from mediawiki/trunk/tools/mwmultiversion/multiversion

• MWMultiVersion.php: Class for determining the database name and version of a wiki. Supports loading from three entry points:
  • Web access of an uploaded file through apache - setSiteInfoForUploadWiki()
  • Web access of a other wiki content through apache - setSiteInfoForWiki()
  • From within a maintenance script - setSiteInfoForMaintenance()
• MWScript.php: Wrapper for running the right version of a maintenance script for a wiki.
• MWVersion.php: Contains two functions. Both locate the path of a MW file for a given wiki, set $IP, and set PHP's working directory to $IP. One functions is for web hits through apache, the other handles maintenance scripts.
• activeMWVersions: This returns a space separated list of items based on the common/wikiversions.dat file. If --home is set then it uses the file on /home instead of /usr. If the --withdb param is set, then each item in the list is the form "version=wiki", otherwise they are of the form "version". The versions do not have a "php-" prefix. This is used for setting bash arrays to iterate over different MW versions.
• checkoutMediaWiki: SVN checks out a new installation from the wmf/ branch. It creates some files and symlinks to common/ that the install will need in order to be used.
• refreshWikiversionsCDB: rebuilds the wikiversions.cdb database from the common/wikiversions.dat file on /home.
• switchAllMediaWikis: switches all wikis from one version to another. It rebuilds the wikiversions.dat file and then rebuilds the wikiversions.cdb file. All effected entries (wikis) will be changed to use the specified version. This does not push (sync) the changes automatically.
• populateWikiversions: rebuilds the wikiversions.dat file (from all.dblist) and then rebuilds the wikiversions.cdb file. ALL entries (wikis) will be changed to use the specified version. At present it's not very useful if you want some wikis to still run the same version. You'll almost never want to run this.

common/live-1.5

This is were the main PHP entries points are. Most files in here just use MWVersion.php to find the right PHP file (in the right MW version directory) and then include it.

common/wmf-config

This is were most of the site configuration lies. Main files include things like:

• CommonSettings.php: Mostly handles shared settings or settings with few exceptions. Also handles the task of getting the InitialiseSettings.php options into globals that can be used for MW, and the caching thereof. Site, language, and DB name globals are extracted from the MWMultiVersion object. A MWMultiVersion object should always be initialized by the time this file is loaded. It also sets the i18n message directory locations for extensions. Other random things are performed here.
• InitialiseSettings.php: Mostly contains MW configuration settings for individual wikis as well as wiki groups (like all "wikibooks" for example).
• extension-list: The list of extensions available for use on the site (though some may not be enabled at the moment). These extensions must be installed on cluster. This list is used to automatically generate the ExtensionsMessages-*.php files needed for localization.

common/php-*

This is where the various MW directories lie.

wikipedia/bin

• RefreshWikiversionsCDB: symlink to the RefreshWikiversionsCDB in multiversion/.
• sync-wikiversions: script to rebuild wikiversions.cdb (by running RefreshWikiversionsCDB), sync wikiversions.dat and wikiversions.cdb, and log the action.

How to do stuff

General remarks on deploying code

Basic common sense

• Be careful. Breaking the site is surprisingly easy!
• If you're deploying code written by someone else, ask them to be around during deployment so they can troubleshoot if necessary.
• Make sure you know about anything hairy, such as additional prerequisites (e.g. schema changes) or potential complications when rolling back.

Deployment Requirements

• Cluster account request through RT; request deployment access (requires manager and/or sr dev approval).
  • If you can ssh into fenari, and ssh into a random srv box (e.g. srv300) from there, you already have this.
• Deployment branch commit access requested by email to RobLa (CC Tim & Chad).
• Some shiny new code with which to terrorize end-users.

Don't leave town

Even if your deploy appears to be working, it's important to be reachable in the hours immediately following your deploy. Ideally, stay online and in IRC channels like #wikimedia-tech for a couple of hours.

If you must go offline, let people know how to reach you (and keep your mobile phone or other communications device on your person). You can use /away messages on IRC, or perhaps send a short email to private-l.

If you are on Wikimedia staff, now might be a great time to check if your contact info is up to date. If you aren't on staff, ask a staffer to add your contact info to that page, under "Important volunteers".

Basic code deployment steps

Step 1a: get the code in the deployment branch

Before you can deploy anything, it has to be in the deployment branch. The deployment branch is in our SVN at /branches/wmf/1.MAJORwmfMINOR where MAJOR and MINOR are numbers that rise over time as we rebranch. At the time of writing, the deployment version number was 1.17wmf1; if you're unsure which of the deployment branches is currently in use, you can look it up here.

To get code into the deployment branch, you have to merge it in (when deploying changes to already-deployed things) or copy it over (when deploying a new extension). This may have been done for you already, but since commit access to the deployment branch is limited, you may have to do it yourself. For details on how to merge things, see this guide on mediawiki.org.

Step 1b: Update make-wmf-branch/default.conf if needed

If you added a new extension to the deployment branch, you should also add it to /trunk/tools/make-wmf-branch/default.conf in SVN (in the $normalExtensions array) so it'll be picked up when the deployment branch is rebranched.

Step 2: get the code on fenari

Once the code is in the deployment branch, you simply run svn up on fenari to get it there:

catrope@fenari:~$ cd /home/wikipedia/common/php/
catrope@fenari:/home/wikipedia/common/php/$ svn up

The code is now automatically running on test.wikipedia.org as well, and can be tested there. If, however, you have svn up'd Javscript/CSS files that do not get loaded with ResourceLoader, you will need to ssh into srv193 (the server that runs testwiki) and run /home/catrope/sync-common before they are live on testwiki.

Step 3: configuration and other prep work

See Change wiki configuration if you need to make site configuration changes.

Step 4: synchronize the changes to the cluster

Small changes: sync individual files

If your change only touches one or a few files, you can sync them individually with sync-file rather than having to run scap. This is preferable because a scap run always shakes the cluster up a bit and takes longer to complete, while a sync-file run is very lightweight. However, sync-file is only capable of synchronizing files wihtin directories that already exist on the cluster, so it won't work with newly added directories. Also, sync-file only synchronizes one file at a time, and creates a log entry each time. Using it repetitively (e.g. with a for loop) to sync multiple files is fine, as long as there's not too many of them (say not more than ~5).

To sync a single file, run sync-file [path to file] [summary]. The summary will be used by logmsgbot to log your sync in #wikimedia-tech, from where it'll go to the server admin log and the identi.ca and Twitter feeds. Typically, you would use the revision ID

• PITFALL: The path argument has to be relative to the common directory, not to the current directory. To preserve your sanity (and tab-completion functionality), I recommend you always cd to common before running sync-file
• PITFALL: If the summary argument contains spaces, you'll have to put it in quotes or only the first word is used. If your summary contains a $, you'll either have to escape it or put your summary in single quotes, to prevent bash's variable expansion from messing it up

When running sync-file, you'll usually see about half a dozen errors from broken servers. We should really fix things some time so this doesn't happen, but in the meantime you can consider this completely normal (sample output below). It usually completes within a few seconds, but in rare cases it may hang on a broken Apache for 1 or 2 minutes.

catrope@fenari:/home/wikipedia/common$ sync-file php/api.php r12345
No syntax errors detected in /home/wikipedia/common/php-1.5/api.php
copying to apaches
srv104: ssh: connect to host srv104 port 22: No route to host
srv125: ssh: connect to host srv125 port 22: No route to host
srv149: ssh: connect to host srv149 port 22: Connection refused
srv169: ssh: connect to host srv169 port 22: Connection refused
srv284: ssh: connect to host srv284 port 22: No route to host
srv145: ssh: connect to host srv145 port 22: Connection timed out
srv206: ssh: connect to host srv206 port 22: Connection timed out
srv266: ssh: connect to host srv266 port 22: Connection timed out

Changes isolated to a directory: sync the directory

You can use sync-dir similar to sync-file, except it works on directories. This has caveats. For example, if you changed localization messages, you will still need to run scap to update the message cache.

More complex changes: sync everything

If you're adding directories, changing many files, or otherwise have a reason why sync-file wouldn't work or would be impractical, you'll have to run scap, which syncs everything and rebuilds caches. scap logs to the server admin log with the revision number you updated to, and reports in #wikimedia-tech (without !log) when it finishes.

awjrichards@fenari:/home/wikipedia/common/php$ scap 'Log message here'
Checking syntax...
Compiling texvc...ok
Copying to fenari...Done.
Updating serialized data files...
Warning: messages are no longer serialized by this makefile.
Updating ExtensionMessages.php...
...snip...

Running scap takes a few minutes. There is usually a load spike and a few hiccups on the cluster immediately after scapping, but that's normal, as long as it doesn't last longer than a few minutes.

Change wiki configuration

See Configuration files for a list of configuration files and their formats

In your own repo via gerrit

If you need someone else to merge and push your change, follow this path.

• for the person making the change
  • in your own checkout of the operations/mediawiki-config project, make the changes you need
  • push the change to gerrit for review
• for the person merging and pushing the change live
  • on fenari, cd to /home/wikipedia/common/
  • run git status and verify there are no pending changes. untracked files are ok.
  • in gerrit, review and merge the change
  • on fenari, run git fetch origin && git diff HEAD origin. Review the diff and make sure it's only the changes that you expect
    • if more than the changes you're making show up, STOP and find out why and make sure they're ok to merge.
  • if it looks good, run git merge origin/master
    • if this creates a merge commit, that means someone has committed a local change without pushing; this isn't the end of the world but it should be fixed, so either fix it yourself or ask for help
  • run sync-file wmf-config/filename 'Summary of change' for each changed file separately.

Change on fenari then commit

If you are going to make the change, review, and push all by yourself, follow this path.

• Edit the file(s) you need to edit.
• cd to /home/wikipedia/common/
• Run git diff wmf-config , which will display a diff of recent changes to the wmf-config directory. Verify that these changes are sane and that there are no traces of someone else messing with the same files
• If everything looks good, run sync-file wmf-config/filename 'Summary of change' for each changed file separately.
• Run git commit wmf-config/filename and add a decent commit summary. Then do git push origin
• Do not forget the git push step above, things will get messed up if you do

Run a maintenance script on a wiki

To enter a maintenance script, use the mwscript command-line utility.

mwscript scriptname.php dbname

The above command will run the script scriptname.php located in /maintenance since no directory was specified. You can also use a path relative to the MediaWiki installation, for example:

mwscript extensions/MyExtension/maintenance/scriptname.php dbname

Internally, this executes /home/wikipedia/common/multiversion/MWScript.php. It will run the version of the maintenance script in /home/wikipedia/common. If you want to run the version under /usr/local/apache/common-local/, or you're on a box without /home mounted, you need to change to /usr/local/apache/common-local/multiversion and run ./MWScript.php instead.

Example:

$ mwscript eval.php enwiki
> print $wgDBname;
enwiki
> print $wgVersion;
1.18wmf1
> exit
$

Run a maintenance script on a group of wikis

If you want to run a maintenance script on all wikis in a .dblist file, you can issue the following command from anywhere:

mwscriptwikiset scriptname.php listname.dblist

This will run the script scriptname.php located in /maintenance since no directory was specified. As with MWScript.php and mwscript, you can also use a script path relative to the MediaWiki installation, for example:

mwscriptwikiset extensions/MyExtension/maintenance/scriptname.php listname.dblist

The .dblist file must be located in /usr/local/apache/common-local.

As with mwscript, mwscriptwikiset will run the "home" version of the maintenance script found in /home.

Switch a wiki to a different MW version

• First make sure that the new MW version exists. If not, see the section of this page concerned with checking out new versions.
• First check if another wiki is already using the version. You can use the command mwversonsinuse --withdb. Note that this will only return *one* wiki using the version, not all of them.
• If no other wiki is already using the MW version:
  • Locate the testwiki entry in wikiversions.dat (on home/) and change it to the new MW version. Make sure it has 'php-' in it.
  • Do cd /php-<MW VERSION>/extensions/TrustedXFF and then run php /home/wikipedia/common/multiversion/MWScript.php extensions/TrustedXFF/generate.php <WIKI NAME>
  • Run scap. This should update ExtensionMessages-*.php for the version and compile binaries for the version, like texvc.
  • Check that testwiki is operational.
• Edit the wikiversions.dat file by finding the wiki's DB name and changing the corresponding version value. Make sure it has 'php-' in it.
• Run sync-wikiversions [DESCRIPTIVE MESSAGE] to rebuild the cdb from the dat file and sync them.

Install a new extension on a wiki

If you're deploying an extension or feature that can be switched off, it's usually best to leave it switched off while you deploy and carefully switch it on after that using a simple configuration change (this is called a dark launch). Even if you do this, you should build any configuration infrastructure (e.g. $wmg variable, adding entry in InitialiseSettings with default false) at this time so all you'll have to do later is flip a switch.

For specific preparations, see the sections below as well as How to do a schema change and How to do a configuration change. Best to perform schema changes before making config changes.

Add a configuration switch for an extension

In /home/wikipedia/common/wmf-config/CommonSettings.php, add:

if ( $wmgEnableMyExtension ) {
	require_once( "$IP/extensions/MyExtension/MyExtension.php" );
	// Set config vars if needed
 
	// If you want to export config vars through InitialiseSettings.php, you need to set $wmgMyExtensionThingy there and do
	#$wgMyExtensionThingy = $wmgMyExtensionThingy;
}

In /home/wikipedia/common/wmf-config/InitialiseSettings.php, add something like:

'wmgEnableMyExtension' => array(
	'default' => false,
	'eswikibooks' => true,
	// etc.
),
// If needed, set $wmgMyExtensionWhatever vars here too

If your extension requires a large-ish amount of configuration, consider putting it in a separate file instead. Currently, AbuseFilter, LiquidThreads and FlaggedRevs do this.

For more documentation on these files and their formats, see Configuration files.

Add new extensions to extension-list

When adding a new extension, you need to add it to the extension list, or its i18n messages won't get picked up. For more information about this setup, see this.

1. cd to /home/wikipedia/common/wmf-config
2. Edit extension-list and add the path to the extension setup file on a line by itself
3. Where X.XX is the current MediaWiki version, run mwscript maintenance/mergeMessageFileList.php --wiki=aawiki --list-file=extension-list --output=ExtensionMessages-X.XX.php and watch the output for errors. Your extension should appear in the output the same way the others do
   • If you get an error, you probably got the path wrong, or the extension setup file is wonky
   • If you get an error, you can also try running /home/catrope/sync-common , then running mergeMessageFileList.php again like above
4. Run svn diff and verify that the changes to ExtensionMessages-X.XX.php make sense. If it does, then svn commit that file.
   • A line like 'myext' => "$IP/extensions/myext/myext.i18n.php", should be added, and there should be no unrelated changes. Check the path to the i18n file for correctness

Deploy the extension

Do what needs to be done to deploy and register changes in these config files. You will need to run scap afterwards (so the config and message cache is updated).

Add to make-wmf-branch

To have your extension added to newly created WMF branches, please add the newly added extension to make-wmf-branch default.conf.

Make a wiki use a special version of an extension

This scenario occurs when we have MyExtension/ and MyExtensionBeta/ in one of the MW deployment directories, say php-X.XX, and we want some wikis using one extension version while other wikis use the other.

• Edit wikiversions.dat by finding the row with [target wikidb] [version] and changing it to [target wikidb] [version] [target wikidb]. This defines the wiki as having a separate i18n cache.
• If this was already in wikiversions.dat:
  • Edit CommonSettings.php to include the desired extension version if $wgDBname equals the target wiki and sync it
• ...otherwise, perform the following steps:
  • Run refreshWikiversionsCDB to rebuild wikiversions.cdb.
  • cd to /home/wikipedia/common/wmf-config
  • Edit CommonSettings.php to include the desired extension version if $wgDBname equals the target wiki
  • Run mwscript mergeMessageFileList.php --wiki=TARGETWIKI --list-file=extension-list --output=ExtensionMessages-MWVERSIONFORTHATWIKI-TARGETWIKI.php and watch the output for errors. MWVERSIONFORTHATWIKI should not include "php-", it should be like "X.XX".
    • If you get an error, you probably got the path wrong, or the extension setup file is wonky.
    • If you get an error, you can also try running /home/catrope/sync-common , then running mergeMessageFileList.php again like above.
  • Check that a new ExtensionMessages-MWVERSIONFORTHATWIKI-TARGETWIKI.php file exists in wmf-config/ and makes sense. If it does, then svn commit the file.
  • Run scap.

Switch all wikis of one version to a different MW version

• First make sure that the new MW version exists. If not, see the section of this page concerned with checking out new versions.
• If no other wiki is already using the MW version then you probably should stop what you're doing.
• Run switchAllMediaWikis php-OLDVERSION php-NEWVERSION to switch all the wikis to using a new version. This can be for upgrades or for downgrades. This will rebuild the wikiversions.dat and wikiversions.cdb files.
• Run scap.

Rebuild the wikiversions.cdb file

Run refreshWikiversionsCDB to rebuild the cdb from the dat file.

Create a new MediaWiki version

• Do cd /home/wikipedia/common/multiversion
• Run checkoutMediaWiki X.XXwmfX php-VERSION to checkout a fresh MW install to common/php-VERSION from the mediawiki/branches/wmf/ SVN URL. This will create LocalSettings.php and some required symlinks.
• Check the symlinks in the new MediaWiki install's lib/ path to make sure they are pointing to the right library versions.
• Copy over some files from the main version of MW running into the new version:

  cp -a /home/wikipedia/common/php-<MAIN VER>/cache/trusted-xff.cdb /home/wikipedia/common/php-<NEW VER>/cache/

  cp -a /home/wikipedia/common/php-<MAIN VER>/cache/interwiki.cdb /home/wikipedia/common/php-<NEW VER>/cache/

• Run sync-common-all to push the unused code out to the apaches