How to deploy code

THIS NEEDS TO BE UPDATED TO REFLECT THE NEW GIT/GERRIT WORKFLOW.

Basic common sense

• Be careful. Breaking the site is surprisingly easy!
• If you're deploying code written by someone else, ask them to be around during deployment so they can troubleshoot if necessary
• Make sure you know about anything hairy, such as additional prerequisites (e.g. schema changes) or potential complications when rolling back

Deployment Requirements

• Cluster account request through RT; request deployment access (requires manager and/or sr dev approval)
  • If you can ssh into fenari, and ssh into a random srv box (e.g. srv300) from there, you already have this.
• Deployment branch commit access requested by email to RobLa (CC Tim & Chad)
• Common sense. See above
• Some shiny code

Step 1: get the code in the deployment branch

Before you can deploy anything, it has to be in the deployment branch. The deployment branch is in our SVN at /branches/wmf/1.MAJORwmfMINOR where MAJOR and MINOR are numbers that rise over time as we rebranch. At the time of writing, the deployment version number was 1.17wmf1; if you're unsure which of the deployment branches is currently in use, you can look it up here.

To get code into the deployment branch, you have to merge it in (when deploying changes to already-deployed things) or copy it over (when deploying a new extension). This may have been done for you already, but since commit access to the deployment branch is limited, you may have to do it yourself. For details on how to merge things, see this guide on mediawiki.org.

Add new extensions to make-wmf-branch/default.conf

If you added a new extension to the deployment branch, you should also add it to /trunk/tools/make-wmf-branch/default.conf in SVN (in the $normalExtensions array) so it'll be picked up when the deployment branch is rebranched.

Step 2: get the code on fenari

Once the code is in the deployment branch, you simply run svn up on fenari to get it there:

catrope@fenari:~$ cd /home/wikipedia/common/php/
catrope@fenari:/home/wikipedia/common/php/$ svn up

The code is now automatically running on test.wikipedia.org as well, and can be tested there. If, however, you have svn up'd Javscript/CSS files that do not get loaded with ResourceLoader, you will need to ssh into srv193 (the server that runs testwiki) and run sync-common before they are live on testwiki.

Step 3: configuration and other prep work

In certain cases, you'll have to change how Wikimedia sites are configured. We generally have the same codebase everywhere, but with different configurations for each wiki.

Maybe you are just changing one configuration variable. Or, perhaps you are adding a brand-new extension, or activating an extension on some wiki where it's never been before. For all of these cases, and more, you'll have to make the changes to the config files to get the desired results.

Configuration files live in their own revision-controlled repository. Everything that follows is just a convenient way to make those changes.

If you're deploying an extension or feature that can be switched off, it's usually best to leave it switched off while you deploy and carefully switch it on after that using a simple configuration change (this is called a dark launch). Even if you do this, you should build any configuration infrastructure (e.g. $wmg variable, adding entry in InitialiseSettings with default false) at this time so all you'll have to do later is flip a switch.

For specific preparations, see the sections below as well as How to do a schema change and How to do a configuration change. Best to perform schema changes before making config changes.

Add a configuration switch for an extension

In /home/wikipedia/common/wmf-config/CommonSettings.php, add:

if ( $wmgEnableMyExtension ) {
	require_once( "$IP/extensions/MyExtension/MyExtension.php" );
	// Set config vars if needed
 
	// If you want to export config vars through InitialiseSettings.php, you need to set $wmgMyExtensionThingy there and do
	#$wgMyExtensionThingy = $wmgMyExtensionThingy;
}

In /home/wikipedia/common/wmf-config/InitialiseSettings.php, add something like:

'wmgEnableMyExtension' => array(
	'default' => false,
	'eswikibooks' => true,
	// etc.
),
// If needed, set $wmgMyExtensionWhatever vars here too

If your extension requires a large-ish amount of configuration, consider putting it in a separate file instead. Currently, AbuseFilter, LiquidThreads and FlaggedRevs do this.

For more documentation on these files and their formats, see Configuration files.

Add new extensions to extension-list

When adding a new extension, you need to add it to the extension list, or its i18n messages won't get picked up. For more information about this setup, see this.

1. cd to /home/wikipedia/common/wmf-config
2. Edit extension-list and add the path to the extension setup file on a line by itself
3. Where X.XX is the current MediaWiki version, run mwscript ../php/maintenance/mergeMessageFileList.php --wiki=aawiki --list-file=extension-list --output=ExtensionMessages-X.XX.php and watch the output for errors. Your extension should appear in the output the same way the others do
   • If you get an error, you probably got the path wrong, or the extension setup file is wonky
   • If you get an error, you can also try running /home/catrope/sync-common , then running mergeMessageFileList.php again like above
4. Run svn diff and verify that the changes to ExtensionMessages-X.XX.php make sense
   • A line like 'myext' => "$IP/extensions/myext/myext.i18n.php", should be added, and there should be no unrelated changes. Check the path to the i18n file for correctness
5. Do what needs to be done to deploy and register changes in these config files.

Step 4: synchronize the changes to the cluster

Small changes: sync individual files

If your change only touches one or a few files, you can sync them individually with sync-file rather than having to run scap. This is preferable because a scap run always shakes the cluster up a bit and takes longer to complete, while a sync-file run is very lightweight. However, sync-file is only capable of synchronizing files wihtin directories that already exist on the cluster, so it won't work with newly added directories. Also, sync-file only synchronizes one file at a time, and creates a log entry each time. Using it repetitively (e.g. with a for loop) to sync multiple files is fine, as long as there's not too many of them (say not more than ~5).

To sync a single file, run sync-file [path to file] [summary]. The summary will be used by logmsgbot to log your sync in #wikimedia-tech, from where it'll go to the server admin log and the identi.ca and Twitter feeds. Typically, you would use the revision ID

• PITFALL: The path argument has to be relative to the common directory, not to the current directory. To preserve your sanity (and tab-completion functionality), I recommend you always cd to common before running sync-file
• PITFALL: If the summary argument contains spaces, you'll have to put it in quotes or only the first word is used. If your summary contains a $, you'll either have to escape it or put your summary in single quotes, to prevent bash's variable expansion from messing it up

When running sync-file, you'll usually see about half a dozen errors from broken servers. We should really fix things some time so this doesn't happen, but in the meantime you can consider this completely normal (sample output below). It usually completes within a few seconds, but in rare cases it may hang on a broken Apache for 1 or 2 minutes.

catrope@fenari:/home/wikipedia/common$ sync-file php/api.php r12345
No syntax errors detected in /home/wikipedia/common/php-1.5/api.php
copying to apaches
srv104: ssh: connect to host srv104 port 22: No route to host
srv125: ssh: connect to host srv125 port 22: No route to host
srv149: ssh: connect to host srv149 port 22: Connection refused
srv169: ssh: connect to host srv169 port 22: Connection refused
srv284: ssh: connect to host srv284 port 22: No route to host
srv145: ssh: connect to host srv145 port 22: Connection timed out
srv206: ssh: connect to host srv206 port 22: Connection timed out
srv266: ssh: connect to host srv266 port 22: Connection timed out

More complex changes: sync everything

If you're adding directories, changing many files, or otherwise have a reason why sync-file wouldn't work or would be impractical, you'll have to run scap, which syncs everything and rebuilds caches. scap logs to the server admin log with the revision number you updated to, and reports in #wikimedia-tech (without !log) when it finishes.

awjrichards@fenari:/home/wikipedia/common/php$ scap 'Log message here'
Checking syntax...
Compiling texvc...ok
Copying to fenari...Done.
Updating serialized data files...
Warning: messages are no longer serialized by this makefile.
Updating ExtensionMessages.php...
...snip...

Running scap takes a few minutes. There is usually a load spike and a few hiccups on the cluster immediately after scapping, but that's normal, as long as it doesn't last longer than a few minutes.

Test your code live

Is it doing what you expected? Unfortunately, testwiki is not exactly like the real wikis. Consider that extensions can activate hooks. Things like CentralNotice or Common.js might change the JS environment too. No one environment can simulate all the wikis that we operate.

Don't leave town

Even if your deploy appears to be working, it's important to be reachable in the hours immediately following your deploy. Ideally, stay online and in IRC channels like #wikimedia-tech for a couple of hours.

If you must go offline, let people know how to reach you (and keep your mobile phone or other communications device on your person). You can use /away messages on IRC, or perhaps send a short email to private-l.

If you are on Wikimedia staff, now might be a great time to check if your contact info is up to date. If you aren't on staff, ask a staffer to add your contact info to that page, under "Important volunteers".

A note on JavaScript and CSS

Since we now have ResourceLoader, there is no need to do anything to re-minify and re-cache JavaScript and CSS. However, it may take up to five minutes before that occurs.