How to deploy code

Revision as of 23:54, 9 August 2012

This document SHOULD be up-to-date for the git migration. If something is wrong in this document, poke Roan.

Introduction

• All code is in version control (operations/mediawiki-config.git repository)
• This repository is checked out on fenari at /h/w/common
• sync scripts take care of synchronizing that from the working copy on fenari onto the servers.

See also

• [[test.wikipedia.org
• sync scripts
• Heterogeneous deployment
• git-deploy (a new deployment system design)

Basic common sense

• Be careful. Breaking the site is surprisingly easy!
• If you're deploying code written by someone else, ask them to be around during deployment so they can troubleshoot if necessary
• Make sure you know about anything hairy, such as additional prerequisites (e.g. schema changes) or potential complications when rolling back

Deployment requirements

• Cluster account request through RT; request deployment access (requires manager and/or sr dev approval)
  • If you can ssh into fenari, and ssh into a random srv box (e.g. srv300) from there, you already have this.
• Deployment branch access requested
  • See mw:Git/Gerrit project ownership
• Common sense. See above
• Some shiny code
• A window of time to deploy during (that doesn't overlap with anyone else's window). See software deployments.

Step 1: get the code in the deployment branch

Before you can deploy anything, it has to be in the deployment branch. Our deployment branches are named wmf/1.MAJORwmfMINOR where MAJOR and MINOR are numbers that increase over time as new branches are cut. A new branch with an incremented MINOR number is cut at the start of each deployment cycle, and after each tarball release MAJOR is incremented and MINOR is reset to 1. Strict access control is enforced on the deployment branches, but you should have access to them if you are a deployer. On the cluster, the checkout of each deployment branch is in /home/wikipedia/common/php-1.MAJORwmfMINOR .

Note that in most cases the cluster will be running on two deployment branches, with some wikis running version N and some running version N+1. To see which wiki is running which version, inspect /home/wikipedia/common/wikiversions.dat (public mirror) or look at Special:Version.

If your code or change needs to go live to all wikis, you will need to change all deployment branches that are in use. An easy way to see all of the versions currently in use is to log onto the staging server (fenari) and run 'mwversionsinuse' from the command line. You can also run 'mwversionsinuse --withdb' to see which wikis are running each version.

NOTE: All examples on this page assume there is a single deployment branch called wmf/1.20wmf1 checked out on the cluster in php-1.20wmf1. You may need to adapt the examples to use a different branch name. If you are updating multiple deployment branches, simply repeat the steps for each deployment branch separately.

NOTE: Also, all git examples assume you have a clean working copy, that is, you have no uncommitted changes. To verify this, run git status, it should say nothing added to commit (working directory clean) or nothing added to commit but untracked files present . If you are doing git-fu with a dirty working copy, there is a high probability you will screw things up, so don't do that unless you know what you're doing.

Case 1a: core changes

You are deploying changes to MediaWiki core. This should be rare because core is updated from master every two to three weeks, but in some cases it might be necessary. For core changes, you will simply need to push or submit changes to the wmf/1.20wmf1 branch in core. The most common use case is to take a commit that is already in the repository somewhere (usually in master, sometimes a commit that's still pending review) and cherry-pick it into the deployment branch, so only that case is documented below.

To cherry-pick a commit into the deployment branch, do the following things locally:

$ cd mediawiki/core      # go to your checkout of mediawiki/core.git

# Set up a local wmf/1.20wmf1 branch that tracks the remote
# You only need to do this once for each branch; if you've already got a wmf/1.20wmf1 branch,
# you can skip this step
$ git branch --track wmf/1.20wmf1 origin/wmf/1.20wmf1
Branch wmf/1.20wmf1 set up to track remote branch wmf/1.20wmf1 from origin.

# Switch to the wmf/1.20wmf1 branch and update it from the remote
$ git checkout wmf/1.20wmf1
$ git pull
$ git submodule update --init

# Cherry-pick a commit from master, identified by its hash
$ git cherry-pick ffb1b38ad83927606c539ac941e9f3eb2653a840

# If there are conflicts, this is how you fix them:
# - run 'git status' to see which files are conflicted
# - start fixing conflicted files using your favorite editor
# - use 'git add filename' to tell git you've fixed the conflicts in a file
# - once all conflicts are resolved, commit the result using 'git commit'

# Submit your cherry-pick commit for review
$ git review
# If you don't want or need this to be reviewed, you can +2 your own
# commit if you are in the wmf-deployment group

Case 1b: extension changes

You are deploying changes to an extension, but you don't just want to deploy master. Instead, you want to deploy the code that is currently deployed, plus your change. If you do want to just update your extension to master, read the extension update section instead.

Extensions have deployment branches too, but they are created on an as-needed basis, so your extension may or may not have a deployment branch with the correct name. If it doesn't exist yet, you'll have to create it. You can check by running git branch -a and checking if remotes/origin/wmf/1.20wmf1 or remotes/gerrit/wmf/1.20wmf1 appears in the list.

Creating an extension deployment branch

If your extension does not yet have a deployment branch, or doesn't have one with the correct name (e.g. there might be a 1.20wmf1 branch but used for an earlier deploy, but you're deploying to 1.20wmf3), you will need to create one. If your extension does have a correctly named deployment branch, you can skip this section.

First, find out the commit hash of the currently deployed commit:

$ cd mediawiki/core            # Go to your checkout of core

# Set up a local wmf/1.20wmf1 branch that tracks the remote
# You only need to do this once for each branch; if you've already got a wmf/1.20wmf1 branch, you can skip this step
$ git branch --track wmf/1.20wmf1 origin/wmf/1.20wmf1
Branch wmf/1.20wmf1 set up to track remote branch wmf/1.20wmf1 from origin.

# Switch to the wmf/1.20wmf1 branch and update it
$ git checkout wmf/1.20wmf1
$ git pull
# Update the extension submodules. This may take a while when you run it for the first time
$ git submodule update --init

# cd to your extension's submodule
$ cd extensions/MyCoolExtension
# Look at the most recent commit
$ git log -1
commit 434a740ef8dcd244029efd720a8e211e2803d343    # <---THIS
Author: Reedy <reedy@wikimedia.org>
Date:   Wed Apr 11 14:45:57 2012 +0100

Special:Version also lists the deployed commit hash for each extension, in shorthand form (e.g. 434a740), you can use that to double-check.

Now, create a new branch based off that commit:

$ cd mediawiki/extensions/MyCoolExtension    # Go to the extension checkout
# Create a new branch and switch to it
# Make sure you use the commit ID, not the change ID
$ git checkout -b wmf/1.20wmf1 434a740ef8dcd244029efd720a8e211e2803d343
Switched to a new branch 'wmf/1.20wmf1'
# Update .gitreview
$ vim .gitreview     # Change 'defaultbranch=master' to 'defaultbranch=wmf/1.20wmf1'
$ git commit .gitreview -m 'Create wmf/1.20wmf1 branch and update .gitreview'
# Push the branch into gerrit
$ git push origin wmf/1.20wmf1
# Let the local branch track the remote branch
$ git branch --set-upstream wmf/1.20wmf1 origin/wmf/1.20wmf1

Now that your extension has a deployment branch

Just like in core, the most common use case for updating a deployment branch is to cherry-pick changes from master:

$ cd mediawiki/extensions/MyCoolExtension      # go to your extension checkout

# Set up a local wmf/1.20wmf1 branch that tracks the remote
# You only need to do this once for each branch; if you've already got a wmf/1.20wmf1 branch, you can skip this step
$ git branch --track wmf/1.20wmf1 origin/wmf/1.20wmf1
Branch wmf/1.20wmf1 set up to track remote branch wmf/1.20wmf1 from origin.

# Switch to the wmf/1.20wmf1 branch and update it from the remote
$ git checkout wmf/1.20wmf1
$ git pull

# Cherry-pick a commit from master, identified by its hash
$ git cherry-pick 176ffdd3b71e463d3ebaa881a6e77b82acba635d
# If there are conflicts, this is how you fix them:
# run 'git status' to see which files are conflicted
# start fixing conflicted files
# use 'git add filename' to tell git you've fixed the conflicts in a file
# once all conflicts are resolved, commit the result using 'git commit'

# Submit your commit for review
$ git review
# If you don't want or need this to be reviewed, you can +2 your own
# commit if you are in the wmf-deployment group

Additionally, however, you will need to update the submodule in core:

$ cd mediawiki/core            # Go to your checkout of core

# Set up a local wmf/1.20wmf1 branch that tracks the remote
# You only need to do this once for each branch; if you've already got a wmf/1.20wmf1 branch, you can skip this step
$ git branch --track wmf/1.20wmf1 origin/wmf/1.20wmf1
Branch wmf/1.20wmf1 set up to track remote branch wmf/1.20wmf1 from origin.

# Switch to the wmf/1.20wmf1 branch and update it
$ git checkout wmf/1.20wmf1
$ git pull
# Update the extension submodules. This may take a while when you run it for the first time
$ git submodule update --init

# cd to your extension's submodule
$ cd extensions/MyCoolExtension
# Update it from the remote
$ git fetch
# Point the submodule to the remote wmf/1.20wmf1 branch
$ git checkout origin/wmf/1.20wmf1
HEAD is now at 96052d0... (bug 34885) Blah blah blah

# cd back to the main repo
$ cd ../..
# Check that you have updated the submodule correctly
# The diff should only show your submodule change, nothing else
$ git diff
diff --git a/extensions/MyCoolExtension b/extensions/MyCoolExtension
index 6a6eaab..96052d0 160000
--- a/extensions/MyCoolExtension
+++ b/extensions/MyCoolExtension
@@ -1 +1 @@
-Subproject commit 6a6eaabcdae29201c0f3e7bbcbac2d1953c574a5
+Subproject commit 96052d0d9733f89d5f8fd43c6ba05ecc756ea1ec

# Commit the submodule update and submit it for review
$ git commit -a -m "Update MyCoolExtension"
$ git review
# If you don't want or need this to be reviewed, you can +2 your own
# commit if you are in the wmf-deployment group

Case 1c: extension update

You are deploying an update to your extension, updating it to the current master. If you want to apply changes to your extension but don't want to update it to exactly the current master state, read the extension changes section instead.

All you need to do in this case is update the submodule in the core deployment branch:

$ cd mediawiki/core            # Go to your checkout of core

# Set up a local wmf/1.20wmf1 branch that tracks the remote
# You only need to do this once for each branch; if you've already got a wmf/1.20wmf1 branch, you can skip this step
# If you get an error, try 'git remote update' first
$ git branch --track wmf/1.20wmf1 origin/wmf/1.20wmf1
Branch wmf/1.20wmf1 set up to track remote branch wmf/1.20wmf1 from origin.

# Switch to the wmf/1.20wmf1 branch and update it
$ git checkout wmf/1.20wmf1
$ git pull
# Update the extension submodules. This may take a while when you run it for the first time
$ git submodule update --init

# cd to your extension's submodule
$ cd extensions/MyCoolExtension
# Update it from the remote
$ git fetch
# Point the submodule to the remote master branch
# Hint: If you want the 2nd to last version of master use "origin/master~1", etc.
$ git checkout origin/master
HEAD is now at 96052d0... (bug 34885) Blah blah blah

# cd back to the main repo
$ cd ../..
# Check that you have updated the submodule correctly
# The diff should only show your submodule change, nothing else
$ git diff
diff --git a/extensions/MyCoolExtension b/extensions/MyCoolExtension
index 6a6eaab..96052d0 160000
--- a/extensions/MyCoolExtension
+++ b/extensions/MyCoolExtension
@@ -1 +1 @@
-Subproject commit 6a6eaabcdae29201c0f3e7bbcbac2d1953c574a5
+Subproject commit 96052d0d9733f89d5f8fd43c6ba05ecc756ea1ec

# Commit the submodule update and submit it for review
$ git commit -a -m "Update MyCoolExtension to master"
# Submit your commit for review
$ git review
# If it gives you a warning about submitting more than one commit, you may want to do a 'git fetch gerrit' and retry
# If you don't want or need this to be reviewed, you can +2 your own
# commit if you are in the wmf-deployment group

Case 1d: new extension

You are adding an entirely new extension that wasn't deployed before, and you're deploying from master (if you need to deploy something other than the master state, that's possible, but it generally shouldn't be done for an initial deployment; master should just be clean and deployable).

You need to add a submodule to the core deployment branch:

$ cd mediawiki/core            # Go to your checkout of core

# Set up a local wmf/1.20wmf1 branch that tracks the remote
# You only need to do this once for each branch; if you've already got a wmf/1.20wmf1 branch, you can skip this step
$ git branch --track wmf/1.20wmf1 origin/wmf/1.20wmf1
Branch wmf/1.20wmf1 set up to track remote branch wmf/1.20wmf1 from origin.

# Switch to the wmf/1.20wmf1 branch and update it
$ git checkout wmf/1.20wmf1
$ git pull
# Update the extension submodules. This may take a while when you run it for the first time
$ git submodule update --init

# Add a submodule for your extension
$ git submodule add https://gerrit.wikimedia.org/r/p/mediawiki/extensions/MyCoolExtension.git extensions/MyCoolExtension
Cloning into extensions/MyCoolExtension...
# Check the diff. Make sure the .gitmodules entry is in line with the others, and check that the subproject commit hash points to master
$ git diff --cached
diff --git a/.gitmodules b/.gitmodules
index 3ab3d48..9a4cc66 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -346,3 +346,6 @@
 [submodule "extensions/PrefSwitch"]
        path = extensions/PrefSwitch
        url = https://gerrit.wikimedia.org/r/p/mediawiki/extensions/PrefSwitch.git
+[submodule "extensions/MyCoolExtension"]
+       path = extensions/MyCoolExtension
+       url = https://gerrit.wikimedia.org/r/p/mediawiki/extensions/MyCoolExtension.git
diff --git a/extensions/AllTimeZones b/extensions/MyCoolExtension
new file mode 160000
index 0000000..46727ad
--- /dev/null
+++ b/extensions/MyCoolExtension
@@ -0,0 +1 @@
+Subproject commit 46727ad74adda33621323deb2bebdc2527cb4917

# Commit the submodule addition and submit it for review
$ git commit -a -m "Add MyCoolExtension"
$ git review
# If you don't want or need this to be reviewed, you can +2 your own
# commit if you are in the wmf-deployment group

After adding a new extension to the deployment branch, you also have to add it to make-wmf-branch/default.conf in git (https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/tools/release.git) (in the $normalExtensions array) so it'll be picked up when the deployment branch is rebranched.

Step 2: get the code on fenari

Once the code is in the deployment branch, you simply run git pull on fenari to get it there:

catrope@fenari:~$ cd /home/wikipedia/common/php-1.20wmf1
catrope@fenari:/home/wikipedia/common/php-1.20wmf1/$ git pull

If you're deploying changes or updates to an existing extension, you'll need to update the submodule separately:

catrope@fenari:/home/wikipedia/common/php-1.20wmf1/$ git submodule update extensions/MyCoolExtension

If you're deploying a new extension, you have to explicitly tell it to initialize the submodule:

catrope@fenari:/home/wikipedia/common/php-1.20wmf1/$ git submodule update --init extensions/MyCoolExtension

The code is now automatically running on test.wikipedia.org as well, and can be tested there. If, however, you have updated Javscript/CSS files that do not get loaded with ResourceLoader (or are loaded with ResourceLoader in debug mode), you will need to ssh into srv193 (the server that runs testwiki) and run sync-common before they are live on testwiki.

If you need to flush the i18n cache on test.wikipedia.org, you can try running the rebuildLocalisationCache script:

mwscript rebuildLocalisationCache.php --wiki=testwiki --outdir=/home/wikipedia/common/php-1.20wmf1/cache/l10n

Step 3: configuration and other prep work

In certain cases, you'll have to change how Wikimedia sites are configured. We generally have the same codebase everywhere, but with different configurations for each wiki.

Maybe you are just changing one configuration variable. Or, perhaps you are adding a brand-new extension, or activating an extension on some wiki where it's never been before. For all of these cases, and more, you'll have to make the changes to the config files to get the desired results.

Configuration files live in their own revision-controlled repository. Everything that follows is just a convenient way to make those changes.

If you're deploying an extension or feature that can be switched off, it's usually best to leave it switched off while you deploy and carefully switch it on after that using a simple configuration change (this is called a dark launch). Even if you do this, you should build any configuration infrastructure (e.g. $wmg variable, adding entry in InitialiseSettings with default false) at this time so all you'll have to do later is flip a switch.

For specific preparations, see the sections below as well as How to do a schema change and How to do a configuration change. Best to perform schema changes before making config changes.

Add a configuration switch for an extension

In /home/wikipedia/common/wmf-config/CommonSettings.php, add:

if ( $wmgEnableMyExtension ) {
	require_once( "$IP/extensions/MyExtension/MyExtension.php" );
	// Set config vars if needed
 
	// If you want to export config vars through InitialiseSettings.php, you need to set $wmgMyExtensionThingy there and do
	#$wgMyExtensionThingy = $wmgMyExtensionThingy;
}

In /home/wikipedia/common/wmf-config/InitialiseSettings.php, add something like:

'wmgEnableMyExtension' => array(
	'default' => false,
	'eswikibooks' => true,
	// etc.
),
// If needed, set $wmgMyExtensionWhatever vars here too

If your extension requires a large-ish amount of configuration, consider putting it in a separate file instead. Currently, AbuseFilter, LiquidThreads and FlaggedRevs do this.

For more documentation on these files and their formats, see Configuration files.

Add new extensions to extension-list

When adding a new extension, you need to add it to the extension list, or its i18n messages won't get picked up. For more information about this setup, see this.

1. cd to /home/wikipedia/common/wmf-config
2. Edit extension-list and add the path to the extension setup file on a line by itself
3. Run scap. Make sure your extension is only enabled on testwiki at this point

Step 4: synchronize the changes to the cluster

Small changes: sync individual files

If your change only touches one or a few files or directories and does not change i18n messages, you can sync the files/dirs individually with sync-file or sync-dir as appropriate, rather than having to run scap. This is preferable because a scap run always shakes the cluster up a bit and takes longer to complete, while a sync-file run is very lightweight. However, sync-file is only capable of synchronizing files within directories that already exist on the cluster, so it won't work with newly added directories. Also, sync-file only synchronizes one file at a time, and creates a log entry each time. Using it repetitively (e.g. with a for loop) to sync multiple files is fine, as long as there's not too many of them (say not more than ~5).

To sync a single file, run sync-file [path to file] [summary]. To sync a directory, run sync-dir [path to directory] [summary]. The summary will be used by logmsgbot to log your sync in #wikimedia-tech, from where it'll go to the server admin log and the identi.ca and Twitter feeds.

• PITFALL: The path argument has to be relative to the common directory, not to the current directory. To preserve your sanity (and tab-completion functionality), I recommend you always cd to common before running sync-file
• PITFALL: If the summary argument contains spaces, you'll have to put it in quotes or only the first word is used. If your summary contains a $, you'll either have to escape it or put your summary in single quotes, to prevent bash's variable expansion from messing it up
• PITFALL: sync-file and sync-dir do not work correctly for syncing i18n changes. They will appear to work, but the i18n changes won't take effect. To sync i18n changes, you must use scap

When running sync-file or sync-dir, you'll usually see about half a dozen errors from broken servers. We should really fix things some time so this doesn't happen, but in the meantime you can consider this completely normal (sample output below). It usually completes within a few seconds, but in rare cases it may hang on a broken Apache for 1 or 2 minutes.

catrope@fenari:/home/wikipedia/common$ sync-file php-1.20wmf1/api.php 'API security fix'
No syntax errors detected in /home/wikipedia/common/php-1.20wmf1/api.php
copying to apaches
mw60: ssh: connect to host mw60 port 22: Connection timed out
srv189: ssh: connect to host srv189 port 22: Connection timed out
srv174: ssh: connect to host srv174 port 22: Connection timed out
srv162: ssh: connect to host srv162 port 22: Connection timed out
srv206: ssh: connect to host srv206 port 22: Connection timed out
srv284: ssh: connect to host srv284 port 22: Connection timed out
srv281: ssh: connect to host srv281 port 22: Connection timed out
srv266: ssh: connect to host srv266 port 22: Connection timed out

More complex changes: sync everything

If you're adding directories, changing many files, changing i18n messages, or otherwise have a reason why sync-file wouldn't work or would be impractical, you'll have to run scap, which syncs everything and rebuilds caches. scap logs to the server admin log, and reports in #wikimedia-tech (without !log) when it finishes.

awjrichards@fenari:/home/wikipedia/common/php$ scap 'Log message here'
Checking syntax...
Copying to fenari...Done.
Updating serialized data files...
Warning: messages are no longer serialized by this makefile.
Updating ExtensionMessages-1.19.php...
Updating ExtensionMessages-1.20wmf1.php...
Updating LocalisationCache for 1.19...
Updating LocalisationCache for 1.20wmf1...
...snip...

Running scap can take up to 15 minutes; the LocalisationCache rebuilds (usually two of them, one for each version) can also take a while. There is usually a load spike and a few hiccups on the cluster immediately after scapping, but that's normal, as long as it subsides a few minutes after scap finishes running.

Test your code live

Is it doing what you expected? Unfortunately, testwiki is not exactly like the real wikis. Consider that extensions can activate hooks. Things like CentralNotice or Common.js might change the JS environment too. No one environment can simulate all the wikis that we operate.

Don't leave town

Even if your deploy appears to be working, it's important to be reachable in the hours immediately following your deploy. Ideally, stay online and in IRC channels like #wikimedia-tech for a couple of hours.

If you must go offline, let people know how to reach you (and keep your mobile phone or other communications device on your person). You can use /away messages on IRC, or perhaps send a short email to private-l.

If you are on Wikimedia staff, now might be a great time to check if your contact info is up to date. If you aren't on staff, ask a staffer to add your contact info to that page, under "Important volunteers".

A note on JavaScript and CSS

Since we now have ResourceLoader, there is no need to do anything to re-minify and re-cache JavaScript and CSS. However, it may take up to five minutes before that occurs.