Deployment checklist features

This page contains a checklist and some guidelines for planning deployments executed by me (Roan), typically for the features development team. Other deployers may or may not use the same procedures and criteria; when in doubt, talk to them.

The basic principle is that I am only responsible for the actual deployment, and possibly review, of code. I am not responsible for fixing your code if something turns out to be wrong with it at the last minute, or for finding out any of the things on this checklist on my own. It's your responsibility to have everything ready in time, or I can and usually will postpone or reject the deployment.

Another basic but sometimes forgotten rule is that you should avoid wasting my time as much as you can. For scheduled deployments, be on time, and be responsive during. Don't make me wait for extended periods of time. The whole point of this checklist is that you have to provide me with nice pre-cooked blobs of information rather than having me find out things myself.

Audience

This page is aimed at the people coordinating directly with me to get something deployed. If you're just a "random" developer working on the code, this page is not directly targeted at you. Ideally, the person coordinating the deployment with me is also the primary developer of the code being deployed, but this doesn't always have to be the case.

Checklist

The basic rule here is that each of these things should be done at least 24 hours before the scheduled deployment (except of course for the scheduling itself, which needs to happen earlier). The number 24 is not set in stone: for very small deployments, shorter notice will suffice, and for larger deployments of e.g. extensions where I have to review things, I may need more time. Just provide me with as much information as you can as early as you can, and we can work things out from there.

• Preferably, you are the primary developer of the code being deployed. If not, tell me who is, and make sure that person is at least around during the deployment to help solve any problems.
• Tell me which EPM or higher manager-like person is overseeing the deployment. For features things, this is usually Alolita. I will be keeping this person in the loop on all things related to the deployment and possible review. If there's anyone else that should be kept in the loop, tell me about that too.
• All to-be-deployed code must have been reviewed first. Unless the amount of code to deploy is small, it has to have been reviewed before a deployment can be scheduled.
• Work with me to find a time slot to deploy it.
  • After we've settled on one, put it in the WMF engineering Google Calendar and invite me. Also invite the responsible EPM, per the above
  • Add the deployment to the software deployments page, and move it to the 'completed' section when it's done. Mention both your name and mine.
• Tell me what to deploy.
• Provide me with detailed instructions for anything else I need to do besides deploying the code, such as making configuration changes, applying database schema changes or running maintenance scripts.
• Tell me as much as you can about the operational impact of the deployment, so I can assess whether ops involvement is needed.
  • Specifically, if your deployment involves non-trivial database schema changes (i.e. anything other than creating new tables), we'll need ops assistance
• Tell me how I can reverse the deployment in case it causes problems. For standard garden-variety deployments it's just "undo the code changes, push to cluster, done" but for more complicated deployments rolling back can be more involved.

Deployment procedure

These are the steps for a typical deployment. Deployments where additional tasks must be performed may be a bit different. "Deploy" in this context means doing some or all of: updating code, making configuration changes and applying database schema changes.

• I copy or merge the to-be-deployed code to the deployment branch
  • If you have commit access to this branch and are willing to do the merge yourself, that's awesome, but please don't commit it until right before the deployment
• I deploy to testwiki
• We test it briefly
  • This is not the moment for extensive testing, you should've done that already. It's just a sanity check to make sure nothing unexpected happens in WMF's sometimes idiosyncratic environment
  • This testing is primarily your responsibility. I'll usually help you out a bit, but you're the expert
• Once you give me the go-ahead, I deploy to the cluster

IMPORTANT NOTE: If a deployment is scheduled for e.g. 10am, that means that we'll start this process at approximately 10am. It does NOT mean the changes will go live at 10am sharp. This usually happens between 10:15 and 10:30, or even later if the deployment is large or complex or if unexpected issues pop up. Deployment time slots are always at least 1 hour long (longer for complex deployments as appropriate), and the 'actual' deployment (i.e. the code going live on the wikis) may happen at any time during that time slot. If, for some reason, you need something to be deployed at an exact time or by a certain time, that can usually be accommodated, as long as you communicate these requirements clearly and early.

Deployment scheduling

Now obsolete since Roan's in San Francisco -- check with him.

My availability is a bit weird sometimes, because I go to school with varying schedules and am in a different timezone. I am 9 hours ahead of San Francisco, so the most common deployment times are in these ranges:

• 10am-noon SF time (=7-9pm my time) (except on Thursdays)
• 1pm-2pm (=10-11pm) (not always possible, e.g. if I have to get up at 6 the next morning)
• 11pm-1am (=8-10am the next day) (not always possible, e.g. if I have school that morning)

You do not absolutely have to stay within these, but it's what I usually end up with. Feel free to propose whichever time works best for you and we'll work something out. Also note that it may not be possible to deploy on the specific day you want, because I might be unavailable or one of the restrictions below might apply.

As a rule, deployments are never scheduled for:

• Fridays, Saturdays or Sundays
• Lunchtime (i.e. noon-1pm SF time)
• Late afternoon (i.e. 4pm SF time or later)

This is to prevent the site from breaking while few ops people are around. Additionally, for deployments that require ops involvement, we'll need to work with the schedules of the ops people in question.

Except in unusual cases, a deployment is scheduled for a one-hour timeslot, to allow for enough time to merge, test and deploy, and deal with minor setbacks.

How to tell me what I need to deploy

How to tell me what to deploy depends on the kind of deployment you want to do.

• New extension, deployment from trunk: tell me the name and SVN path of the extension
• Existing extension, update to trunk state: tell me the name and SVN path of the extension, and that you want it updated to trunk state
• Individual revisions to existing code (extensions or core): tag the revisions you want deployed in CodeReview and tell me the name of the tag you used. You're free to choose the name of the tag, as long as it's unique (not used for any other purpose at that time).
  • My personal naming convention is an acronym for the extension name followed by 'deploy', e.g. 'uwdeploy' for UploadWizard and 'aftdeploy' for ArticleFeedback
  • You can tag revisions by using the 'Add tag' and 'Remove tag' fields on revision views such as mw:Special:Code/MediaWiki/86000. On pages with revision lists such as mw:Special:Code/MediaWiki/author/catrope, you can use the checkboxes and the 'Add tag' and 'Remove tag' fields at the bottom of the page to mass-tag revisions.
  • You can view a list of revisions with a certain tag at mw:Special:Code/MediaWiki/tag/tagname

Code review

All code deployed to the Wikimedia cluster must be reviewed, and any issues raised must be resolved to the satisfaction of the reviewer(s). Exceptions can be made for minor issues that don't really affect deployment. However, we generally strive to get code in good shape and do things right the first time, rather than deploying it first and cleaning it up later. In my personal opinion, this is important because otherwise "later" often becomes "never" and we end up with low-quality code that works but is not very maintainable.

First, you need to find a reviewer. If you don't know who should review your code, ask your EPM. For people in the features team, it's usually me. You also need to tell the reviewer what to review, the same way you tell me what I need to deploy.

Reviews take time, depending on the amount of code involved, so budget for this. Also, reviewers aren't available around the clock and have other work to do, so request reviews early. Your code doesn't have to be 100% finished for a reviewer to start reviewing, but it should be mostly stable.

Assume that reviewers will find things wrong with your code (that's their job), and be prepared to fix them swiftly. It's very nice for reviewers when developers follow up quickly, because it allows them to review the follow=ups shortly after reviewing the original code. Remember that a review is not complete until the resulting follow-up commits have also been reviewed: people make mistakes all the time, also when fixing their earlier mistakes :)

A quick rundown of code review etiquette:

• Reviewers will leave comments on CodeReview, so make sure your SVN username is linked to your wiki username (you can see this at mw:Special:Code/MediaWiki/author/yoursvnname) and that you have e-mail notifications enabled in your preferences (and have set an e-mail address, of course).
• When changing something due to a reviewer's comment on a revision, put something like "followup r12345" in the commit summary. This will link the revisions together in CodeReview, and the reviewer will get a notification e-mail.
• If a reviewer marked your revision 'fixme' and you've addressed the issues, don't mark it 'resolved'; set the status to 'new' instead.

For more information about code review, see mw:Code review and mw:Code review guide.