Sartoris

Using this tool: https://github.com/git-deploy/git-deploy

git-deploy brainstorming:

deploying:

   git deploy start
   <bring core and extensions into what you want>
   git deploy sync
   <sync will tag the release, and run a sync-hook>

git-deploy sync:

1. Update bare repository
   1. Fetch bare repo from non-bare repo
   2. Purge varnish using ban.url .
2. Make the application servers do a fetch
   1. Keep a list of servers that we couldn't connect to
   2. If the list of failed servers is too large, exit, aborting the deploy
   3. If the list is within the threshold, depool the systems that failed
3. Once all application servers have done a fetch:
   1. Switch core to the tag
      • Which slot gets updated depends on the git-deploy prefix, accessible via $GIT_DEPLOY_HOOK_PREFIX
   2. Update (with --init) the submodules
   3. Keep a list of systems that failed to update - at this point it's too late to abort, but we should depool the systems that failed to update

naming:

   php-slot0 <- current
   php-slot1 <- next
   php-slot2 <- next + 1
   ...

On the deployment system, we should symlink version numbers to the slots, so that it's easy to tell version we are on, for instance:

   /home/w/common/php-1.20wmf1 -> /home/w/common/php-slot0
   /home/w/common/php-1.20wmf2 -> /home/w/common/php-slot1
   ...
   

On the appservers:

   /usr/local/apache/common/php-slot0
   /usr/local/apache/common/php-slot1
   ...

This may be a good place for something like perl's 'storable' which allows you to serialize/deserialize complex data structures for writing to disk or transfer. Depending on what we use slots for it's an efficient way to store more data--e.g. metadata about deployment versions

   Python's equivalent is pickle, and in php, we're already using cdb for version info (hetdeploy).  The slots scheme would need to work with our hetdeploy stuff, which I think assumes versions.  Either we'd need to sync the symlinks to the versions, or do a lot of work on hetdeploy.

Timeline for slots:

   slot0=wmf1, slot1=wmf2
   move all wmf1 wikis to wmf2 over time
   replace wmf1 with wmf3
   slot0=wmf3 slot1=wmf2
   start deploying wmf3
   move all wmf2 wikis to wmf3 over time
   replace wmf2 with wmf4
   slot0=wmf3 slot1=wmf4
   etc etc

Or:

   slot0=wmf1, slot1=wmf2
   move all wmf1 wikis to wmf2 over time
   once all are moved, switch slot0 to wmf2, move wikis to slot0
   rinse/repeat for next cycle

Examples

Example deploy of a core change:

   cd /home/w/common/php-1.20wmf1
   git deploy start
   git deploy sync

In the above scenario, 1.20wmf is the current version of MediaWiki we are running. /home/w/common/php-1.20wmf1 is a symlink to /home/w/common/php-slot0. When it syncs to the application servers, it is making git fetch and switch to a tag at /usr/local/apache/common/php-slot0. After switching to the tag, it'll also update all submodules to the versions listed in the tag point.

Example of changing versions of mediawiki:

   cd /home/w/common
   ln -s slot1 php-1.20wmf2
   cd php-1.20wmf2
   git deploy start
   git branch --track wmf/1.20wmf2 origin/wmf/1.20wmf2
   git checkout wmf/1.20wmf2
   git submodule update --init
   git deploy sync

This example will the the same thing as the previous example, but it will update /usr/local/apache/common/php-slot1 rather than /usr/local/apache/common/php-slot0.

Example of an emergency live hack:

   cd /home/w/common/php-1.20wmf1
   git deploy --no-remote start
   <make changes>
   git commit
   git deploy sync --no-remote

Code

https://gerrit.wikimedia.org/r/#/c/8732/1

Installation

This requires salt-minion installed on all application servers. salt-master should likely be installed on the puppet master. The module should be installed at:

/srv/salt/_modules/git_deploy.py

on the salt master. It can be deployed to all nodes via the following command from the master:

salt '*' saltutil.sync_modules

Technically it only needs to be sync'd to the application servers, but it doesn't hurt for the module to be installed everywhere.

The runner must currently be installed on the master at:

/usr/lib/pymodules/python2.6/salt/runners/git_deploy.py

and linked to:

/usr/share/pyshared/salt/runners/git_deploy.py

There's a (closed) bug in salt to have this be configurable. We need to upgrade versions, then we can install this in a saner location.

git-deploy must be installed somewhere in the path

• TODO: package this

After doing so, configure git deployment configuration location in git's global configuration. Then add a git deploy configuration, at minimum pointing the hooks directory to a central spot (for example, /usr/local/git-deploy/), then install the sync scripts.

• TODO: have puppet do all of this
• TODO: add a finish script to write out to IRC
• TODO: make a library that does common functions for all application sync scripts and make the sync-scripts specific to each application, rather than linking them as doing now
• TODO: make the sync script depool minions that fail, or fail the deployment if too many minions fail