Sartoris

Using this tool: https://github.com/git-deploy/git-deploy

git-deploy brainstorming:

Deploying

   git deploy start
   <bring core and extensions into what you want>
   git deploy sync
   <sync will tag the release, and run a sync-hook>

Sync hook

1. Update bare repository
   1. Fetch bare repo from non-bare repo
   2. Purge varnish using ban.url .
2. Make the application servers do a fetch
   1. Keep a list of servers that we couldn't connect to
   2. If the list of failed servers is too large, exit, aborting the deploy
   3. If the list is within the threshold, depool the systems that failed
3. Once all application servers have done a fetch:
   1. Switch core to the tag
      • Which slot gets updated depends on the git-deploy prefix, accessible via $GIT_DEPLOY_HOOK_PREFIX
   2. Update (with --init) the submodules
   3. Keep a list of systems that failed to update - at this point it's too late to abort, but we should depool the systems that failed to update

Naming

   php-slot0 <- current
   php-slot1 <- next
   php-slot2 <- next + 1
   ...

On the deployment system, we should symlink version numbers to the slots, so that it's easy to tell version we are on, for instance:

   /home/w/common/php-1.20wmf1 -> /home/w/common/php-slot0
   /home/w/common/php-1.20wmf2 -> /home/w/common/php-slot1
   ...

On the appservers:

   /usr/local/apache/common/php-slot0
   /usr/local/apache/common/php-slot1
   ...

This may be a good place for something like perl's 'storable' which allows you to serialize/deserialize complex data structures for writing to disk or transfer. Depending on what we use slots for it's an efficient way to store more data--e.g. metadata about deployment versions

Python's equivalent is pickle, and in php, we're already using cdb for version info (hetdeploy). The slots scheme would need to work with our hetdeploy stuff, which I think assumes versions. Either we'd need to sync the symlinks to the versions, or do a lot of work on hetdeploy.

Timeline for slots

   slot0=wmf1, slot1=wmf2
   move all wmf1 wikis to wmf2 over time
   replace wmf1 with wmf3
   slot0=wmf3 slot1=wmf2
   start deploying wmf3
   move all wmf2 wikis to wmf3 over time
   replace wmf2 with wmf4
   slot0=wmf3 slot1=wmf4
   etc etc

Or:

   slot0=wmf1, slot1=wmf2
   move all wmf1 wikis to wmf2 over time
   once all are moved, switch slot0 to wmf2, move wikis to slot0
   rinse/repeat for next cycle

Examples

Example deploy of a core change

   cd /home/w/common/php-1.20wmf1
   git deploy start
   git deploy sync

In the above scenario, 1.20wmf is the current version of MediaWiki we are running. /home/w/common/php-1.20wmf1 is a symlink to /home/w/common/php-slot0. When it syncs to the application servers, it is making git fetch and switch to a tag at /usr/local/apache/common/php-slot0. After switching to the tag, it'll also update all submodules to the versions listed in the tag point.

Example of changing versions of mediawiki

   cd /home/w/common
   ln -s slot1 php-1.20wmf2
   cd php-1.20wmf2
   git deploy start
   git branch --track wmf/1.20wmf2 origin/wmf/1.20wmf2
   git checkout wmf/1.20wmf2
   git submodule update --init
   git deploy sync

This example will the the same thing as the previous example, but it will update /usr/local/apache/common/php-slot1 rather than /usr/local/apache/common/php-slot0.

Example of an emergency live hack

   cd /home/w/common/php-1.20wmf1
   git deploy --no-remote start
   <make changes>
   git commit
   git deploy --no-remote sync

Trying it

https://gerrit.wikimedia.org/r/#/c/8732/1

Installation

This requires salt-minion installed on all application servers. salt-master should likely be installed on the puppet master. The module should be installed at:

/srv/salt/_modules/git_deploy.py

on the salt master. It can be deployed to all nodes via the following command from the master:

salt '*' saltutil.sync_modules

Technically it only needs to be sync'd to the application servers, but it doesn't hurt for the module to be installed everywhere.

The runner must currently be installed on the master at:

/usr/lib/pymodules/python2.6/salt/runners/git_deploy.py

and linked to:

/usr/share/pyshared/salt/runners/git_deploy.py

There's a (closed) bug in salt to have this be configurable. We need to upgrade versions, then we can install this in a saner location.

git-deploy must be installed somewhere in the path

After doing so, configure git deployment configuration location in git's global configuration. Then add a git deploy configuration, at minimum pointing the hooks directory to a central spot (for example, /usr/local/git-deploy/), then install the sync scripts.

Testing the demo in Labs

You'll need access to the demo project for this, and you'll currently need to be added as a sudoer.

On demo-deployment1, the deployment repos exist at:

/mnt/deployment/common
/mnt/deployment/common/slot0
/mnt/deployment/common/slot1

git-deploy is installed at /data/project/deployment/git-deploy; it's in root's path.

demo-web1 and demo-web2 have the repos cloned at:

/usr/local/apache/common
/usr/local/apache/common/slot0
/usr/local/apache/common/slot1

TODOs

• Package git-deploy
• Have puppet configure git deploy
  • Salt should actually do this, not puppet. The configuration is currently in the gerrit repo for git-deploy, but maybe this should be moved to the puppet repo.
• Add a finish script to git-deploy to write out to IRC
• Make a library for all of the git-deploy scripts that does common functions for all application sync scripts and make the sync-scripts specific to each application, rather than linking them as doing now
• Make the git-deploy sync script depool minions that fail, or fail the deployment if too many minions fail
  • We're currently doing this by checking all hosts by globs. salt 0.10.2 support minion data caching, so we should be able to do the same thing with grains, which would be more flexible.
• Create a gerrit user for deployment that only has rights to push tags to the repos
• Create a sudo policy for wikidev users to be able to call the salt runners
  • When 0.10.3 is released, also add an ACL so that we can run this without a sudo policy
• Write a salt peer communication policy to allow the deployment host to run the appropriate runners on the salt master
  • Add a master.d directory for the policies
• Add a .gitmodules_internal file for the repo, that points at the deployment system
• Puppetize salt master and minion