Eqiad Migration Planning

Coordination

We now have an incomplete tracking ticket in RT that depends on more specific tickets.
Sept 12 Update - http://etherpad.wikimedia.org/TechOps-12Sept2012

App and API Apaches
- Image scalers? (completed Precise build and deployed at Tampa). Ready to deploy @ Eqiad
- Apache - completed, and being deployed in both Tampa (in production) and Eqiad (for testing)
JobRunners
- will be on separate boxes in eqiad
- Peter will set up separate (i.e. non-appserver) job runners in Tampa first, as part of the precise upgrades - completed
- set up the same in eqiad
- we'll raise the fork limit from 5 to something like 12

Memcached servers
- mc01 - mc16 (Tampa) in production - done
- mc1001-mc1016 OS installed, ready for puppet to be run.
- Networked but issues with the Intel 10gNics and Dell's SPF+ - workaround available; getting new SPF+
- Decided to use Redis and use MW multi-write feature to write to both existing MC and the new MC servers, then enable Redis replication from Tampa to Eqiad

Databases - done
- ~~one more slave is needed per shard~~
Poolcounter
- Done: helium and potassium are installed and puppetized
Netapp
- /home/wikipedia for deployments
- /home - completed in Tampa, not strictly necessary in eqiad
Deployment server (fenari's deployment support infrastructure part, misc::deployment etc)
- awaiting new misc server racking next week
Hume equivalent (misc::maintenance) - postponed
Application logging server - for mediawiki wmerrors + apache syslog
- ~~eqiad version of the udp2log instance on nfs1 that writes to /home/w/logs~~
- Done: server 'flourine' for apache logs
Upload Varnish - done
Deployment Host (10 minutes job according to Tim, and he volunteered to own it)
- ~~Server OS install~~
- Deploy from deployment host to all application servers
- rsync the deployment code from the primary deployment server to the secondary
- Require a clean git repo
- Application servers in the other datacenter will use the secondary deployment system for rsync

Varnish software to handle media streaming efficiently
- awaiting patch from Varnish Software (target Sept?)
- patch MediaWiki to use a different upload hostname for large files. Then we could use Squid or some specialized media streaming proxy for large files.
MediaWiki deploy support for per colo config variances [Bugzilla 39082]
- generating eqiad and pmtpa dsh groups
- new mediawiki conf files for eqiad
replicating the git checkouts, etc. to new /home

deploy db.php with all shards set to read-only in both pmtpa and eqiad
deploy squid and mobile + bits varnish configs pointing to eqiad apaches
master swap every core db and writable es shard to eqiad
deploy db.php in eqiad removing the read-only flag, leave it read-only in pmtpa
- the above master-swap + db.php deploys can be done shard by shard to limit the time certain projects are read-only
dns changes - our current steady state is to point wikipedia-lb.wikimedia.org in the US to eqiad but future scenarios may include external dns switches.
Swift replication reversal - from Eqiad to Tampa

Identify and plan around the deployment/migration date - ~~tentatively Oct 15, 2012~~ [see below]. Need to communicate date.
- Migration needs to happen before Fundraising season starts in Nov.
- Vacation 'freeze'; all hands on deck week before and after deployment
  - Why? Not every person is vital to migration.
- migrate ns1 from tampa to ashburn, but not a critical item.

An update from CT Woo from October 2012 regarding the status of the migration is available here. It looks like it'll be pushed back to January or February 2013 (post-annual fundraiser).