Eqiad Migration Planning

Needed Server Builds

• App and API Apaches
  • Image scalers?
• Swift
  • servers online; needs cluster replication enabled
• Memcached servers
  • half still need to be racked
  • 10GB cards need to be reconfigured to enable PXE
• Databases
  • one more slave is needed per shard
• Poolcounted
  • Not allocated? Should run a 2-3 misc servers
• Netapp /home
  • part of the mediawiki and apache deploy infrastructure
• Application logging server - for mediawiki wmerrors + apache syslog
  • eqiad version of the udp2log instance on nfs1 that writes to /home/w/logs
• External DNS
  • bring up an ns3 in eqiad?

Software / Config Requirements

• MediaWiki deploy support for per colo config variances [Bugzilla 39082]
  • generating eqiad and pmtpa dsh groups
  • new mediawiki conf files for eqiad
• replicating the git checkouts, etc. to new /home

Actually Failing Over

• deploy db.php with all shards set to read-only in both pmtpa and eqiad
• deploy squid and mobile + bits varnish configs pointing to eqiad apaches
• master swap every core db and writable es shard to eqiad
• deploy db.php in eqiad removing the read-only flag, leave it read-only in pmtpa
  • the above master-swap + db.php deploys can be done shard by shard to limit the time certain projects are read-only
• dns changes - our current steady state is to point wikipedia-lb.wikimedia.org in the US to eqiad but future scenarios may include external dns switches.

Improving Failover

• pre-generate squid + varnish configs for different primary datacenter roles
• implement MHA to better automate the mysql master failovers
• migrate session storage to redis, with redundant replicas across colos

See more

• Category:Eqiad cluster