View source for Switch master

== The Fairly Easy Way (asher uses this, domas did something similar) == 

# cd to your svn checkout of [https://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/switch-master/ trunk/tools/switch-master]
# Have a <code>~/.my.cnf</code> setup with the root mysql password 
# Create a config file according to <code>config.sample</code> for the cluster you're switching. Make sure the secondary master in in the slaves section! (it can be added in for the form of sX-secondary)
# Edit [https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=blob;f=wmf-config/db.php <code>db.php</code>], moving the new master to position 0. 
## Commit to the repository in this form (the post-switch version)
## Resume editing to comment out the old / current master (to drain it of all connections before the switch instead of sending a bunch of new slave queries to it - this results in a quicker swap) and to add this cluster to <code>readOnlyBySection</code>.
## <code>sync-file wmf-config/db.php "switching master for sX to dbXX"</code>
# run <code>./switch</code> and step through it
# <code>git checkout wmf-config/db.php</code> (reverts local changes from step 4.2)
# <code>sync-file wmf-config/db.php "completed master switch for sX"</code>
# make sure to <code>!log</code> the new master binlog and position for the cluster in #wikimedia-operations
# edit DNS to update the sX-master cname
# edit puppet to update the $masters array in mysql.pp

== The easy way (needs some updating) ==
: '''''DO NOT USE THIS IF THE MASTER IS DOWN ALREADY''''' (unless you really know what you are doing)

* NOTE: This tool doesn't currently switch the secondary master in the replication tree - '''make sure that is switched as well!!'''  You should also update the $masters section in mysql.pp, and update the $cluster-master or $cluster-secondary dns cnames - they must be accurate for heartbeat monitoring.

<pre>
cd /home/wikipedia/src/mediawiki/tools/switch-master
php switch.php <old master> <new master>
</pre>

==The hard way==

The script above (switch.php) does the following tasks, which you can do manually if you feel that way inclined:

* Check that the new master has binlog enabled. All slaves should be configured this way, for convenience. All servers, including masters, should have read_only mode on by default.
* Run RESET MASTER on the new master
* Log in to the mysql on the old master as root. Check for any long running queries. Kill them. If there is a single long-running query such as a backup, which will restart after you kill it, then get its thread ID ready in a kill command, ready to press enter later.
* Set read-only mode in db.php, sync it.
* Set read-only mode on the old master using SET GLOBAL read_only=1;
* Run FLUSH TABLES on the old master. This will block until all queries complete, so you may need to kill some now.
* Check for slave lag. Slave lag must be zero on the new master before you continue.
* Run STOP SLAVE on the new master
* Run SHOW MASTER STATUS on the new master to get the new replication position
* Run a CHANGE MASTER TO query on each of the slaves including the old master
* Set read/write mode on the new master using SET GLOBAL read_only=0;
* Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration.
* Run RESET SLAVE on the new master, to prevent it from replicating the old master after restart.

What could be easier than managing a replicated cluster in MySQL?


[[Category:How-To]]
[[Category:MySQL]]