Switch master

Revision as of 17:47, 24 August 2011

The easy way

• DO NOT USE THIS IF THE MASTER IS DOWN ALREADY (unless you really know what you are doing)

cd /home/wikipedia/src/mediawiki/tools/switch-master
php switch.php <old master> <new master>

cd /home/wikipedia/conf/nagios ./sync -- FIXME: Nagios monitoring of mysql + master/slave replication appears to have been broken when moving to puppet.

The hard way

The script above (switch.php) does the following tasks, which you can do manually if you feel that way inclined:

• Check that the new master has binlog enabled. All slaves should be configured this way, for convenience. All servers, including masters, should have read_only mode on by default.
• Run RESET MASTER on the new master
• Log in to the mysql on the old master as root. Check for any long running queries. Kill them. If there is a single long-running query such as a backup, which will restart after you kill it, then get its thread ID ready in a kill command, ready to press enter later.
• Set read-only mode in db.php, sync it.
• Set read-only mode on the old master using SET GLOBAL read_only=1;
• Run FLUSH TABLES on the old master. This will block until all queries complete, so you may need to kill some now.
• Check for slave lag. Slave lag must be zero on the new master before you continue.
• Run STOP SLAVE on the new master
• Run SHOW MASTER STATUS on the new master to get the new replication position
• Run a CHANGE MASTER TO query on each of the slaves including the old master
• Set read/write mode on the new master using SET GLOBAL read_only=0;
• Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration.
• Run RESET SLAVE on the new master, to prevent it from replicating the old master after restart.

What could be easier than managing a replicated cluster in MySQL?