Switch master

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(two more steps following observed pitfall)
(updated documentation for new switch method)
Line 1: Line 1:
−
To switch masters:
+
==The easy way==
  
−
* Check that the new master has binlog enabled. All slaves should be configured this way, for convenience. All servers, including masters, should have read_only mode on by default.
+
<pre>
 +
cd /home/wikipedia/src/mediawiki/tools/switch-master
 +
php switch.php <old master> <new master>
 +
cd /home/wikipedia/conf/nagios
 +
./sync
 +
</pre>
  
−
cd [http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/switch-master/ ~tstarling/src/tools/switch-master]
+
==The hard way==
  
−
* Edit config.s1, config.s2 or config.s3 depending on which master you are switching. Then copy it to config. Config is the file that is actually used.
+
The script above (switch.php) does the following tasks, which you can do manually if you feel that way inclined:
−
** ''The "slaves" line should list only those which are neither old nor new master.''
+
 
 +
* Check that the new master has binlog enabled. All slaves should be configured this way, for convenience. All servers, including masters, should have read_only mode on by default.
 
* Run RESET MASTER on the new master
 
* Run RESET MASTER on the new master
 
* Log in to the mysql on the old master as root. Check for any long running queries. Kill them. If there is a single long-running query such as a backup, which will restart after you kill it, then get its thread ID ready in a kill command, ready to press enter later.
 
* Log in to the mysql on the old master as root. Check for any long running queries. Kill them. If there is a single long-running query such as a backup, which will restart after you kill it, then get its thread ID ready in a kill command, ready to press enter later.
 
* Set read-only mode in db.php, sync it.
 
* Set read-only mode in db.php, sync it.
−
* Start the script
+
* Set read-only mode on the old master using SET GLOBAL read_only=1;
−
 
+
* Run FLUSH TABLES on the old master. This will block until all queries complete, so you may need to kill some now.
−
./switch
+
−
 
+
−
* Press enter, this will attempt to switch the master to read-only. If there are long-running queries, kill them now. The script will block at FLUSH TABLES until you do.
+
 
* Check for slave lag. Slave lag must be zero on the new master before you continue.
 
* Check for slave lag. Slave lag must be zero on the new master before you continue.
−
* Press enter a few more times until it's done, these steps usually work. There may be an error from the old master saying that SLAVE STOP requires a running slave, you can ignore it.
+
* Run STOP SLAVE on the new master
 +
* Run SHOW MASTER STATUS on the new master to get the new replication position
 +
* Run a CHANGE MASTER TO query on each of the slaves including the old master
 +
* Set read/write mode on the new master using SET GLOBAL read_only=0;
 
* Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration.
 
* Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration.
 
* Run RESET SLAVE on the new master, to prevent it from replicating the old master after restart.
 
* Run RESET SLAVE on the new master, to prevent it from replicating the old master after restart.
−
* Update [[nagios]] configuration so it doesn't whine about slaves off:
 
−
 
−
(cd /home/wikipedia/conf/nagios && ./sync)
 
  
−
== Script requirements ==
+
What could be easier than managing a replicated cluster in MySQL?
−
* ~/.my.cnf or environment set up with db root pw so you don't have to type it in
+
−
** if missing, it doesn't seem to bother prompting :P
+

Revision as of 14:08, 21 May 2009

The easy way

cd /home/wikipedia/src/mediawiki/tools/switch-master
php switch.php <old master> <new master>
cd /home/wikipedia/conf/nagios
./sync

The hard way

The script above (switch.php) does the following tasks, which you can do manually if you feel that way inclined:

  • Check that the new master has binlog enabled. All slaves should be configured this way, for convenience. All servers, including masters, should have read_only mode on by default.
  • Run RESET MASTER on the new master
  • Log in to the mysql on the old master as root. Check for any long running queries. Kill them. If there is a single long-running query such as a backup, which will restart after you kill it, then get its thread ID ready in a kill command, ready to press enter later.
  • Set read-only mode in db.php, sync it.
  • Set read-only mode on the old master using SET GLOBAL read_only=1;
  • Run FLUSH TABLES on the old master. This will block until all queries complete, so you may need to kill some now.
  • Check for slave lag. Slave lag must be zero on the new master before you continue.
  • Run STOP SLAVE on the new master
  • Run SHOW MASTER STATUS on the new master to get the new replication position
  • Run a CHANGE MASTER TO query on each of the slaves including the old master
  • Set read/write mode on the new master using SET GLOBAL read_only=0;
  • Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration.
  • Run RESET SLAVE on the new master, to prevent it from replicating the old master after restart.

What could be easier than managing a replicated cluster in MySQL?

Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Switch_master&oldid=20196"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox