Switch master
From Wikitech
(Difference between revisions)
m (wikify) |
|||
| Line 1: | Line 1: | ||
| − | == The Fairly Easy Way (asher | + | == The Fairly Easy Way (asher uses this, domas did something similar) == |
| − | + | # cd to your svn checkout of [https://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/switch-master/ trunk/tools/switch-master] | |
| − | + | # Have a <code>~/.my.cnf</code> setup with the root mysql password | |
| − | + | # Create a config file according to <code>config.sample</code> for the cluster you're switching. Make sure the secondary master in in the slaves section! (it can be added in for the form of sX-secondary) | |
| − | + | # Edit [https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=blob;f=wmf-config/db.php <code>db.php</code>], moving the new master to position 0. | |
| − | + | #* Commit to the repository in this form (the post-switch version) | |
| − | + | #* Resume editing to comment out the old / current master (to drain it of all connections before the switch instead of sending a bunch of new slave queries to it - this results in a quicker swap) and to add this cluster to <code>readOnlyBySection</code>. | |
| − | * | + | #* <code>sync-file wmf-config/db.php "switching master for sX to dbXX"</code> |
| − | + | # run <code>./switch</code> and step through it | |
| − | + | # <code>git checkout wmf-config/db.php</code> (reverts local changes) | |
| − | + | # <code>sync-file wmf-config/db.php "completed master switch for sX"</code> | |
| − | + | # make sure to <code>!log</code> the new master binlog and position for the cluster in #wikimedia-operations | |
| − | + | # edit DNS to update the sX-master cname | |
| − | + | # edit puppet to update the $masters array in mysql.pp | |
| − | ==The easy way (needs some updating)== | + | == The easy way (needs some updating) == |
| − | + | : '''''DO NOT USE THIS IF THE MASTER IS DOWN ALREADY''''' (unless you really know what you are doing) | |
* NOTE: This tool doesn't currently switch the secondary master in the replication tree - '''make sure that is switched as well!!''' You should also update the $masters section in mysql.pp, and update the $cluster-master or $cluster-secondary dns cnames - they must be accurate for heartbeat monitoring. | * NOTE: This tool doesn't currently switch the secondary master in the replication tree - '''make sure that is switched as well!!''' You should also update the $masters section in mysql.pp, and update the $cluster-master or $cluster-secondary dns cnames - they must be accurate for heartbeat monitoring. | ||
Revision as of 16:33, 27 May 2012
The Fairly Easy Way (asher uses this, domas did something similar)
- cd to your svn checkout of trunk/tools/switch-master
- Have a
~/.my.cnfsetup with the root mysql password - Create a config file according to
config.samplefor the cluster you're switching. Make sure the secondary master in in the slaves section! (it can be added in for the form of sX-secondary) - Edit
db.php, moving the new master to position 0.- Commit to the repository in this form (the post-switch version)
- Resume editing to comment out the old / current master (to drain it of all connections before the switch instead of sending a bunch of new slave queries to it - this results in a quicker swap) and to add this cluster to
readOnlyBySection. -
sync-file wmf-config/db.php "switching master for sX to dbXX"
- run
./switchand step through it -
git checkout wmf-config/db.php(reverts local changes) -
sync-file wmf-config/db.php "completed master switch for sX" - make sure to
!logthe new master binlog and position for the cluster in #wikimedia-operations - edit DNS to update the sX-master cname
- edit puppet to update the $masters array in mysql.pp
The easy way (needs some updating)
- DO NOT USE THIS IF THE MASTER IS DOWN ALREADY (unless you really know what you are doing)
- NOTE: This tool doesn't currently switch the secondary master in the replication tree - make sure that is switched as well!! You should also update the $masters section in mysql.pp, and update the $cluster-master or $cluster-secondary dns cnames - they must be accurate for heartbeat monitoring.
cd /home/wikipedia/src/mediawiki/tools/switch-master php switch.php <old master> <new master>
The hard way
The script above (switch.php) does the following tasks, which you can do manually if you feel that way inclined:
- Check that the new master has binlog enabled. All slaves should be configured this way, for convenience. All servers, including masters, should have read_only mode on by default.
- Run RESET MASTER on the new master
- Log in to the mysql on the old master as root. Check for any long running queries. Kill them. If there is a single long-running query such as a backup, which will restart after you kill it, then get its thread ID ready in a kill command, ready to press enter later.
- Set read-only mode in db.php, sync it.
- Set read-only mode on the old master using SET GLOBAL read_only=1;
- Run FLUSH TABLES on the old master. This will block until all queries complete, so you may need to kill some now.
- Check for slave lag. Slave lag must be zero on the new master before you continue.
- Run STOP SLAVE on the new master
- Run SHOW MASTER STATUS on the new master to get the new replication position
- Run a CHANGE MASTER TO query on each of the slaves including the old master
- Set read/write mode on the new master using SET GLOBAL read_only=0;
- Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration.
- Run RESET SLAVE on the new master, to prevent it from replicating the old master after restart.
What could be easier than managing a replicated cluster in MySQL?
