Switch master
From Wikitech
(Difference between revisions)
| (17 intermediate revisions by 7 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | == The Fairly Easy Way (asher uses this, domas did something similar) == | |
| − | + | # cd to your svn checkout of [https://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/switch-master/ trunk/tools/switch-master] | |
| + | # Have a <code>~/.my.cnf</code> setup with the root mysql password | ||
| + | # Create a config file according to <code>config.sample</code> for the cluster you're switching. Make sure the secondary master in in the slaves section! (it can be added in for the form of sX-secondary) | ||
| + | # Edit [https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=blob;f=wmf-config/db.php <code>db.php</code>], moving the new master to position 0. | ||
| + | ## Commit to the repository in this form (the post-switch version) | ||
| + | ## Resume editing to comment out the old / current master (to drain it of all connections before the switch instead of sending a bunch of new slave queries to it - this results in a quicker swap) and to add this cluster to <code>readOnlyBySection</code>. | ||
| + | ## <code>sync-file wmf-config/db.php "switching master for sX to dbXX"</code> | ||
| + | # run <code>./switch</code> and step through it | ||
| + | # <code>git checkout wmf-config/db.php</code> (reverts local changes from step 4.2) | ||
| + | # <code>sync-file wmf-config/db.php "completed master switch for sX"</code> | ||
| + | # make sure to <code>!log</code> the new master binlog and position for the cluster in #wikimedia-operations | ||
| + | # edit DNS to update the sX-master cname | ||
| + | # edit puppet to update the $masters array in mysql.pp | ||
| − | + | == The easy way (needs some updating) == | |
| + | : '''''DO NOT USE THIS IF THE MASTER IS DOWN ALREADY''''' (unless you really know what you are doing) | ||
| − | * | + | * NOTE: This tool doesn't currently switch the secondary master in the replication tree - '''make sure that is switched as well!!''' You should also update the $masters section in mysql.pp, and update the $cluster-master or $cluster-secondary dns cnames - they must be accurate for heartbeat monitoring. |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | <pre> | |
| + | cd /home/wikipedia/src/mediawiki/tools/switch-master | ||
| + | php switch.php <old master> <new master> | ||
| + | </pre> | ||
| − | + | ==The hard way== | |
| + | |||
| + | The script above (switch.php) does the following tasks, which you can do manually if you feel that way inclined: | ||
| + | |||
| + | * Check that the new master has binlog enabled. All slaves should be configured this way, for convenience. All servers, including masters, should have read_only mode on by default. | ||
| + | * Run RESET MASTER on the new master | ||
| + | * Log in to the mysql on the old master as root. Check for any long running queries. Kill them. If there is a single long-running query such as a backup, which will restart after you kill it, then get its thread ID ready in a kill command, ready to press enter later. | ||
| + | * Set read-only mode in db.php, sync it. | ||
| + | * Set read-only mode on the old master using SET GLOBAL read_only=1; | ||
| + | * Run FLUSH TABLES on the old master. This will block until all queries complete, so you may need to kill some now. | ||
* Check for slave lag. Slave lag must be zero on the new master before you continue. | * Check for slave lag. Slave lag must be zero on the new master before you continue. | ||
| − | * | + | * Run STOP SLAVE on the new master |
| + | * Run SHOW MASTER STATUS on the new master to get the new replication position | ||
| + | * Run a CHANGE MASTER TO query on each of the slaves including the old master | ||
| + | * Set read/write mode on the new master using SET GLOBAL read_only=0; | ||
* Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration. | * Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration. | ||
| − | * | + | * Run RESET SLAVE on the new master, to prevent it from replicating the old master after restart. |
| + | |||
| + | What could be easier than managing a replicated cluster in MySQL? | ||
| + | |||
| + | |||
| + | [[Category:How-To]] | ||
| + | [[Category:MySQL]] | ||
Latest revision as of 16:36, 27 May 2012
[edit] The Fairly Easy Way (asher uses this, domas did something similar)
- cd to your svn checkout of trunk/tools/switch-master
- Have a
~/.my.cnfsetup with the root mysql password - Create a config file according to
config.samplefor the cluster you're switching. Make sure the secondary master in in the slaves section! (it can be added in for the form of sX-secondary) - Edit
db.php, moving the new master to position 0.- Commit to the repository in this form (the post-switch version)
- Resume editing to comment out the old / current master (to drain it of all connections before the switch instead of sending a bunch of new slave queries to it - this results in a quicker swap) and to add this cluster to
readOnlyBySection. -
sync-file wmf-config/db.php "switching master for sX to dbXX"
- run
./switchand step through it -
git checkout wmf-config/db.php(reverts local changes from step 4.2) -
sync-file wmf-config/db.php "completed master switch for sX" - make sure to
!logthe new master binlog and position for the cluster in #wikimedia-operations - edit DNS to update the sX-master cname
- edit puppet to update the $masters array in mysql.pp
[edit] The easy way (needs some updating)
- DO NOT USE THIS IF THE MASTER IS DOWN ALREADY (unless you really know what you are doing)
- NOTE: This tool doesn't currently switch the secondary master in the replication tree - make sure that is switched as well!! You should also update the $masters section in mysql.pp, and update the $cluster-master or $cluster-secondary dns cnames - they must be accurate for heartbeat monitoring.
cd /home/wikipedia/src/mediawiki/tools/switch-master php switch.php <old master> <new master>
[edit] The hard way
The script above (switch.php) does the following tasks, which you can do manually if you feel that way inclined:
- Check that the new master has binlog enabled. All slaves should be configured this way, for convenience. All servers, including masters, should have read_only mode on by default.
- Run RESET MASTER on the new master
- Log in to the mysql on the old master as root. Check for any long running queries. Kill them. If there is a single long-running query such as a backup, which will restart after you kill it, then get its thread ID ready in a kill command, ready to press enter later.
- Set read-only mode in db.php, sync it.
- Set read-only mode on the old master using SET GLOBAL read_only=1;
- Run FLUSH TABLES on the old master. This will block until all queries complete, so you may need to kill some now.
- Check for slave lag. Slave lag must be zero on the new master before you continue.
- Run STOP SLAVE on the new master
- Run SHOW MASTER STATUS on the new master to get the new replication position
- Run a CHANGE MASTER TO query on each of the slaves including the old master
- Set read/write mode on the new master using SET GLOBAL read_only=0;
- Edit db.php again, turning off read-only mode and simultaneously switching the loads configuration.
- Run RESET SLAVE on the new master, to prevent it from replicating the old master after restart.
What could be easier than managing a replicated cluster in MySQL?
