Memcached
ArielGlenn (Talk | contribs) (→Editing mc.php) |
|||
| (18 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
| − | The | + | The memcache setup has changed '''recently''' for the multi-honed datacenter deployment (SDTPA vs EQIAD). |
| − | + | ||
| − | + | When a memcache server is offline, it should be entirely removed from the configuration (if permanent) or commented out (if temporary.) Unlike the old slot format config file, the new format doesn't require you to keep an identical number of members deployed in the memcached pool. Please note that one shouldn't add and remove a lot of members from this list in a short period of time, as the consistent hashing may result in a server being unpooled, then repooled to cause the keys to remap directly back to the same server, resulting in undesired performance. So if a server is down, fix it ASAP. If it cannot be rebooted and fixed ASAP, then it should be removed. If it is removed, it may be prudent to leave it unpooled for 24 hours? (Aaron advises we have some items that can cache in mc for 24 hours.) | |
| − | + | ||
| − | + | The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch. | |
| + | <tt>mediawiki-config/wmf-config/mc-eqiad.php</tt> | ||
| + | <tt>mediawiki-config/wmf-config/mc-pmtpa.php</tt> | ||
| − | + | Memcached installation is handed via the puppet memcached role; and are currently deployed to dedicated memcache systems (mcX and mcXXXX). | |
| − | + | ||
| − | |||
| − | + | You can test the servers to see which memcached are functioning and which are not with the following on fenari: | |
| + | <tt>cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php</tt> | ||
| − | + | Note that all servers that are up will return <tt>incr: 100 get: 100</tt> and servers that are down will return <tt>incr: 0 get: 0</tt>. | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | <tt> | + | |
| − | + | ||
| − | + | ||
| − | + | == Editing mc.php == | |
| − | + | The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch. | |
| + | <tt>mediawiki-config/wmf-config/mc-eqiad.php</tt> | ||
| + | <tt>mediawiki-config/wmf-config/mc-pmtpa.php</tt> | ||
| − | + | If you want to add a server from the spare list to the active list, please test it first. You can run on fenari | |
| − | + | <tt>cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php enwiki ip-address-here:11000</tt> | |
| − | + | (it runs on a nonstandard port). As above, servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0. | |
| − | Once you finish updating the file, you need to sync | + | Once you finish updating the file, you must '''git commit''' and then '''git review'''. You should be logged in as your own username, as your user keys will be checked. Then you need to git fetch and merge on fenari in the /home/wikimedia/common/wmf-deployment directory and sync out changes. |
| − | + | [[Category:How-To]] | |
Latest revision as of 00:33, 12 February 2013
The memcache setup has changed recently for the multi-honed datacenter deployment (SDTPA vs EQIAD).
When a memcache server is offline, it should be entirely removed from the configuration (if permanent) or commented out (if temporary.) Unlike the old slot format config file, the new format doesn't require you to keep an identical number of members deployed in the memcached pool. Please note that one shouldn't add and remove a lot of members from this list in a short period of time, as the consistent hashing may result in a server being unpooled, then repooled to cause the keys to remap directly back to the same server, resulting in undesired performance. So if a server is down, fix it ASAP. If it cannot be rebooted and fixed ASAP, then it should be removed. If it is removed, it may be prudent to leave it unpooled for 24 hours? (Aaron advises we have some items that can cache in mc for 24 hours.)
The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.
mediawiki-config/wmf-config/mc-eqiad.php mediawiki-config/wmf-config/mc-pmtpa.php
Memcached installation is handed via the puppet memcached role; and are currently deployed to dedicated memcache systems (mcX and mcXXXX).
You can test the servers to see which memcached are functioning and which are not with the following on fenari:
cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php
Note that all servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.
[edit] Editing mc.php
The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.
mediawiki-config/wmf-config/mc-eqiad.php mediawiki-config/wmf-config/mc-pmtpa.php
If you want to add a server from the spare list to the active list, please test it first. You can run on fenari
cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php enwiki ip-address-here:11000
(it runs on a nonstandard port). As above, servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.
Once you finish updating the file, you must git commit and then git review. You should be logged in as your own username, as your user keys will be checked. Then you need to git fetch and merge on fenari in the /home/wikimedia/common/wmf-deployment directory and sync out changes.
