Memcached

From Wikitech
(Difference between revisions)
Jump to: navigation, search
Line 1: Line 1:
It is very important that as soon as a memcache server is down, it MUST be replaced as soon as possible in mc.php.
+
The memcache setup has changed '''recently''' for the multi-honed datacenter deployment (SDTPA vs EQIAD).
  
The memcached master php file is located in:
+
The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.
  <tt>/home/w/common/wmf-config/mc.php</tt>
+
  <tt>mediawiki-config/wmf-config/mc-eqiad.php</tt>
 +
<tt>mediawiki-config/wmf-config/mc-pmtpa.php</tt>
  
Memcached installation is handed via the puppet memcached role.
+
Memcached installation is handed via the puppet memcached role; and are currently deployed to dedicated memcache systems (mcX and mcXXXX).
  
  
Line 14: Line 15:
 
== Editing mc.php ==
 
== Editing mc.php ==
  
Please note what the file says, the order and format is '''very important.''' Make sure you add servers to the correct ***DOWN*** or ***SPARE*** sections, removing the slot number and replacing it with XXIf you move a server into the memcached pool, ensure you remove the XX and put in the correct number.
+
The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.
 +
  <tt>mediawiki-config/wmf-config/mc-eqiad.php</tt>
 +
  <tt>mediawiki-config/wmf-config/mc-pmtpa.php</tt>
  
 
If you want to add a server from the spare list to the active list, please test it first. You can run on fenari
 
If you want to add a server from the spare list to the active list, please test it first. You can run on fenari
Line 21: Line 24:
  
 
(it runs on a nonstandard port). As above, servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.  
 
(it runs on a nonstandard port). As above, servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.  
 
If you fix a memcached server and it is not immediately needed to go into rotation, be sure to move it from DOWN to SPARE.
 
  
 
Once you finish updating the file, you must '''git commit''' and then '''git push'''.  You should be logged in as your own username, as your user keys will be checked.  Then you need to sync it out with the following:
 
Once you finish updating the file, you must '''git commit''' and then '''git push'''.  You should be logged in as your own username, as your user keys will be checked.  Then you need to sync it out with the following:
Line 29: Line 30:
  
 
the configchange command will syntax check the file, attempt to check it into svn(this fails, do not worry), distribute it to the apaches, and log your change to the [[Server_admin_log|server admin log]] (via #wikimedia-tech).
 
the configchange command will syntax check the file, attempt to check it into svn(this fails, do not worry), distribute it to the apaches, and log your change to the [[Server_admin_log|server admin log]] (via #wikimedia-tech).
 
=== Adding Servers to the Memcache pool ===
 
 
Currently, this is a very serious undertaking, as even adding a single server to the total number of servers causes the need for all memcached servers to completely rebuild their cache.  Simple explination by Domas is crc32(key) % modulus.  That is on our side, not memcached, and thus is the issue.
 
 
We are not sure the impact, only that it possibly may not lat long, but we have no real info on this.
 
  
 
[[Category:How-To]]
 
[[Category:How-To]]

Revision as of 23:59, 11 February 2013

The memcache setup has changed recently for the multi-honed datacenter deployment (SDTPA vs EQIAD).

The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.

mediawiki-config/wmf-config/mc-eqiad.php
mediawiki-config/wmf-config/mc-pmtpa.php

Memcached installation is handed via the puppet memcached role; and are currently deployed to dedicated memcache systems (mcX and mcXXXX).


You can test the servers to see which memcached are functioning and which are not with the following on fenari:

cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php

Note that all servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.

Editing mc.php

The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.

mediawiki-config/wmf-config/mc-eqiad.php
mediawiki-config/wmf-config/mc-pmtpa.php

If you want to add a server from the spare list to the active list, please test it first. You can run on fenari

 cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php enwiki ip-address-here:11000

(it runs on a nonstandard port). As above, servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.

Once you finish updating the file, you must git commit and then git push. You should be logged in as your own username, as your user keys will be checked. Then you need to sync it out with the following:

cd /home/w/common; configchange wmf-config/mc.php 'enter the reason for updates' 

the configchange command will syntax check the file, attempt to check it into svn(this fails, do not worry), distribute it to the apaches, and log your change to the server admin log (via #wikimedia-tech).

Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox