Memcached

The memcached master php file is located in:

/home/wikipedia/common/wmf-deployment/wmf-config/mc.php

Memcached installation is handed via the puppet memcached role.

You can test the servers to see which memcached are functioning and which are not with the following:

cd /home/w/common/wmf-deployment/maintenance; php mctest.php

Note that all servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.

Editing mc.php

Please note what the file says, the order and format is very important. Make sure you add servers to the correct ***DOWN*** or ***SPARE*** sections, removing the slot number and replacing it with XX. If you move a server into the memcached pool, ensure you remove the XX and put in the correct number.

If you want to add a server from the spare list to the active list, please test it first. You can run

 php /home/wikipedia/common/wmf-deployment/maintenance/mctest.php enwiki ip-address-here:11000

(it runs on a nonstandard port). As above, servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.

If you fix a memcached server and it is not immediately needed to go into rotation, be sure to move it from DOWN to SPARE.

Once you finish updating the file, you need to sync it out with the following:

sync-file mc.php enter the reason for updates 

Adding Servers to the Memcache pool

Currently, this is a very serious undertaking, as even adding a single server to the total number of servers causes the need for all memcached servers to completely rebuild their cache. Simple explination by Domas is crc32(key) % modulus. That is on our side, not memcached, and thus is the issue.

We are not sure the impact, only that it possibly may not lat long, but we have no real info on this.