Media server

Deploying a new media server

Checklist of things that need to be done when setting up a media server (from scratch, no jumpstart):

Base Solaris 10 install
fix timezone, get /tmp mounted, fix dump and swap sizes
Convert root filesystem to use ZFS
fix grub menu,
Set up raid
Install pkgtool from sourceforge
root home dir /root, get ssh keys over
add local nonroot user with path /export/home/username and install privs
Install from wmf spec files: pkgtool, pca
set proxy environ var for this, do as nonroot user
Check all patches, install using pca
set proxy environ var for this, need sun account with contract
Install from wmf spec files: screen, netcat
set proxy environ var for this, do as nonroot user
Copy media data and unpack
aggregate nge0 and nge1
Install Sun Java Webserver 7
install 1,2,3,4, put in /opt/webserver7/... with /export/upload as path
turn off web admin server and enable regular web server by editing xml files
Install 64 bit dtrace plugin
64-bit copy from ...
Install from wmf spec files: php, ganglia, libogg, libvorbis
copy /etc/gmond.conf from...
enable gmetrics in crontab by...
Set up cron jobs for replication and snapshots if this is a master

Basic procedure for switching to another media server, presuming it has been receiving data via replication:

Halt uploads and delete/restores temporarily
Turn off replication and snapshots on live media server
Do replication manually to new host to copy over data since last push
Mount the new /export/upload filesystem everywhere with a new mount point
Change /home/wikipedia/bin/scap so that it uses the new mount point
Change the mount point everywhere in CommonSettings.php and InitialiseSettings.php
Change the squid settings in upload-settings.php to use new host
Check we can read media as things are now
Stop webserver on old media server, check we can read media
Unmount old /export/upload everywhere, check we can read media
Turn on uploads
Test uploading of images
Change puppet settings for ExtensionDistributor and nfs mount point checks to use the new filesystem (misc-servers.pp, nfs.pp)
Test ExtensionDistributor, make sure it works

If we go with a new mountpoint:

Toss this line: Alias /centralnotice/ /mnt/upload6/centralnotice/ from /etc/apache2/wmf/wikimedia.conf (and make sure centralnotice still works)
Mount the new /export/upload filesystem everywhere with a new mount point, using Puppet
Halt uploads and delete/restores temporarily
on fenari, make a copy of /home/w/common/wmf-config/InitialiseSettings.php
in that same file, set the values for wgEnableUploads for the wikis that have 'true' to 'false'
sync-common-file to push that around
Wait 5-10 minutes for in progress uploads to finish
Change the mount point everywhere in CommonSettings.php, InitialiseSettings.php, filebackend.php, extdist/svn-invoke.conf
Change the settings for it in manifests/misc-servers.pp:misc::extension-distributor and push to fenari
Check that public and private media, ext-dist, math and timeline (did those move to swift yet?), captcha are working
Stop webserver on ms7, see if the above are still all good
Turn on uploads/deletes/restores
copy back the saved copy of InitialiseSettings.php on fenari to /home/w/common/wmf-config/
sync-common-file to push that around
Check that public and private media writes are working and that writes also go to the netapp
Unmount old /export/upload6 everywhere -- maybe not since we can't move captcha!!

(Please fill in with what's missing, etc)

To get live access logs on the media or thumb servers you can run the command

dtrace -qs access_log.d

which relies on the dtrace nsapi plugin and the file /opt/local/share/access_log.d

These are built from mediawiki/trunk/tools/nsapi-dtrace in case you need the sources.