Media server

Revision as of 12:11, 3 October 2012

Checklist of things that need to be done when setting up a media server (from scratch, no jumpstart):

Base Solaris 10 install
fix timezone, get /tmp mounted, fix dump and swap sizes
Convert root filesystem to use ZFS
fix grub menu,
Set up raid
Install pkgtool from sourceforge
root home dir /root, get ssh keys over
add local nonroot user with path /export/home/username and install privs
Install from wmf spec files: pkgtool, pca
set proxy environ var for this, do as nonroot user
Check all patches, install using pca
set proxy environ var for this, need sun account with contract
Install from wmf spec files: screen, netcat
set proxy environ var for this, do as nonroot user
Copy media data and unpack
aggregate nge0 and nge1
Install Sun Java Webserver 7
install 1,2,3,4, put in /opt/webserver7/... with /export/upload as path
turn off web admin server and enable regular web server by editing xml files
Install 64 bit dtrace plugin
64-bit copy from ...
Install from wmf spec files: php, ganglia, libogg, libvorbis
copy /etc/gmond.conf from...
enable gmetrics in crontab by...
Set up cron jobs for replication and snapshots if this is a master

Basic procedure for switching to another media server, presuming it has been receiving data via replication:

Halt uploads and delete/restores temporarily
Turn off replication and snapshots on live media server
Do replication manually to new host to copy over data since last push
Mount the new /export/upload filesystem everywhere with a new mount point
Change /home/wikipedia/bin/scap so that it uses the new mount point
Change the mount point everywhere in CommonSettings.php and InitialiseSettings.php
Change the squid settings in upload-settings.php to use new host
Check we can read media as things are now
Stop webserver on old media server, check we can read media
Unmount old /export/upload everywhere, check we can read media
Turn on uploads
Test uploading of images
Change puppet settings for ExtensionDistributor and nfs mount point checks to use the new filesystem (misc-servers.pp, nfs.pp)
Test ExtensionDistributor, make sure it works

If we go with a new mountpoint:

Mount the new /export/upload filesystem everywhere with a new mount point, using Puppet
Halt uploads and delete/restores temporarily
Change the mount point everywhere in CommonSettings.php, InitialiseSettings.php, filebackend.php, probably filebackend_wmflabs.php, extdist/svn-invoke.conf
Check that public and private media, ext-dist, math and timeline (did those move to swift yet?), captcha are working
Stop webserver on ms7, see if the above are still all good
Turn on uploads/deletes/restores
Check that public and private media writes are working and that writes also go to the netapp
Unmount old /export/upload6 everywhere

(Please fill in with what's missing, etc)

To get live access logs on the media or thumb servers you can run the command

dtrace -qs access_log.d

which relies on the dtrace nsapi plugin and the file /opt/local/share/access_log.d

These are built from mediawiki/trunk/tools/nsapi-dtrace in case you need the sources.

@@ Line 54: / Line 54: @@
 If we go with a new mountpoint:
+* Mount the new /export/upload filesystem everywhere with a new mount point, using Puppet
 * Halt uploads and delete/restores temporarily
-* Mount the new /export/upload filesystem everywhere with a new mount point
 * Change the mount point everywhere in CommonSettings.php, InitialiseSettings.php, filebackend.php, probably filebackend_wmflabs.php, extdist/svn-invoke.conf
-* Change mount point in puppet: misc-servers.pp, add mount point in nfs.pp
 * Check that public and private media, ext-dist, math and timeline (did those move to swift yet?), captcha are working
 * Stop webserver on ms7, see if the above are still all good
-* Unmount old /export/upload6 everywhere
 * Turn on uploads/deletes/restores
 * Check that public and private media writes are working and that writes also go to the netapp
+* Unmount old /export/upload6 everywhere
 (Please fill in with what's missing, etc)