View source for Image scalers

{{Old-doc}}
Image scalers handle generation of thumbnails.  

== How thumbs are served and generated ==

*The client's initial request for a thumbnail (example: http://upload.wikimedia.org/wikipedia/en/b/bc/Wiki.png) goes to [[upload.wikimedia.org]] ([[lvs]] server).  
*This farms out the request to one of a number of front-end [[squid]]s.  
*The front-end squid requests the data from a back-end squid.  
*The back-end squid looks in its cache; if there's no hit, it requests the image from the thumbnail server (currently [[ms4]]).  
*The thumbnail server looks for the file; if it is present it delivers it. If not, it invokes thumbnail-handler.php.
*The perl script, after some basic filename sanity checks, sends a request to the appropriate apache (e.g. en.wikipedia.org) for thumb.php with the filename and other optional parameters.
*The request to en.wikipedia.org (or other server) goes to a front-end squid. 
*The front-end squid asks a back-end squid.
*The back-end squid checks its cache; if there's nothing there it sends the request on to rendering.pmtpa (lvs server).
*The lvs server sends the request on to one of the scalers.
*The scaler reads the image from an nfs-mounted filesystem (hosted by the image server), creates the thumbnail, saves it directly (the filesystem from the thumbnail server is nfs-mounted on the scalers), and returns the data.

== Temp files ==

The scalers need to have the directory /a/magick-tmp on them (owned and writable by apache).  This matches the $wgImageMagickTempDir setting in CommonSettings.php. If the setting is changed, the directory on these servers must be changed as well.  Imagemagick temp files have the form magick-XXnnnnnn and the files get left around when conversion dies for some reason such as overruning reousrces allocated by ulimit.

Ghostscript (gs) is called by Imagemagick when it converts pdfs to thumbnails.  It leaves its temp files in whatever directory is pointed to be $wgTmpDirectory (curently /tmp).  Its files look like gs_nnnnnn.

Current ulimits shoot gs when it creates a scratch file bigger than 100mb.  This is more often than you might think; a pdf of about 10 mb can easily cause gs to write more than that.  And if someone does a search on commons that results in one of these files showing up in the search results, every time they look at that result list it will try to create the thumb.  A few of these can add up quickly, and we only have 4GB free on these boxes on the root partition.

To deal with this, a cron job clears out the above files from these directories every 5 minutes, if they are over an hour old.  As long as we don't accumulate too many of these huge gs_* files in an hour we will be ok.  It's been set up in [[puppet]]. (See /etc/puppet/manifests/imagescaler.pp on [[sockpuppet]].)  If you change these directories in the above config files, you need to change them in these cron jobs as well.

== Notes on Ubuntu Version Upgrades ==

* Packages that we build (at time of writing):
** librsvg (the no external files patch may be upstreamed, tim submitted it)
** imagemagick - main reason for this is to set quantum depth to 8 from 16 to save mem.
** all php packages that all apaches use

* packages to watch (could cause regressions/might need to build)
** ghostscript
** oggvideotools
** ffmpeg2theora
** libogg
** libtheora

[[Category:Image handling]]