Upload.wikimedia.org
(→Backend: storage) |
(→Backend: scaling) |
||
| Line 44: | Line 44: | ||
== Backend: scaling == | == Backend: scaling == | ||
| − | < | + | Wikimedia's wikis are configured to defer most image scaling/rasterization operations (<tt>$wgGenerateThumbnailOnParse</tt> off), just putting "optimistic" URLs <img> tags and letting a 404 handler on the upload servers deal with making sure the images are in place. |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | '' | + | This does a couple nice things: |
| + | * Less interaction between primary servers and NFS | ||
| + | * Encapsulation of thumbnailing/scaling problems on the "image scaling cluster" so the entire grid doesn't go down due to runaway 'convert' processes etc. | ||
| + | |||
| + | The actual image servers ([[amane]] and [[storage1]]) farm out the scaling to the "image cluster" machines, a sub-cluster of apaches running MediaWiki just for <tt>thumb.php</tt>. | ||
| + | |||
| + | These backend scalers produce the rendered images and save them back to NFS, and they also get served back out to the requesting user-agent none the wiser. | ||
| + | |||
| + | Note that thumb.php can be accessed directly from the web as well on the regular "text" Apaches. Ideally we shouldn't do that? :) | ||
== Private wikis == | == Private wikis == | ||
Revision as of 23:14, 12 August 2008
Uploaded image and media files for Wikimedia's wikis appear on the separate subdomain upload.wikimedia.org.
This provides separate physical hosting for efficiency, a separate domain name to aid in browser connection splitting, and a separate domain name to aid in JavaScript security rules.
Contents |
Caching layer
Uploads have a separate set of squid proxy caches from the text squids; this avoids contention between the two data sets, which have different characteristics for object size, update rate, etc.
Backend: storage
| server | path | purpose |
|---|---|---|
| amane | upload3 | Most primary image dirs... |
| storage1 | upload4 | Some thumbs and... maybe some primary image dirs? |
| amane | math | texvc-rendered images |
Some horrible snipped from squid conf:
'apaches' => array(
'pmtpa' => array(
'amane.pmtpa.wmnet',
'=de_wiki' => 'storage1.wikimedia.org',
'=en_thumbs' => 'storage1.wikimedia.org',
'=commons_thumbs' => 'storage1.wikimedia.org',
'=wikisource_images' => 'storage1.wikimedia.org',
#'=en_thumbs' => 'anthony.wikimedia.org',
#'=de_thumbs' => 'srv6.wikimedia.org',
),
),
Backend: scaling
Wikimedia's wikis are configured to defer most image scaling/rasterization operations ($wgGenerateThumbnailOnParse off), just putting "optimistic" URLs <img> tags and letting a 404 handler on the upload servers deal with making sure the images are in place.
This does a couple nice things:
- Less interaction between primary servers and NFS
- Encapsulation of thumbnailing/scaling problems on the "image scaling cluster" so the entire grid doesn't go down due to runaway 'convert' processes etc.
The actual image servers (amane and storage1) farm out the scaling to the "image cluster" machines, a sub-cluster of apaches running MediaWiki just for thumb.php.
These backend scalers produce the rendered images and save them back to NFS, and they also get served back out to the requesting user-agent none the wiser.
Note that thumb.php can be accessed directly from the web as well on the regular "text" Apaches. Ideally we shouldn't do that? :)
Private wikis
Our private wikis do not serve their files to the public through this interface; they're served through img_auth.php on the local domain, which enforced authentication.
Compatibility links
Most of our wikis have an rewrite rule to redirect requests from /upload and /math on the primary wiki domain to the appropriate subdirectory on upload.wikimedia.org. This provides compatbility for old direct image links from the days before the separate image hosting.
Special:Filepath can also send redirects here.
