MobileFrontend
Contents |
Ops Configuration / Deployed Files
Varnish 3.0
- varnish3 class in varnish.pp (once bits is migrated, this will just be the varnish class)
- templates/varnish/mobile-frontend.vcl.erb (port 80)
- templates/varnish/mobile-backend.vcl.erb (port 81)
Squid
- /home/wikipedia/conf/squid/mobile_acls.conf (included by frontend.conf.php)
- puppet/files/squid/redirector (compiled C)
MediaWiki
InitialiseSettings.php: wmgMobileFrontend - default => true
Traffic Flow
Redirect to mobile
Redirection from main wikis to their mobile version is handled by the frontend text squids. Mobile devices are detected based on useragent acls and then passed to a redirector helper coded in C (/svnroot/mediawiki/trunk/debs/squid/redirector.c and installed via puppet) if the "stopMobileRedirect" cookie is not set.
The redirector matches on http://(\w+).wikipedia.org/wiki/(.*). In the case of a match, a 302 is returned with "Location: $1.m.wikipedia.org/wiki/$2", otherwise the request is served normally by squid. This to ensure that mobile devices can still make things like api calls in the future without squid configuration changes.
The squid redirect logic is in fenari:/home/w/conf/squid/mobile_acls.conf - useragents that look like mobile devices but which shouldn't be redirected (such as tablets) should be added to the nomobileua browser acl.
Mobile requests
The m.wikipedia.org vip is load balanced via lvs to a set of varnish servers running a set of front and backend varnish instances. The frontend varnish on port 80 includes VCL (puppet/templates/varnish/mobile-frontend.vcl.erb) that rewrites the host to drop the /m./ portion and sets an X-Device header based on UserAgent. The presence of this header activates the MediaWiki MobileExtension and is used to determine which mobile template to use. The frontend instance uses 256MB of ram for caching and hashes requests to one of the backend varnish instances which provide larger caches using the file store.
If a request is for the main page via just "/" - "?useformate=mobile" is appended to the backend request and the response cache ttl is set to 900 seconds because purge messages will never actually be received for this URI. This bypasses the redirect logic in MediaWiki that sends a 301 for / to /$main_page, where $main_page is different for every language wiki. If not bypassed, the 301 would be sent relative to the main wiki domain instead of the mobile site.
Requests not cached in either varnish instance are sent to the main apache appserver vip. Responses are returned with the same X-Device header added on in the frontend, and with X-Device added to Vary.
Cache purging
Cache purging is handled via varnishhtcpd (installed and started via puppet - see varnish.pp) which listens to the multicast PURGE messages also used by the squids. It currently only sends http purge requests to the backend varnish. Since host names are rewritten to strip out /m./ at the frontend, the same PURGE messages just work. That said, varnishhtcpd uses the LWP::UserAgent perl module to send purge requests to varnish in http proxy mode, which results in requests that look like:
PURGE http://en.wikipedia.org/wiki/Cat HTTP/1.1 Host: en.wikipedia.org
This would never actually match a cached article, so there is code in vcl_recv() to strip out the (http://[\w.]+)/ portion.
To prevent having to send purge messages to the frontend as well (possibly requiring a host transform), the backend returns cache-control headers that set a ttl of 300 seconds for all pages. This could be a problem if we add mobile editing, although in that cases, logged in editing mobile users would likely bypass caching entirely.
Logging
Our varnish3 package contains a patched version of varnishncsa (/trunk/debs/varnish3/debian/patches/01-varnishncsa-udp.dpatch) that adds sequence numbers and udp logging with multicast support. The default format has been changed to be compatible with that of our squids. It is run as a daemon, reads from the shared memory belonging to the frontend instance (specified via the "-n frontend" option) and sends udp log messages to our existing logging infrastructure.
