LocalisationUpdate

Revision as of 21:48, 17 December 2011

Purpose

This MediaWiki extension fetches updates to MediaWiki core and extension localization files and makes them available to an installation without updating its own source. This allows us to pull updated translations being committed to development trunk even though our live install is running a deployment branch.

Source

• http://www.mediawiki.org/wiki/Extension:LocalisationUpdate

Big picture overview

• Translators tear it up on TranslateWiki
• TranslateWiki staff commit translations to trunk
• LocalisationUpdate cron job runs every night at 02:00 UTC on fenari
  • Pulls translations from SVN and writes updated messages to a cache directory
  • Syncs cache directory out to the Apaches
• Apaches use cache directory as a source for translations along with the Messages$LANGCODE.php files, with the cache directory taking precedence

What the cron job on fenari does

• Runs as the l10nupdate user. This user has a passphraseless SSH key
• Updates the SVN checkouts in /var/lib/l10nupdate/trunk/phase3 and /var/lib/l10nupdate/trunk/extensions
• Runs /home/wikipedia/common/php-$VERSION/extensions/LocalisationUpdate/update.php for each MediaWiki version
  • For core and for each extension, this script reads the i18n file(s) from the trunk checkout and compares them with the current translations on the cluster
  • If a message has a new/updated translation, and the English version of the message has not changed, updates that message
  • Writes message updates for each language to /home/wikipedia/common/php-$VERSION/cache/l10n/l10nupdate-$LANGCODE.cache in serialized PHP format
• Runs /usr/local/bin/sync-l10nupdate which syncs /home/wikipedia/common/php-$VERSION/cache/l10n on fenari to /apache/common/php-$VERSION/cache/l10n on the Apaches.
• Runs /home/wikipedia/common/php-$VERSION/maintenance/wmf/clearMessageBlobs.php, which clears message blobs cached by ResourceLoader
• Logs to /var/log/l10nupdatelog/l10nupdate.log in append mode. Log file is rotated by logrotate.
• logmsgbot !log's success or failure in #wikimedia-tech

Running LU manually

Run l10nupdate on fenari. Syncing the message updates requires deploy access (i.e. ssh access to the Apaches).

Note that getting a few "Permission denied (publickey)" errors is normal. The following hosts do not have the l10nupdate user's SSH key:

• image scalers
• snapshot hosts
• search indexers
• bits Apaches
• fenari
• hume

Files

• /usr/local/bin/l10nupdate-1
  • Script that does all the work. The cron job runs this as the l10nupdate user
• /usr/local/bin/l10nupdate
  • Wrapper that runs l10nupdate-1 as the l10nupdate user using sudo
• /var/lib/l10nupdate/trunk
  • MediaWiki checkouts of trunk phase3 and extensions
• /home/wikipedia/common/php-$VERSION/cache/l10n
  • Storage for serialized updates
• /apache/common/php-$VERSION/cache/l10n
  • Synced home of serialized updates per web server
• /usr/local/bin/sync-l10nupdate-1
  • Script that syncs the updates to the Apaches
• /usr/local/bin/sync-l10nupdate
  • Wrapper that runs sync-l10nupdate-1 as the l10nupdate user using sudo

Where this are in puppet

• misc::l10nupdate (in manifests/misc-servers.pp)
  • Installs cron job
  • Installs l10nupdate, l10nupdate-1, sync-l10nupdate and sync-l10nupdate-1 scripts
  • Creates logging directory and SVN checkout directory
  • Installs logrotate config file
• files/misc/l10nupdate
  • Scripts live here
• files/logrotate/l10nupdate
  • logrotate config file

Config

MediaWiki configuration variables:

• $wmgUseLocalisationUpdate (to enable/disable extension; lives in InitialiseSettings.php)
• $wgLocalisationUpdateDirectory = dirname( $IP ) . "/php-$wmfExtendedVersionNumber/cache/l10n"; (set in CommonSettings.php)

Todo

• Trim out test DBs/tables (localisation and localisation_file_hash) from previous incarnation:
  • Singletons:
    • enwiki on s1
    • dewiki on s2dewiki s5
    • frwiki on s3frja s6
    • jawiki on s3frja s6
    • commonswiki on s4
  • Shared:
    • l10nwiki on s2 (run via bgwiki)
    • l10nwiki on s3 (run via aawiki)