View source for LocalisationUpdate

== Purpose ==
This MediaWiki extension fetches updates to MediaWiki core and extension localization files and makes them available to an installation without updating its own source. This allows us to pull updated translations being committed to development trunk even though our live install is running a deployment branch.

== Source ==
* http://www.mediawiki.org/wiki/Extension:LocalisationUpdate

== Big picture overview ==
* Translators tear it up on [http://translatewiki.net TranslateWiki]
* TranslateWiki staff commit translations to trunk
* LocalisationUpdate cron job runs every night at 02:00 UTC on fenari
** Pulls translations from SVN and writes updated messages to a cache directory
** Syncs cache directory out to the Apaches
* Apaches use cache directory as a source for translations along with the <code>Messages$LANGCODE.php</code> files, with the cache directory taking precedence

== What the cron job on fenari does ==
* Runs as the l10nupdate user. This user has a passphraseless SSH key
* Updates the SVN checkouts in <code>/var/lib/l10nupdate/trunk/phase3</code> and <code>/var/lib/l10nupdate/trunk/extensions</code>
* Runs <code>/home/wikipedia/common/php-$VERSION/extensions/LocalisationUpdate/update.php</code> for each MediaWiki version
** For core and for each extension, this script reads the i18n file(s) from the trunk checkout and compares them with the current translations on the cluster
** If a message has a new/updated translation, and the English version of the message has not changed, updates that message
** Writes message updates for each language to <code>/home/wikipedia/common/php-$VERSION/cache/l10n/l10nupdate-$LANGCODE.cache</code> in serialized PHP format
* Runs <code>/usr/local/bin/sync-l10nupdate</code> which syncs <code>/home/wikipedia/common/php-$VERSION/cache/l10n</code> on fenari to <code>/apache/common/php-$VERSION/cache/l10n</code> on the Apaches.
* Runs <code>/home/wikipedia/common/php-$VERSION/maintenance/wmf/clearMessageBlobs.php</code>, which clears message blobs cached by ResourceLoader
* Logs to <code>/var/log/l10nupdatelog/l10nupdate.log</code> in append mode. Log file is rotated by logrotate.
* logmsgbot !log's success or failure in #wikimedia-tech

== Running LU manually ==
Run <code>l10nupdate</code> on fenari. Syncing the message updates requires deploy access (i.e. ssh access to the Apaches).

Note that getting a few "Permission denied (publickey)" errors is normal. The following hosts do not have the l10nupdate user's SSH key:
* image scalers
* snapshot hosts
* search indexers
* bits Apaches
* fenari
* hume

== Files ==
* /usr/local/bin/l10nupdate-1
** Script that does all the work. The cron job runs this as the l10nupdate user
* /usr/local/bin/l10nupdate
** Wrapper that runs l10nupdate-1 as the l10nupdate user using sudo
* <s>/var/lib/l10nupdate/trunk
** MediaWiki checkouts of trunk phase3 and extensions</s>
* /var/lib/l10nupdate/mediawiki
** MediaWiki checkouts of trunk core and extensions
* /home/wikipedia/common/php-$VERSION/cache/l10n
** Storage for serialized updates
* /apache/common/php-$VERSION/cache/l10n
** Synced home of serialized updates per web server
* /usr/local/bin/sync-l10nupdate-1
** Script that syncs the updates to the Apaches
* /usr/local/bin/sync-l10nupdate
** Wrapper that runs sync-l10nupdate-1 as the l10nupdate user using sudo

== Where this are in puppet ==
* misc::l10nupdate (in manifests/misc-servers.pp)
** Installs cron job
** Installs l10nupdate, l10nupdate-1, sync-l10nupdate and sync-l10nupdate-1 scripts
** Creates logging directory and SVN checkout directory
** Installs logrotate config file
* files/misc/l10nupdate
** Scripts live here
* files/logrotate/l10nupdate
** logrotate config file

== Config ==
MediaWiki configuration variables:
* <code>$wmgUseLocalisationUpdate</code> (to enable/disable extension; lives in InitialiseSettings.php)
* <code>$wgLocalisationUpdateDirectory = dirname( $IP ) . "/php-$wmfExtendedVersionNumber/cache/l10n";</code> (set in CommonSettings.php)

== Todo ==
* Trim out test DBs/tables (localisation and localisation_file_hash) from previous incarnation:
** Singletons:
*** enwiki on s1
*** dewiki on <s>s2dewiki</s> s5
*** frwiki on <s>s3frja</s> s6
*** jawiki on <s>s3frja</s> s6
*** commonswiki on s4
** Shared:
*** l10nwiki on s2 (run via bgwiki)
*** l10nwiki on s3 (run via aawiki)


[[Category:Software]]
[[Category:Batch jobs]]