Wikimedia binaries

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(update based on source code)
(→‎scap)
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
{{TOC|align=right}}
 
{{TOC|align=right}}
−
The following executable '''binaries''' are maintained in <code>/home/wikipedia/bin</code> on [[fenari]].
+
The following executable '''binaries''' are maintained in version control ([https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=tree;f=files/misc/scripts;hb=HEAD operations/puppet : <code>files/misc/scripts</code>]) and copied/synced to servers (such as [[fenari]]) by puppet into <code>/usr/local/bin</code>.
  
 
== bin ==
 
== bin ==
Line 9: Line 9:
 
: ''This script operates on all servers in the following dsh groups: '''apaches'''
 
: ''This script operates on all servers in the following dsh groups: '''apaches'''
 
* Runs <code>apache-graceful</code> locally on all apaches.
 
* Runs <code>apache-graceful</code> locally on all apaches.
 +
 +
=== <code>sync-apache</code> ===
 +
: ''This script operates on all servers in the following dsh groups: '''apaches, image_scalers, snapshot, searchidx'''
 +
: '''Note:''' The conf files are outside of <code>/home/wikipedia/''common''</code>, and as such are '''not''' synced by the <code>sync-common</code>-like scripts.
 +
* syncs <code>/home/wikipedia/conf/httpd</code> to <code>/apache/conf</code>
  
 
=== <code>sync-common</code> ===
 
=== <code>sync-common</code> ===
Line 16: Line 21:
 
=== <code>sync-common-all</code> ===
 
=== <code>sync-common-all</code> ===
 
: ''This is an [[#All-script|all-script]]!''
 
: ''This is an [[#All-script|all-script]]!''
−
* syncs <code>/home/wikipedia/common</code> -> <code>/apache/common</code>
+
* <s>syncs <code>/home/wikipedia/common</code> -> <code>/apache/common</code></s><br/>=> Runs [[#scap|scap]]
  
 
=== <code>sync-common-file</code> ===
 
=== <code>sync-common-file</code> ===
 
: ''This is an [[#All-script|all-script]]!''
 
: ''This is an [[#All-script|all-script]]!''
−
* syncs <code>/home/wikipedia/common/ (some file)</code> -> <code>/apache/common</code>
+
* for a single directory or file
 +
* syncs <code>/home/wikipedia/common/ (some path)</code> -> <code>/apache/common</code>
  
 
=== <code>sync-dblist</code> ===
 
=== <code>sync-dblist</code> ===
Line 29: Line 35:
 
: ''This is an [[#All-script|all-script]]!''
 
: ''This is an [[#All-script|all-script]]!''
 
* syncs <code>/home/wikipedia/common/docroot</code> -> <code>/apache/common/docroot</code>
 
* syncs <code>/home/wikipedia/common/docroot</code> -> <code>/apache/common/docroot</code>
 +
 +
=== <code>sync-dir</code> ===
 +
: ''This is an [[#All-script|all-script]]!''
 +
* for a single directory
 +
* checks PHP syntax
 +
* syncs <code>/home/wikipedia/common/ (some dir)</code> -> <code>/apache/common</code> with <code>[[#sync-common-file|sync-common-file]]</code>
  
 
=== <code>sync-file</code> ===
 
=== <code>sync-file</code> ===
 
: ''This is an [[#All-script|all-script]]!''
 
: ''This is an [[#All-script|all-script]]!''
 +
* for a single file
 
* checks PHP syntax
 
* checks PHP syntax
−
* syncs <code>/home/wikipedia/common/wmf-deployment/ (some file)</code> -> <code>/apache/common/wmf-deployment</code>
+
* syncs <code>/home/wikipedia/common/ (some file)</code> -> <code>/apache/common</code> with <code>[[#sync-common-file|sync-common-file]]</code>
 +
 
 +
=== <code>sync-wikiversions</code>===
 +
: ''This script operates on all servers in the following dsh groups: '''mediawiki-installation'''
 +
* syncs <code>common/wikiversions.dat</code> -> <code>/apache/common-local/wikiversions.dat</code>
 +
* syncs <code>common/wikiversions.cdb</code> -> <code>/apache/common-local/wikiversions.cdb</code>
  
 
=== <code>scap</code> ===
 
=== <code>scap</code> ===
 
: ''This is an [[#All-script|all-script]]!''
 
: ''This is an [[#All-script|all-script]]!''
 
{{shortcut|scap}}
 
{{shortcut|scap}}
−
* checks PHP syntax
+
* lints PHP files in <code>./wmf-config</code>, <code>./multiversion</code> and <code>php-*</code>
−
* rebuilds localization caches and texvc
+
* for all mediawiki versions currently deployed (usually 2) it:
−
* syncs <code>/home/wikipedia/common/wmf-deployment</code> -> <code>/apache/common/wmf-deployment</code>
+
** rebuilds localization caches of core and extensions
−
 
+
** recompiles texvc
−
=== <code>sync-apache</code> ===
+
* runs [[sync-wikiversions]]
−
: ''This script operates on all servers in the following dsh groups: '''apaches, image_scalers, snapshot, searchidx'''
+
* syncs <code>/home/wikipedia/common/php-{versions}</code> -> <code>/apache/common/php-{versions}</code>
−
: '''Note:''' The conf files are outside of <code>/home/wikipedia/''common''</code>, and as such are '''not''' synced by the <code>sync-common</code>-like scripts.
+
* <small>(does it also sync other directories in <code>/common</code> such as <code>live-1.5</code> and <code>docroot</code>?)</small>
−
* syncs <code>/home/wikipedia/conf/httpd</code> to <code>/apache/conf</code>
+
  
 
== Unsorted ==
 
== Unsorted ==
Line 81: Line 98:
 
== Examples ==
 
== Examples ==
 
=== <code>sync-apache</code> example ===
 
=== <code>sync-apache</code> example ===
−
# On [[fenari]]:<br/><code>cd /home/wikipedia/bin</code>
+
# On [[fenari]]:
−
# (optionally) run:<br/><code>./sync-apache-simulated</code><br/>This just has an "-n" added to the [[rsync]] commands. (<code>--dry-run | </code> show what would have been transferred)
+
# (optionally) run:<br/><code>sync-apache-simulated</code><br/>This just has an "-n" added to the [[rsync]] commands. (<code>--dry-run | </code> show what would have been transferred)
−
# If that looks fine, actually sync:<br/><code>./sync-apache</code>
+
# If that looks fine, actually sync:<br/><code>sync-apache</code>
 
# Connect to a single random "srv" server, confirm the expected change has arrived here, and restart [[Apache]] to check your change did not break things.
 
# Connect to a single random "srv" server, confirm the expected change has arrived here, and restart [[Apache]] to check your change did not break things.
 
## (how to restart that apache?)
 
## (how to restart that apache?)
−
# If that also looks fine, restart all Apaches:<br/><code>./apache-graceful-all</code>
+
# If that also looks fine, restart all Apaches:<br/><code>apache-graceful-all</code>
  
 
After this you might want to [[Purge URL|purge a URL]] from [[Squid]] cache. Also see [[Squids#Purge a given external (non WMF) url]].
 
After this you might want to [[Purge URL|purge a URL]] from [[Squid]] cache. Also see [[Squids#Purge a given external (non WMF) url]].
Line 93: Line 110:
  
 
== See also ==
 
== See also ==
−
* [[Heterogeneous deployment]].
+
* [[Heterogeneous deployment]]
 +
 
 
[[Category:Software]]
 
[[Category:Software]]
 
[[Category:How-To]]
 
[[Category:How-To]]

Latest revision as of 08:45, 27 February 2013

Contents

The following executable binaries are maintained in version control (operations/puppet : files/misc/scripts) and copied/synced to servers (such as fenari) by puppet into /usr/local/bin.

[edit] bin

[edit] apache-graceful

Performs the action only on the one server from which the command is ran

[edit] apache-graceful-all

This script operates on all servers in the following dsh groups: apaches
  • Runs apache-graceful locally on all apaches.

[edit] sync-apache

This script operates on all servers in the following dsh groups: apaches, image_scalers, snapshot, searchidx
Note: The conf files are outside of /home/wikipedia/common, and as such are not synced by the sync-common-like scripts.
  • syncs /home/wikipedia/conf/httpd to /apache/conf

[edit] sync-common

Performs the action only on the one server from which the command is ran
  • syncs /home/wikipedia/common -> /apache/common

[edit] sync-common-all

This is an all-script!
  • syncs /home/wikipedia/common -> /apache/common
    => Runs scap

[edit] sync-common-file

This is an all-script!
  • for a single directory or file
  • syncs /home/wikipedia/common/ (some path) -> /apache/common

[edit] sync-dblist

This is an all-script!
  • syncs /home/wikipedia/common/*.dblist -> /apache/common

[edit] sync-docroot

This is an all-script!
  • syncs /home/wikipedia/common/docroot -> /apache/common/docroot

[edit] sync-dir

This is an all-script!
  • for a single directory
  • checks PHP syntax
  • syncs /home/wikipedia/common/ (some dir) -> /apache/common with sync-common-file

[edit] sync-file

This is an all-script!
  • for a single file
  • checks PHP syntax
  • syncs /home/wikipedia/common/ (some file) -> /apache/common with sync-common-file

[edit] sync-wikiversions

This script operates on all servers in the following dsh groups: mediawiki-installation
  • syncs common/wikiversions.dat -> /apache/common-local/wikiversions.dat
  • syncs common/wikiversions.cdb -> /apache/common-local/wikiversions.cdb

[edit] scap

This is an all-script!
shortcut: scap
  • lints PHP files in ./wmf-config, ./multiversion and php-*
  • for all mediawiki versions currently deployed (usually 2) it:
    • rebuilds localization caches of core and extensions
    • recompiles texvc
  • runs sync-wikiversions
  • syncs /home/wikipedia/common/php-{versions} -> /apache/common/php-{versions}
  • (does it also sync other directories in /common such as live-1.5 and docroot?)

[edit] Unsorted

These are outdated, redundant, broken or otherwise undocumented:
  • scap14, scap14-1, scap14-2
    • obsolete from MediaWiki 1.4 days
  • scap15, scap15-1, scap15-1skins
    • 'scap15' is actual script for 'scap' currently; probably could use cleanup
  • sync-file13, sync-file14
  • sync-file-log14 ?
  • sync-to-seoul
  • sync-common-allphp
  • sync-common-clean
  • sync-common-it
  • sync-common-php
  • sync-common-php-x
  • sync-mirror-dag.sh
  • sync-mirror-fedora.sh
  • sync-users-and-groups.sh
  • syncout
  • sync-cgibin

[edit] All-script

An all-script is a script that operates on mediawiki-installation boxes via dsh. These scripts performs the action on all servers. These servers to be exact:

[edit] Examples

[edit] sync-apache example

  1. On fenari:
  2. (optionally) run:
    sync-apache-simulated
    This just has an "-n" added to the rsync commands. (--dry-run | show what would have been transferred)
  3. If that looks fine, actually sync:
    sync-apache
  4. Connect to a single random "srv" server, confirm the expected change has arrived here, and restart Apache to check your change did not break things.
    1. (how to restart that apache?)
  5. If that also looks fine, restart all Apaches:
    apache-graceful-all

After this you might want to purge a URL from Squid cache. Also see Squids#Purge a given external (non WMF) url.

In most cases this is enough. However if the change involves more than a simple change to redirects, depooling/ repooling may be necessary. Make sure you find out whether this is needed ahead of time (while at it, start the depooling page and link to it, we need documentation!).

[edit] See also

Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Wikimedia_binaries&oldid=56876"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox