Wikimedia binaries

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(notice)
(→‎scap)
 
(20 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
'''See [[Heterogeneous deployment]] for now'''
+
{{TOC|align=right}}
 +
The following executable '''binaries''' are maintained in version control ([https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=tree;f=files/misc/scripts;hb=HEAD operations/puppet : <code>files/misc/scripts</code>]) and copied/synced to servers (such as [[fenari]]) by puppet into <code>/usr/local/bin</code>.
  
−
== Operating on current server ==
+
== bin ==
−
* sync-common
+
=== <code>apache-graceful</code> ===
−
** syncs /home/wikipedia/common -> /apache/common on ''this server''
+
: ''Performs the action '''only on the one server''' from which the command is ran''
  
−
== Operating on mediawiki-installation boxes via dsh ==
+
=== <code>apache-graceful-all</code> ===
−
These scripts all sync to: apaches, image scalers, and misc other hosts running apache (search servers, live html snapshot browsers, the secure server, etc)
+
: ''This script operates on all servers in the following dsh groups: '''apaches'''
−
* sync-common-all
+
* Runs <code>apache-graceful</code> locally on all apaches.
−
** syncs /home/wikipedia/common -> /apache/common
+
−
* sync-common-file
+
−
** syncs a given file from /home/wikipedia/common -> /apache/common
+
−
* sync-dblist
+
−
** syncs *.dblist from /home/wikipedia/common -> /apache/common
+
−
* sync-docroot
+
−
** syncs /home/wikipedia/common/docroot -> /apache/common/docroot
+
−
* sync-file
+
−
** Checks PHP syntax and syncs a given file from /home/wikipedia/common/wmf-deployment -> /apache/common/wmf-deployment
+
−
* scap
+
−
** Checks PHP syntax, rebuilds localization caches and texvc, and syncs /home/wikipedia/common/wmf-deployment -> /apache/common/wmf-deployment
+
  
−
== Operating on apaches and image_scalers dsh groups ==
+
=== <code>sync-apache</code> ===
−
* sync-apache
+
: ''This script operates on all servers in the following dsh groups: '''apaches, image_scalers, snapshot, searchidx'''
−
** Syncs /home/wikipedia/conf/httpd to /apache/conf
+
: '''Note:''' The conf files are outside of <code>/home/wikipedia/''common''</code>, and as such are '''not''' synced by the <code>sync-common</code>-like scripts.
−
*** ''note these files are not in /home/wikipedia/common and are *not* copied by sync-common family''
+
* syncs <code>/home/wikipedia/conf/httpd</code> to <code>/apache/conf</code>
  
 +
=== <code>sync-common</code> ===
 +
: ''Performs the action '''only on the one server''' from which the command is ran''
 +
* syncs <code>/home/wikipedia/common</code> -> <code>/apache/common</code>
  
−
== Mysterious or obsolete things? ==
+
=== <code>sync-common-all</code> ===
 +
: ''This is an [[#All-script|all-script]]!''
 +
* <s>syncs <code>/home/wikipedia/common</code> -> <code>/apache/common</code></s><br/>=> Runs [[#scap|scap]]
 +
 
 +
=== <code>sync-common-file</code> ===
 +
: ''This is an [[#All-script|all-script]]!''
 +
* for a single directory or file
 +
* syncs <code>/home/wikipedia/common/ (some path)</code> -> <code>/apache/common</code>
 +
 
 +
=== <code>sync-dblist</code> ===
 +
: ''This is an [[#All-script|all-script]]!''
 +
* syncs <code>/home/wikipedia/common/*.dblist</code> -> <code>/apache/common</code>
 +
 
 +
=== <code>sync-docroot</code> ===
 +
: ''This is an [[#All-script|all-script]]!''
 +
* syncs <code>/home/wikipedia/common/docroot</code> -> <code>/apache/common/docroot</code>
 +
 
 +
=== <code>sync-dir</code> ===
 +
: ''This is an [[#All-script|all-script]]!''
 +
* for a single directory
 +
* checks PHP syntax
 +
* syncs <code>/home/wikipedia/common/ (some dir)</code> -> <code>/apache/common</code> with <code>[[#sync-common-file|sync-common-file]]</code>
 +
 
 +
=== <code>sync-file</code> ===
 +
: ''This is an [[#All-script|all-script]]!''
 +
* for a single file
 +
* checks PHP syntax
 +
* syncs <code>/home/wikipedia/common/ (some file)</code> -> <code>/apache/common</code> with <code>[[#sync-common-file|sync-common-file]]</code>
 +
 
 +
=== <code>sync-wikiversions</code>===
 +
: ''This script operates on all servers in the following dsh groups: '''mediawiki-installation'''
 +
* syncs <code>common/wikiversions.dat</code> -> <code>/apache/common-local/wikiversions.dat</code>
 +
* syncs <code>common/wikiversions.cdb</code> -> <code>/apache/common-local/wikiversions.cdb</code>
 +
 
 +
=== <code>scap</code> ===
 +
: ''This is an [[#All-script|all-script]]!''
 +
{{shortcut|scap}}
 +
* lints PHP files in <code>./wmf-config</code>, <code>./multiversion</code> and <code>php-*</code>
 +
* for all mediawiki versions currently deployed (usually 2) it:
 +
** rebuilds localization caches of core and extensions
 +
** recompiles texvc
 +
* runs [[sync-wikiversions]]
 +
* syncs <code>/home/wikipedia/common/php-{versions}</code> -> <code>/apache/common/php-{versions}</code>
 +
* <small>(does it also sync other directories in <code>/common</code> such as <code>live-1.5</code> and <code>docroot</code>?)</small>
 +
 
 +
== Unsorted ==
 +
: These are outdated, redundant, broken or otherwise undocumented:
 +
 
* scap14, scap14-1, scap14-2
 
* scap14, scap14-1, scap14-2
 
** obsolete from MediaWiki 1.4 days
 
** obsolete from MediaWiki 1.4 days
Line 39: Line 79:
 
* sync-common-php
 
* sync-common-php
 
* sync-common-php-x
 
* sync-common-php-x
−
 
−
== No idea what these are ==
 
 
* sync-mirror-dag.sh
 
* sync-mirror-dag.sh
 
* sync-mirror-fedora.sh
 
* sync-mirror-fedora.sh
Line 47: Line 85:
 
* sync-cgibin
 
* sync-cgibin
  
−
[[Category:Software Updates]]
+
== All-script ==
−
[[Category:Operations]]
+
An '''all-script''' is a script that operates on mediawiki-installation boxes via dsh. ''These scripts performs the action on '''all''' servers.''
 +
These servers to be exact:
 +
* [[apache]]s
 +
* [[image scalers]]
 +
* misc other hosts running apache:
 +
** [[search]] servers
 +
** live html snapshot browsers
 +
** the [[secure]] server
 +
** ''etc.''
 +
 
 +
== Examples ==
 +
=== <code>sync-apache</code> example ===
 +
# On [[fenari]]:
 +
# (optionally) run:<br/><code>sync-apache-simulated</code><br/>This just has an "-n" added to the [[rsync]] commands. (<code>--dry-run | </code> show what would have been transferred)
 +
# If that looks fine, actually sync:<br/><code>sync-apache</code>
 +
# Connect to a single random "srv" server, confirm the expected change has arrived here, and restart [[Apache]] to check your change did not break things.
 +
## (how to restart that apache?)
 +
# If that also looks fine, restart all Apaches:<br/><code>apache-graceful-all</code>
 +
 
 +
After this you might want to [[Purge URL|purge a URL]] from [[Squid]] cache. Also see [[Squids#Purge a given external (non WMF) url]].
 +
 
 +
In most cases this is enough. However if the change involves more than a simple change to redirects, [[depooling]]/ repooling may be necessary. Make sure you find out whether this is needed ahead of time{{#ifexist: depooling ||&nbsp;(while at it, start the [[depooling]] page and link to it, we need documentation!)}}.
 +
 
 +
== See also ==
 +
* [[Heterogeneous deployment]]
 +
 
 +
[[Category:Software]]
 
[[Category:How-To]]
 
[[Category:How-To]]

Latest revision as of 08:45, 27 February 2013

Contents

The following executable binaries are maintained in version control (operations/puppet : files/misc/scripts) and copied/synced to servers (such as fenari) by puppet into /usr/local/bin.

[edit] bin

[edit] apache-graceful

Performs the action only on the one server from which the command is ran

[edit] apache-graceful-all

This script operates on all servers in the following dsh groups: apaches
  • Runs apache-graceful locally on all apaches.

[edit] sync-apache

This script operates on all servers in the following dsh groups: apaches, image_scalers, snapshot, searchidx
Note: The conf files are outside of /home/wikipedia/common, and as such are not synced by the sync-common-like scripts.
  • syncs /home/wikipedia/conf/httpd to /apache/conf

[edit] sync-common

Performs the action only on the one server from which the command is ran
  • syncs /home/wikipedia/common -> /apache/common

[edit] sync-common-all

This is an all-script!
  • syncs /home/wikipedia/common -> /apache/common
    => Runs scap

[edit] sync-common-file

This is an all-script!
  • for a single directory or file
  • syncs /home/wikipedia/common/ (some path) -> /apache/common

[edit] sync-dblist

This is an all-script!
  • syncs /home/wikipedia/common/*.dblist -> /apache/common

[edit] sync-docroot

This is an all-script!
  • syncs /home/wikipedia/common/docroot -> /apache/common/docroot

[edit] sync-dir

This is an all-script!
  • for a single directory
  • checks PHP syntax
  • syncs /home/wikipedia/common/ (some dir) -> /apache/common with sync-common-file

[edit] sync-file

This is an all-script!
  • for a single file
  • checks PHP syntax
  • syncs /home/wikipedia/common/ (some file) -> /apache/common with sync-common-file

[edit] sync-wikiversions

This script operates on all servers in the following dsh groups: mediawiki-installation
  • syncs common/wikiversions.dat -> /apache/common-local/wikiversions.dat
  • syncs common/wikiversions.cdb -> /apache/common-local/wikiversions.cdb

[edit] scap

This is an all-script!
shortcut: scap
  • lints PHP files in ./wmf-config, ./multiversion and php-*
  • for all mediawiki versions currently deployed (usually 2) it:
    • rebuilds localization caches of core and extensions
    • recompiles texvc
  • runs sync-wikiversions
  • syncs /home/wikipedia/common/php-{versions} -> /apache/common/php-{versions}
  • (does it also sync other directories in /common such as live-1.5 and docroot?)

[edit] Unsorted

These are outdated, redundant, broken or otherwise undocumented:
  • scap14, scap14-1, scap14-2
    • obsolete from MediaWiki 1.4 days
  • scap15, scap15-1, scap15-1skins
    • 'scap15' is actual script for 'scap' currently; probably could use cleanup
  • sync-file13, sync-file14
  • sync-file-log14 ?
  • sync-to-seoul
  • sync-common-allphp
  • sync-common-clean
  • sync-common-it
  • sync-common-php
  • sync-common-php-x
  • sync-mirror-dag.sh
  • sync-mirror-fedora.sh
  • sync-users-and-groups.sh
  • syncout
  • sync-cgibin

[edit] All-script

An all-script is a script that operates on mediawiki-installation boxes via dsh. These scripts performs the action on all servers. These servers to be exact:

[edit] Examples

[edit] sync-apache example

  1. On fenari:
  2. (optionally) run:
    sync-apache-simulated
    This just has an "-n" added to the rsync commands. (--dry-run | show what would have been transferred)
  3. If that looks fine, actually sync:
    sync-apache
  4. Connect to a single random "srv" server, confirm the expected change has arrived here, and restart Apache to check your change did not break things.
    1. (how to restart that apache?)
  5. If that also looks fine, restart all Apaches:
    apache-graceful-all

After this you might want to purge a URL from Squid cache. Also see Squids#Purge a given external (non WMF) url.

In most cases this is enough. However if the change involves more than a simple change to redirects, depooling/ repooling may be necessary. Make sure you find out whether this is needed ahead of time (while at it, start the depooling page and link to it, we need documentation!).

[edit] See also

Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Wikimedia_binaries&oldid=56876"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox