Apaches

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(logging)
(→‎All)
 
(60 intermediate revisions by 24 users not shown)
Line 1: Line 1:
−
__TOC__
+
{{Old-doc}}
 +
 
 +
Apparently not :-P
 +
 
 +
The Apache configs are both in puppet and in /home/wikipedia/conf/httpd. This guide mostly cover the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track change.
 +
 
 +
==Deploying config==
 +
 
 +
==via gerrit (recommended)==
 +
 
 +
#submit change to [[gerrit]] (project: operations/apache-config)
 +
#merge via gerrit
 +
#'git pull origin master' on [[fenari]]  (in /h/w/conf/httpd/)
 +
#run sync-apache (also see [[Sync_scripts#Operating_on_apaches_and_image_scalers_dsh_groups|sync scripts]])
 +
 
 +
{{warning}} This list does not include a check if the config does not break Apache! You can do this by just syncing to a single server and restarting that manually before pushing out to all. There are also test scripts to check multiple URLs to see if redirect changes work. You can also use [[curl]] to ask a single server for a specific virtual host from fenari shell to confirm a change works before pushing it out.
 +
 
 +
#apache-graceful-all
 +
#test change from external
 +
 
 +
===local hacking===
 +
 
 +
git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache
 +
 
 
==Restarting==
 
==Restarting==
  
 
===All===
 
===All===
−
Connect to zwinger and type ''apache-graceful-all''. If you see permission denied or a password request, say which server name appears in the error message or password request in the IRC channel and one of the roots will add you to that server.
+
Connect to [[fenari]] and type <code>./[[apache-graceful-all]]</code>. This will restart all apache processes for servers in the apaches dsh node group.  You must be root to do this.
  
 
===One, to test a change===
 
===One, to test a change===
−
Either connect to zwinger and ssh wherever or connect to the web server you want to test on. Then type ''apache-graceful'' to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:
+
Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type ''apache-graceful'' to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:
  
 
<br>''telnet localhost 80''
 
<br>''telnet localhost 80''
Line 20: Line 43:
 
==Logging==
 
==Logging==
  
−
Text output on stderr from various external commands, as well as some apache errors, are in <tt>/apache/logs/error_log</tt> on each server. PHP errors are in <tt>/apache/logs/php-errors</tt> . Most apache error messages, such as file not found errors, are in <tt>''hostname''-error_log</tt>, for example <tt>/apache/logs/en.wikipedia.org-error_log</tt> .  
+
Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.
  
 
Apache access logs are mostly disabled. Webalizer statistics are made using the [[squids]] logs instead.
 
Apache access logs are mostly disabled. Webalizer statistics are made using the [[squids]] logs instead.
  
 +
Text output on stderr from various external commands is in various log files in <tt>/home/wikipedia/logs</tt> on fenari. PHP errors are in <tt>/home/wikipedia/logs/syslog/syslog</tt> on fenari.
  
 
==Apache setup checklist==
 
==Apache setup checklist==
−
===main apache farm===
 
−
* install apache from <tt>/home/wikipedia/src/apache_1.3.29</tt>
 
−
** doesn't work on ariel
 
−
* install PHP from <tt>/home/wikipedia/src/php-4.3.4</tt> (is this the right version?)
 
−
** doesn't work on ariel
 
−
** how to setup Turck MMcache?
 
−
* tex: apt-get install tetex*
 
−
* tidy: apt-get install tidy
 
−
* ploticus: scp /usr/local/bin/pl from one of the other machines to /usr/local/bin/
 
−
* imagemagick
 
−
* sudo
 
−
* database permissions
 
−
**Command to run in mysql shell (on master) to allow connections from the new apache :
 
−
***grant all on %wiki.* to wikiuser@apachename identified by 'password';
 
−
***grant all on %wiktionary.* to wikiuser@apachename identified by 'password';
 
−
*** what about webshop?? and Board ??
 
  
−
===download.wikimedia.org etc===
+
* Follow the [[Automated installation]] instructions for the base install
−
* currently on zwinger
+
* Run the following on the server:
−
* blah blah
+
:* <tt>apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit </tt>
 +
* Wait for the server to come back online, ensure it starts apache correctly
 +
** <tt>echo 'GET /' | nc localhost 80</tt> or any of the number of tests listed below
 +
* If the server is part of the memcached group, follow instructions on [[Memcached]]
 +
* Run the setup of [[Ganglia]]
 +
* If the server is new, you will need to do the following:
 +
:* Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
 +
* If the server is not new do the following:
 +
:* Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
 +
* You will need to add the server to [[DSH]] groups if new, or check if they are commented, if the server is not new:
 +
:* Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
 +
:* Reload nagios to accept the changes to the node groups:
 +
::* <tt>cd /home/wikipedia/conf/nagios && ./sync </tt>
 +
* Verify that the server is tacking traffic and doing work
 +
:* <tt>ipvsadm -L | grep SERVERNAME </tt>
 +
:* traffic logs?
 +
 
 +
==Test cases==
 +
 
 +
Here are some test cases you can use to test the apache configuration after changing something.
 +
 
 +
<pre>
 +
GET /wiki/Foo HTTP/1.1
 +
Host: en.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET /wiki/Foo HTTP/1.1
 +
Host: www.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: en2.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET /wiki/Main_Page HTTP/1.1
 +
Host: www.wikipedia.com
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: wikipedia.com
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: wikibooks.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: wikiquote.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: dk.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: foo.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET /wiki/Main_Page HTTP/1.1
 +
Host: test.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: webshop.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: boards.wikimedia.org
 +
User-agent: testthing
 +
 
 +
GET /wiki/Foo HTTP/1.1
 +
Host: en.wikipedia.org
 +
User-Agent: Exalead
 +
 
 +
GET /wiki/Foo HTTP/1.1
 +
Host: meta.wikimedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: en.wiktionary.org
 +
User-agent: testthing
 +
 
 +
 
 +
</pre>
 +
[[Category:Servers by usage| Apache]]

Latest revision as of 00:12, 10 August 2012

This page page may be outdated and could contain incorrect details. Please update it if you can.

Apparently not :-P

The Apache configs are both in puppet and in /home/wikipedia/conf/httpd. This guide mostly cover the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track change.

Contents

[edit] Deploying config

[edit] via gerrit (recommended)

  1. submit change to gerrit (project: operations/apache-config)
  2. merge via gerrit
  3. 'git pull origin master' on fenari (in /h/w/conf/httpd/)
  4. run sync-apache (also see sync scripts)

Dialog-warning.svg This list does not include a check if the config does not break Apache! You can do this by just syncing to a single server and restarting that manually before pushing out to all. There are also test scripts to check multiple URLs to see if redirect changes work. You can also use curl to ask a single server for a specific virtual host from fenari shell to confirm a change works before pushing it out.

  1. apache-graceful-all
  2. test change from external

[edit] local hacking

git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache

[edit] Restarting

[edit] All

Connect to fenari and type ./apache-graceful-all. This will restart all apache processes for servers in the apaches dsh node group. You must be root to do this.

[edit] One, to test a change

Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type apache-graceful to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:


telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /fundraising HTTP/1.1
Host:wikimediafoundation.org
(an empty line)

The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.

[edit] Logging

Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.

Apache access logs are mostly disabled. Webalizer statistics are made using the squids logs instead.

Text output on stderr from various external commands is in various log files in /home/wikipedia/logs on fenari. PHP errors are in /home/wikipedia/logs/syslog/syslog on fenari.

[edit] Apache setup checklist

  • apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit
  • Wait for the server to come back online, ensure it starts apache correctly
    • echo 'GET /' | nc localhost 80 or any of the number of tests listed below
  • If the server is part of the memcached group, follow instructions on Memcached
  • Run the setup of Ganglia
  • If the server is new, you will need to do the following:
  • Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
  • If the server is not new do the following:
  • Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
  • You will need to add the server to DSH groups if new, or check if they are commented, if the server is not new:
  • Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
  • Reload nagios to accept the changes to the node groups:
  • cd /home/wikipedia/conf/nagios && ./sync
  • Verify that the server is tacking traffic and doing work
  • ipvsadm -L | grep SERVERNAME
  • traffic logs?

[edit] Test cases

Here are some test cases you can use to test the apache configuration after changing something. 

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: www.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en2.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: www.wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikibooks.org
User-agent: testthing

GET / HTTP/1.1
Host: wikiquote.org
User-agent: testthing

GET / HTTP/1.1
Host: dk.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: foo.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: test.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: webshop.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: boards.wikimedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-Agent: Exalead

GET /wiki/Foo HTTP/1.1
Host: meta.wikimedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en.wiktionary.org
User-agent: testthing
Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Apaches&oldid=50053"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox