Apaches

From Wikitech
(Difference between revisions)
Jump to: navigation, search
 
(→‎All)
 
(70 intermediate revisions by 27 users not shown)
Line 1: Line 1:
 +
{{Old-doc}}
  
−
Apaches log are either disabled or deleted. Webalizer statistics are made using the [[squids]] logs instead.
+
Apparently not :-P
 +
 
 +
The Apache configs are both in puppet and in /home/wikipedia/conf/httpd. This guide mostly cover the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track change.
 +
 
 +
==Deploying config==
 +
 
 +
==via gerrit (recommended)==
 +
 
 +
#submit change to [[gerrit]] (project: operations/apache-config)
 +
#merge via gerrit
 +
#'git pull origin master' on [[fenari]]  (in /h/w/conf/httpd/)
 +
#run sync-apache (also see [[Sync_scripts#Operating_on_apaches_and_image_scalers_dsh_groups|sync scripts]])
 +
 
 +
{{warning}} This list does not include a check if the config does not break Apache! You can do this by just syncing to a single server and restarting that manually before pushing out to all. There are also test scripts to check multiple URLs to see if redirect changes work. You can also use [[curl]] to ask a single server for a specific virtual host from fenari shell to confirm a change works before pushing it out.
 +
 
 +
#apache-graceful-all
 +
#test change from external
 +
 
 +
===local hacking===
 +
 
 +
git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache
 +
 
 +
==Restarting==
 +
 
 +
===All===
 +
Connect to [[fenari]] and type <code>./[[apache-graceful-all]]</code>.  This will restart all apache processes for servers in the apaches dsh node group.  You must be root to do this.
 +
 
 +
===One, to test a change===
 +
Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type ''apache-graceful'' to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:
 +
 
 +
<br>''telnet localhost 80''
 +
<br>Trying 127.0.0.1...
 +
<br>Connected to localhost.
 +
<br>Escape character is '^]'.
 +
<br>''GET /fundraising HTTP/1.1''
 +
<br>''Host:wikimediafoundation.org''
 +
<br>''(an empty line)''
 +
 
 +
The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.
 +
 
 +
==Logging==
 +
 
 +
Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.
 +
 
 +
Apache access logs are mostly disabled. Webalizer statistics are made using the [[squids]] logs instead.
 +
 
 +
Text output on stderr from various external commands is in various log files in <tt>/home/wikipedia/logs</tt> on fenari. PHP errors are in <tt>/home/wikipedia/logs/syslog/syslog</tt> on fenari.
 +
 
 +
==Apache setup checklist==
 +
 
 +
* Follow the [[Automated installation]] instructions for the base install
 +
* Run the following on the server:
 +
:* <tt>apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit </tt>
 +
* Wait for the server to come back online, ensure it starts apache correctly
 +
** <tt>echo 'GET /' | nc localhost 80</tt> or any of the number of tests listed below
 +
* If the server is part of the memcached group, follow instructions on [[Memcached]]
 +
* Run the setup of [[Ganglia]]
 +
* If the server is new, you will need to do the following:
 +
:* Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
 +
* If the server is not new do the following:
 +
:* Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
 +
* You will need to add the server to [[DSH]] groups if new, or check if they are commented, if the server is not new:
 +
:* Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
 +
:* Reload nagios to accept the changes to the node groups:
 +
::* <tt>cd /home/wikipedia/conf/nagios && ./sync </tt>
 +
* Verify that the server is tacking traffic and doing work
 +
:* <tt>ipvsadm -L | grep SERVERNAME </tt>
 +
:* traffic logs?
 +
 
 +
==Test cases==
 +
 
 +
Here are some test cases you can use to test the apache configuration after changing something.
 +
 
 +
<pre>
 +
GET /wiki/Foo HTTP/1.1
 +
Host: en.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET /wiki/Foo HTTP/1.1
 +
Host: www.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: en2.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET /wiki/Main_Page HTTP/1.1
 +
Host: www.wikipedia.com
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: wikipedia.com
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: wikibooks.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: wikiquote.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: dk.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: foo.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET /wiki/Main_Page HTTP/1.1
 +
Host: test.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: webshop.wikipedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: boards.wikimedia.org
 +
User-agent: testthing
 +
 
 +
GET /wiki/Foo HTTP/1.1
 +
Host: en.wikipedia.org
 +
User-Agent: Exalead
 +
 
 +
GET /wiki/Foo HTTP/1.1
 +
Host: meta.wikimedia.org
 +
User-agent: testthing
 +
 
 +
GET / HTTP/1.1
 +
Host: en.wiktionary.org
 +
User-agent: testthing
 +
 
 +
 
 +
</pre>
 +
[[Category:Servers by usage| Apache]]

Latest revision as of 00:12, 10 August 2012

This page page may be outdated and could contain incorrect details. Please update it if you can.

Apparently not :-P

The Apache configs are both in puppet and in /home/wikipedia/conf/httpd. This guide mostly cover the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track change.

Contents

[edit] Deploying config

[edit] via gerrit (recommended)

  1. submit change to gerrit (project: operations/apache-config)
  2. merge via gerrit
  3. 'git pull origin master' on fenari (in /h/w/conf/httpd/)
  4. run sync-apache (also see sync scripts)

Dialog-warning.svg This list does not include a check if the config does not break Apache! You can do this by just syncing to a single server and restarting that manually before pushing out to all. There are also test scripts to check multiple URLs to see if redirect changes work. You can also use curl to ask a single server for a specific virtual host from fenari shell to confirm a change works before pushing it out.

  1. apache-graceful-all
  2. test change from external

[edit] local hacking

git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache

[edit] Restarting

[edit] All

Connect to fenari and type ./apache-graceful-all. This will restart all apache processes for servers in the apaches dsh node group. You must be root to do this.

[edit] One, to test a change

Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type apache-graceful to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:


telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /fundraising HTTP/1.1
Host:wikimediafoundation.org
(an empty line)

The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.

[edit] Logging

Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.

Apache access logs are mostly disabled. Webalizer statistics are made using the squids logs instead.

Text output on stderr from various external commands is in various log files in /home/wikipedia/logs on fenari. PHP errors are in /home/wikipedia/logs/syslog/syslog on fenari.

[edit] Apache setup checklist

  • apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit
  • Wait for the server to come back online, ensure it starts apache correctly
    • echo 'GET /' | nc localhost 80 or any of the number of tests listed below
  • If the server is part of the memcached group, follow instructions on Memcached
  • Run the setup of Ganglia
  • If the server is new, you will need to do the following:
  • Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
  • If the server is not new do the following:
  • Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
  • You will need to add the server to DSH groups if new, or check if they are commented, if the server is not new:
  • Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
  • Reload nagios to accept the changes to the node groups:
  • cd /home/wikipedia/conf/nagios && ./sync
  • Verify that the server is tacking traffic and doing work
  • ipvsadm -L | grep SERVERNAME
  • traffic logs?

[edit] Test cases

Here are some test cases you can use to test the apache configuration after changing something. 

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: www.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en2.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: www.wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikibooks.org
User-agent: testthing

GET / HTTP/1.1
Host: wikiquote.org
User-agent: testthing

GET / HTTP/1.1
Host: dk.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: foo.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: test.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: webshop.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: boards.wikimedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-Agent: Exalead

GET /wiki/Foo HTTP/1.1
Host: meta.wikimedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en.wiktionary.org
User-agent: testthing
Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Apaches&oldid=50053"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox