Apaches

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(→‎main apache farm)
(→‎All)
 
(33 intermediate revisions by 12 users not shown)
Line 1: Line 1:
−
__TOC__
+
{{Old-doc}}
 +
 
 +
Apparently not :-P
 +
 
 +
The Apache configs are both in puppet and in /home/wikipedia/conf/httpd. This guide mostly cover the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track change.
 +
 
 +
==Deploying config==
 +
 
 +
==via gerrit (recommended)==
 +
 
 +
#submit change to [[gerrit]] (project: operations/apache-config)
 +
#merge via gerrit
 +
#'git pull origin master' on [[fenari]]  (in /h/w/conf/httpd/)
 +
#run sync-apache (also see [[Sync_scripts#Operating_on_apaches_and_image_scalers_dsh_groups|sync scripts]])
 +
 
 +
{{warning}} This list does not include a check if the config does not break Apache! You can do this by just syncing to a single server and restarting that manually before pushing out to all. There are also test scripts to check multiple URLs to see if redirect changes work. You can also use [[curl]] to ask a single server for a specific virtual host from fenari shell to confirm a change works before pushing it out.
 +
 
 +
#apache-graceful-all
 +
#test change from external
 +
 
 +
===local hacking===
 +
 
 +
git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache
 +
 
 
==Restarting==
 
==Restarting==
  
 
===All===
 
===All===
−
Connect to zwinger and type ''apache-graceful-all''. If you see permission denied or a password request, say which server name appears in the error message or password request in the IRC channel and one of the roots will add you to that server.
+
Connect to [[fenari]] and type <code>./[[apache-graceful-all]]</code>. This will restart all apache processes for servers in the apaches dsh node group.  You must be root to do this.
  
 
===One, to test a change===
 
===One, to test a change===
−
Either connect to zwinger and ssh wherever or connect to the web server you want to test on. Then type ''apache-graceful'' to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:
+
Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type ''apache-graceful'' to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:
  
 
<br>''telnet localhost 80''
 
<br>''telnet localhost 80''
Line 17: Line 40:
  
 
The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.
 
The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.
−
 
−
==Restarting httpd==
 
−
If httpd is broken, ''sudo apache-restart'' to restart a single httpd. Use dsh to apply it to all. If you get hundreds of instances counted by ''ps aux|grep httpd|wc -l'' and the server is having problems you might need to do this.
 
−
 
−
==Starting load-balancing [[ICP agent|ICPAgent]]==
 
−
This is started automatically when the computer restarts and is necessary for the Apache to take any work. To start:
 
−
*use /usr/local/bin/icpagent, has the right timings for each host
 
−
 
−
==Stopping load-balancing ICPAgent==
 
−
If you need to take the load from this Apache use either:
 
−
*<tt>killall icpagent</tt> as root
 
−
*remote management? as a member of group wikidev
 
  
 
==Logging==
 
==Logging==
  
−
Text output on stderr from various external commands, as well as some apache errors, are in <tt>/apache/logs/error_log</tt> on each server. PHP errors are in <tt>/apache/logs/php-errors</tt> . Most apache error messages, such as file not found errors, are in <tt>''hostname''-error_log</tt>, for example <tt>/apache/logs/en.wikipedia.org-error_log</tt> .  
+
Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.
  
 
Apache access logs are mostly disabled. Webalizer statistics are made using the [[squids]] logs instead.
 
Apache access logs are mostly disabled. Webalizer statistics are made using the [[squids]] logs instead.
  
 +
Text output on stderr from various external commands is in various log files in <tt>/home/wikipedia/logs</tt> on fenari. PHP errors are in <tt>/home/wikipedia/logs/syslog/syslog</tt> on fenari.
  
 
==Apache setup checklist==
 
==Apache setup checklist==
−
===automated procedure===
 
−
New apache deployments (on FC3) may be done with deployment scripts. Before you do anything, please add DNS entries and set up NFS ACLs. Then:
 
  
−
mount 10.0.0.4:/home /home
+
* Follow the [[Automated installation]] instructions for the base install
−
cd /home/wikipedia/deployment/apache
+
* Run the following on the server:
−
sh prepare-host
+
:* <tt>apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit </tt>
−
reboot
+
* Wait for the server to come back online, ensure it starts apache correctly
−
 
+
** <tt>echo 'GET /' | nc localhost 80</tt> or any of the number of tests listed below
−
Hey, yeah, you've got a running apache with all checklists below executed...
+
* If the server is part of the memcached group, follow instructions on [[Memcached]]
−
 
+
* Run the setup of [[Ganglia]]
−
===main apache farm===
+
* If the server is new, you will need to do the following:
−
 
+
:* Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
−
As root, run /home/wikipedia/src/packages/setup-apache
+
* If the server is not new do the following:
−
 
+
:* Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
−
In theory, this should install all dependencies. If steps from below are missing, they should get added to this script...
+
* You will need to add the server to [[DSH]] groups if new, or check if they are commented, if the server is not new:
−
 
+
:* Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
−
* Install apt-get
+
:* Reload nagios to accept the changes to the node groups:
−
* Add server to export lists at zwinger:/etc/exports and yongle:/etc/exports
+
::* <tt>cd /home/wikipedia/conf/nagios && ./sync </tt>
−
* Update fstab, mount file systems
+
* Verify that the server is tacking traffic and doing work
−
* install apache from <tt>/home/wikipedia/src/apache_1.3.33</tt>
+
:* <tt>ipvsadm -L | grep SERVERNAME </tt>
−
** doesn't work on ariel
+
:* traffic logs?
−
* install PHP from <tt>/home/wikipedia/src/php/php-4.3.10</tt> (is this the right version? yes)
+
−
** doesn't work on ariel
+
−
** how to setup Turck MMcache?
+
−
***<tt>/home/wikipedia/src/turck</tt> (choose the right version for the php)
+
−
* tex, tidy, imagemagick: apt-get install tetex*
+
−
apt-get install ImageMagick tetex tetex-latex tetex-fonts tetex-dvips tidy libtidy sendmail-cf
+
−
* for EasyTimeline, install ploticus: scp /usr/local/bin/pl from one of the other machines to /usr/local/bin/
+
−
* sudo
+
−
* configure sendmail
+
−
** in /etc/mail/sendmail.mc, set
+
−
define(`SMART_HOST',`smtp.pmtpa.wmnet')
+
−
** in /etc/mail, call make
+
−
* database permissions
+
−
**Add the host to maintenance/ourusers.php
+
−
**Run the file and source the resulting SQL as root
+
−
* add hostname to appropriate files in <tt>/usr/local/dsh/node_groups/</tt> on zwinger
+
−
* ganglia
+
−
ln -sf /home/wikipedia/gmond/internal_apache.conf /etc/gmond.conf
+
−
* ICPAgent:
+
−
** scp zwinger:/usr/local/bin/icpagent /usr/local/bin
+
−
** add /usr/local/bin/icpagent to /etc/rc.local
+
−
** possibly adjust the weight in /usr/local/bin/icpagent a bit if the default (5.0) isn't right
+
−
* Install librsvg2
+
−
 
+
−
===download.wikimedia.org etc===
+
−
* currently on zwinger
+
  
 
==Test cases==
 
==Test cases==
Line 93: Line 77:
 
GET /wiki/Foo HTTP/1.1
 
GET /wiki/Foo HTTP/1.1
 
Host: en.wikipedia.org
 
Host: en.wikipedia.org
 +
User-agent: testthing
  
 
GET /wiki/Foo HTTP/1.1
 
GET /wiki/Foo HTTP/1.1
 
Host: www.wikipedia.org
 
Host: www.wikipedia.org
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: en2.wikipedia.org
 
Host: en2.wikipedia.org
 +
User-agent: testthing
  
 
GET /wiki/Main_Page HTTP/1.1
 
GET /wiki/Main_Page HTTP/1.1
 
Host: www.wikipedia.com
 
Host: www.wikipedia.com
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: wikipedia.com
 
Host: wikipedia.com
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: wikibooks.org
 
Host: wikibooks.org
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: wikiquote.org
 
Host: wikiquote.org
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: dk.wikipedia.org
 
Host: dk.wikipedia.org
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: foo.wikipedia.org
 
Host: foo.wikipedia.org
 +
User-agent: testthing
  
 
GET /wiki/Main_Page HTTP/1.1
 
GET /wiki/Main_Page HTTP/1.1
 
Host: test.wikipedia.org
 
Host: test.wikipedia.org
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: webshop.wikipedia.org
 
Host: webshop.wikipedia.org
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: boards.wikimedia.org
 
Host: boards.wikimedia.org
 +
User-agent: testthing
  
 
GET /wiki/Foo HTTP/1.1
 
GET /wiki/Foo HTTP/1.1
Line 133: Line 129:
 
GET /wiki/Foo HTTP/1.1
 
GET /wiki/Foo HTTP/1.1
 
Host: meta.wikimedia.org
 
Host: meta.wikimedia.org
 +
User-agent: testthing
  
 
GET / HTTP/1.1
 
GET / HTTP/1.1
 
Host: en.wiktionary.org
 
Host: en.wiktionary.org
−
 
+
User-agent: testthing
  
  
 
</pre>
 
</pre>
 +
[[Category:Servers by usage| Apache]]

Latest revision as of 00:12, 10 August 2012

This page page may be outdated and could contain incorrect details. Please update it if you can.

Apparently not :-P

The Apache configs are both in puppet and in /home/wikipedia/conf/httpd. This guide mostly cover the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track change.

Contents

[edit] Deploying config

[edit] via gerrit (recommended)

  1. submit change to gerrit (project: operations/apache-config)
  2. merge via gerrit
  3. 'git pull origin master' on fenari (in /h/w/conf/httpd/)
  4. run sync-apache (also see sync scripts)

Dialog-warning.svg This list does not include a check if the config does not break Apache! You can do this by just syncing to a single server and restarting that manually before pushing out to all. There are also test scripts to check multiple URLs to see if redirect changes work. You can also use curl to ask a single server for a specific virtual host from fenari shell to confirm a change works before pushing it out.

  1. apache-graceful-all
  2. test change from external

[edit] local hacking

git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache

[edit] Restarting

[edit] All

Connect to fenari and type ./apache-graceful-all. This will restart all apache processes for servers in the apaches dsh node group. You must be root to do this.

[edit] One, to test a change

Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type apache-graceful to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:


telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /fundraising HTTP/1.1
Host:wikimediafoundation.org
(an empty line)

The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.

[edit] Logging

Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.

Apache access logs are mostly disabled. Webalizer statistics are made using the squids logs instead.

Text output on stderr from various external commands is in various log files in /home/wikipedia/logs on fenari. PHP errors are in /home/wikipedia/logs/syslog/syslog on fenari.

[edit] Apache setup checklist

  • apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit
  • Wait for the server to come back online, ensure it starts apache correctly
    • echo 'GET /' | nc localhost 80 or any of the number of tests listed below
  • If the server is part of the memcached group, follow instructions on Memcached
  • Run the setup of Ganglia
  • If the server is new, you will need to do the following:
  • Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
  • If the server is not new do the following:
  • Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
  • You will need to add the server to DSH groups if new, or check if they are commented, if the server is not new:
  • Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
  • Reload nagios to accept the changes to the node groups:
  • cd /home/wikipedia/conf/nagios && ./sync
  • Verify that the server is tacking traffic and doing work
  • ipvsadm -L | grep SERVERNAME
  • traffic logs?

[edit] Test cases

Here are some test cases you can use to test the apache configuration after changing something. 

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: www.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en2.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: www.wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikibooks.org
User-agent: testthing

GET / HTTP/1.1
Host: wikiquote.org
User-agent: testthing

GET / HTTP/1.1
Host: dk.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: foo.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: test.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: webshop.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: boards.wikimedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-Agent: Exalead

GET /wiki/Foo HTTP/1.1
Host: meta.wikimedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en.wiktionary.org
User-agent: testthing
Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Apaches&oldid=50053"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox