Apaches

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(→‎Starting load-balancing ICPAgent)
(→‎All)
 
(23 intermediate revisions by 7 users not shown)
Line 1: Line 1:
−
{{Document me}}
+
{{Old-doc}}
 +
 
 +
Apparently not :-P
 +
 
 +
The Apache configs are both in puppet and in /home/wikipedia/conf/httpd. This guide mostly cover the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track change.
 +
 
 +
==Deploying config==
 +
 
 +
==via gerrit (recommended)==
 +
 
 +
#submit change to [[gerrit]] (project: operations/apache-config)
 +
#merge via gerrit
 +
#'git pull origin master' on [[fenari]]  (in /h/w/conf/httpd/)
 +
#run sync-apache (also see [[Sync_scripts#Operating_on_apaches_and_image_scalers_dsh_groups|sync scripts]])
 +
 
 +
{{warning}} This list does not include a check if the config does not break Apache! You can do this by just syncing to a single server and restarting that manually before pushing out to all. There are also test scripts to check multiple URLs to see if redirect changes work. You can also use [[curl]] to ask a single server for a specific virtual host from fenari shell to confirm a change works before pushing it out.
 +
 
 +
#apache-graceful-all
 +
#test change from external
 +
 
 +
===local hacking===
 +
 
 +
git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache
  
−
__TOC__
 
 
==Restarting==
 
==Restarting==
  
 
===All===
 
===All===
−
Connect to zwinger and type ''apache-graceful-all''. If you see permission denied or a password request, say which server name appears in the error message or password request in the IRC channel and one of the roots will add you to that server.
+
Connect to [[fenari]] and type <code>./[[apache-graceful-all]]</code>. This will restart all apache processes for servers in the apaches dsh node group.  You must be root to do this.
  
 
===One, to test a change===
 
===One, to test a change===
−
Either connect to zwinger and ssh wherever or connect to the web server you want to test on. Then type ''apache-graceful'' to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:
+
Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type ''apache-graceful'' to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:
  
 
<br>''telnet localhost 80''
 
<br>''telnet localhost 80''
Line 19: Line 40:
  
 
The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.
 
The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.
−
 
−
==Restarting httpd==
 
−
If httpd is broken, ''sudo apache-restart'' to restart a single httpd. Use dsh to apply it to all. If you get hundreds of instances counted by ''ps aux|grep httpd|wc -l'' and the server is having problems you might need to do this.
 
−
 
−
 
−
 
−
==Stopping load-balancing ICPAgent==
 
−
If you need to take the load from this Apache use either:
 
−
*<tt>killall icpagent</tt> as root
 
−
*remote management? as a member of group wikidev
 
  
 
==Logging==
 
==Logging==
  
−
Text output on stderr from various external commands, as well as some apache errors, are in <tt>/apache/logs/error_log</tt> on each server. PHP errors are in <tt>/apache/logs/php-errors</tt> . Most apache error messages, such as file not found errors, are in <tt>''hostname''-error_log</tt>, for example <tt>/apache/logs/en.wikipedia.org-error_log</tt> .  
+
Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.
  
 
Apache access logs are mostly disabled. Webalizer statistics are made using the [[squids]] logs instead.
 
Apache access logs are mostly disabled. Webalizer statistics are made using the [[squids]] logs instead.
  
 +
Text output on stderr from various external commands is in various log files in <tt>/home/wikipedia/logs</tt> on fenari. PHP errors are in <tt>/home/wikipedia/logs/syslog/syslog</tt> on fenari.
  
 
==Apache setup checklist==
 
==Apache setup checklist==
  
−
* Ensure the server is in the netboot.cfg file located in /srv/autoinstall/netboot.cfg on khaldun in the apaches section.
+
* Follow the [[Automated installation]] instructions for the base install
−
* Ensure an entry exists in /etc/dhcpd.conf on zwinger.
+
−
* Netboot the server and install Ubuntu.
+
−
* SSH into the server and set the root password.
+
 
* Run the following on the server:
 
* Run the following on the server:
 
:* <tt>apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit </tt>
 
:* <tt>apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit </tt>
 
* Wait for the server to come back online, ensure it starts apache correctly
 
* Wait for the server to come back online, ensure it starts apache correctly
 +
** <tt>echo 'GET /' | nc localhost 80</tt> or any of the number of tests listed below
 
* If the server is part of the memcached group, follow instructions on [[Memcached]]
 
* If the server is part of the memcached group, follow instructions on [[Memcached]]
 
* Run the setup of [[Ganglia]]
 
* Run the setup of [[Ganglia]]
Line 52: Line 62:
 
* If the server is not new do the following:
 
* If the server is not new do the following:
 
:* Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
 
:* Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
−
* You will need to add the server to DSH groups if new, or check if they are commented, if the server is not new:
+
* You will need to add the server to [[DSH]] groups if new, or check if they are commented, if the server is not new:
 
:* Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
 
:* Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
 
:* Reload nagios to accept the changes to the node groups:
 
:* Reload nagios to accept the changes to the node groups:
 
::* <tt>cd /home/wikipedia/conf/nagios && ./sync </tt>
 
::* <tt>cd /home/wikipedia/conf/nagios && ./sync </tt>
−
 
+
* Verify that the server is tacking traffic and doing work
−
===download.wikimedia.org etc===
+
:* <tt>ipvsadm -L | grep SERVERNAME </tt>
−
* currently on zwinger
+
:* traffic logs?
  
 
==Test cases==
 
==Test cases==
Line 127: Line 137:
  
 
</pre>
 
</pre>
 +
[[Category:Servers by usage| Apache]]

Latest revision as of 00:12, 10 August 2012

This page page may be outdated and could contain incorrect details. Please update it if you can.

Apparently not :-P

The Apache configs are both in puppet and in /home/wikipedia/conf/httpd. This guide mostly cover the later. We used to have a subversion repository tracking the changes, it has been made obsolete on July 11th 2012 so we are now fully using git to track change.

Contents

[edit] Deploying config

[edit] via gerrit (recommended)

  1. submit change to gerrit (project: operations/apache-config)
  2. merge via gerrit
  3. 'git pull origin master' on fenari (in /h/w/conf/httpd/)
  4. run sync-apache (also see sync scripts)

Dialog-warning.svg This list does not include a check if the config does not break Apache! You can do this by just syncing to a single server and restarting that manually before pushing out to all. There are also test scripts to check multiple URLs to see if redirect changes work. You can also use curl to ask a single server for a specific virtual host from fenari shell to confirm a change works before pushing it out.

  1. apache-graceful-all
  2. test change from external

[edit] local hacking

git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache

[edit] Restarting

[edit] All

Connect to fenari and type ./apache-graceful-all. This will restart all apache processes for servers in the apaches dsh node group. You must be root to do this.

[edit] One, to test a change

Either connect to fenari and ssh wherever or connect to the web server you want to test on. Then type apache-graceful to restart apache on that web server only. Test your change with telnet, as with this foundation example, with what you type in italics:


telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /fundraising HTTP/1.1
Host:wikimediafoundation.org
(an empty line)

The raw HTML for the page will now be displayed in your window. You can copy and paste that into a file on your hard drive and open it with your browser to see the effect. Host is the name of the web site after the http:// part in your browser URL area. GET /fundraising is the part after the site name. The example gets http://wikimediafoundation.org/fundraising.

[edit] Logging

Apache errors are logged to /home/wikipedia/logs/syslog/apache.log on fenari.

Apache access logs are mostly disabled. Webalizer statistics are made using the squids logs instead.

Text output on stderr from various external commands is in various log files in /home/wikipedia/logs on fenari. PHP errors are in /home/wikipedia/logs/syslog/syslog on fenari.

[edit] Apache setup checklist

  • apt-get update && apt-get dist-upgrade -y && apt-get install wikimedia-task-appserver && reboot && exit
  • Wait for the server to come back online, ensure it starts apache correctly
    • echo 'GET /' | nc localhost 80 or any of the number of tests listed below
  • If the server is part of the memcached group, follow instructions on Memcached
  • Run the setup of Ganglia
  • If the server is new, you will need to do the following:
  • Login to the LVS server for apaches (lvs3 as of 2009-02-13) and add the new servers to /etc/pybal/apaches
  • If the server is not new do the following:
  • Ensure the server is now enabled in pybal on the LVS server in the file /etc/pybal/apaches
  • You will need to add the server to DSH groups if new, or check if they are commented, if the server is not new:
  • Add/Uncomment the host to /usr/local/dsh/node_groups/apaches and mediawiki-installation, as well as any other groups needed (apaches_pmtpa perhaps?)
  • Reload nagios to accept the changes to the node groups:
  • cd /home/wikipedia/conf/nagios && ./sync
  • Verify that the server is tacking traffic and doing work
  • ipvsadm -L | grep SERVERNAME
  • traffic logs?

[edit] Test cases

Here are some test cases you can use to test the apache configuration after changing something. 

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: www.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en2.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: www.wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikipedia.com
User-agent: testthing

GET / HTTP/1.1
Host: wikibooks.org
User-agent: testthing

GET / HTTP/1.1
Host: wikiquote.org
User-agent: testthing

GET / HTTP/1.1
Host: dk.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: foo.wikipedia.org
User-agent: testthing

GET /wiki/Main_Page HTTP/1.1
Host: test.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: webshop.wikipedia.org
User-agent: testthing

GET / HTTP/1.1
Host: boards.wikimedia.org
User-agent: testthing

GET /wiki/Foo HTTP/1.1
Host: en.wikipedia.org
User-Agent: Exalead

GET /wiki/Foo HTTP/1.1
Host: meta.wikimedia.org
User-agent: testthing

GET / HTTP/1.1
Host: en.wiktionary.org
User-agent: testthing
Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Apaches&oldid=50053"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox