Robots.txt
From Wikitech
The rewrite rule
RewriteRule ^/robots.txt$ /w/robots.php [L]
sends all requests for robots.txt through robots.php. This script checks whether a page Mediawiki:robots.txt exists on the wiki. If it exists, the content of that page will be sent. Otherwise, the file /apache/common/robots.txt will be sent.
To edit the static file, do the following:
- Edit /home/wikipedia/common/robots.txt
- Run sync-common-file robots.txt
