Squid logging
Contents |
Squid side
We use our own UDP logging patch, described here. It is in the Wikimedia APT repository in squid_2.6.9-1wm1. Log lines are buffered and sent to locke in 1450-byte packets.
- Squid log format
- Currently aggregated on locke
Aggregation
UDP packets are sent from all squids to the log host locke.wikimedia.org. A program called udp2log runs there. udp2log logs to an arbitrary number of destination files or pipes, specified in /etc/udp2log. The configuration file format is line based, with each line containing:
pipe <factor> <command line>
or
file <factor> <filename>
Unlike the squid pipe log, these pipes are full shell commands and may contain spaces. There is no quoting, the command goes from after the factor to the end of the line.
<factor> is an integer sampling factor, one in every <factor> packets will be sent to the designated destination.
Comments are lines starting with "#".
Reconfiguration can be done by sending a HUP signal to udp2log. The sequence of events when HUP is received are as follows:
- HUP is received, a flag is set
- Wait for the next UDP packet
- Load the new configuration, open all files and pipes specified
- If all pipes and files were successfully opened, swap in the configuration and close all previously opened pipes and files. Otherwise, close the new pipes and files and retain the old configuration.
- Process the received packet using the new configuration
- Resume the recv() loop
This means that pipe scripts should not obtain any lock on a shared resource until after the first line is received on stdin.
udp2log is designed to run as a daemon for long periods of time with only HUP reloads, not restarts. Restarts should be avoided because they cause packet loss. I haven't quite gotten around to writing the daemonize code yet, so for now it runs in a root screen. It runs as the user "logger".
udp2log is available in svn (http://svn.wikimedia.org/svnroot/mediawiki/trunk/udplog), and in the APT repository in the udplog package.
We have seen cases where udp2log is oversubscribed with filter requests and starts to drop packets. As a result a packless analyser was written by Tim and is documented here.
Inflated Stats
It is very important that any filter that is receiving traffic filters out SpecialPage requests. Our current scheme is to make urls conform to the following format
w/index.php?title=Special:Foo
Otherwise they will be counted as regular page requests and will artificially inflate our page view stats.
Log files
Log files are stored in /a/squid.
- sampled-1000.log
- A 1/1000 sampled log of all requests
- support-requests.log
- Full log click through stats for fund raising stats. Needs to be retained.
- bannerImpressions log
- 1/1 recording of all banner impressions
- landingpages log
- 1/1 recording for a click throughs on banners
hume doesn't have enough space to record the full length of fundraising logs. we've decided to archive our data on storage3 in /archive/udplogs
This section should be updated with details of currently collected log files and aggregation data.
