Fundraising Analytics/Impression Stats
(→monitoring and debugging) |
|||
| Line 1: | Line 1: | ||
| − | Banner impressions and landing page stats are collected from | + | Banner impressions and landing page stats are collected from Squid logs via [[udp2log]] running on [[Locke]]. Every 15 minutes a cron job, running via file_mover@locke's crontab, rotates the log files to a local buffer directory where they're retained for 7 days. The script also copies the files via nfs to the local NetApp [[nas1-a.pmtpa.wmnet]], which is mirrored offsite to [[nas1001-a.eqiad.wmnet]]. |
| − | + | ||
| − | + | Finally, the NetApps are also nfs-mounted to grosley/aluminium where files are parsed by analytics scripts.<br> | |
| − | Finally, files are parsed | + | |
== udp2log proxy log collection == | == udp2log proxy log collection == | ||
Revision as of 18:37, 6 September 2012
Banner impressions and landing page stats are collected from Squid logs via udp2log running on Locke. Every 15 minutes a cron job, running via file_mover@locke's crontab, rotates the log files to a local buffer directory where they're retained for 7 days. The script also copies the files via nfs to the local NetApp nas1-a.pmtpa.wmnet, which is mirrored offsite to nas1001-a.eqiad.wmnet.
Finally, the NetApps are also nfs-mounted to grosley/aluminium where files are parsed by analytics scripts.
Contents |
udp2log proxy log collection
To enable
ssh to locke and uncomment fundraising-related lines in /etc/udp2log/squid to look like this:
... ## Fundraising # Landing pages pipe 1 /a/squid/fundraising/lp-filter >> /a/squid/fundraising/logs/landingpages.log # Banner Impressions pipe 1 /a/squid/fundraising/bi-filter >> /a/squid/fundraising/logs/bannerImpressions.log ...
Then HUP udp2log:
awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log Sending SIGHUP to udp2log...
To disable
SSH into Locke, and comment fundraising-related lines in /etc/udp2log/squid.
Then HUP udp2log:
awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log Sending SIGHUP to udp2log...
proxy log copy to hume
To enable:
Enable this crontab entry for file_mover@locke:
*/15 * * * * /home/file_mover/scripts/rotate_fundraising_logs
To disable:
Comment out this crontab entry for file_mover@locke:
#*/15 * * * * /home/file_mover/scripts/rotate_fundraising_logs
proxy log compression on storage3
To enable:
Enable this crontab entry for logmover@storage3:
*/5 * * * * /home/logmover/scripts/gzip_incoming_logs.pl
To disable:
Comment out this crontab entry for logmover@storage3:
#*/5 * * * * /home/logmover/scripts/gzip_incoming_logs.pl
monitoring and debugging
The cron script logs verbosely and locke:/var/log/syslog will show you actions and errors.
Under normal operation, you should see this sequence:
Sep 6 17:45:01 locke CRON[28592]: (file_mover) CMD (/home/file_mover/scripts/rotate_fundraising_logs) Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/landingpages.log to /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/bannerImpressions-sampled100.log to /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: reload udp2log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: rsync -ar /a/squid/fundraising/logs/buffer/ /a/squid/fundraising/logs/fr_archive/ Sep 6 17:45:02 locke rotate_fundraising_logs[28594]: done!
Note that /a/squid/fundraising/logs/fr_archive is the permanent storage location on the netapp.
