Fundraising Analytics/Impression Stats
| Line 1: | Line 1: | ||
| − | Banner impressions and landing page stats are collected from Squid logs via [[udp2log]] running on [[Locke]]. Every 15 minutes a cron job, running via file_mover@locke's crontab, rotates the log files to a local buffer directory where they're retained for 7 days. The script also copies the files via nfs to the local NetApp [[nas1-a.pmtpa.wmnet]], which is mirrored offsite to [[nas1001-a.eqiad.wmnet]]. | + | Banner impressions and landing page stats are collected from Squid logs via [[udp2log]] running on [[Locke]]. Every 15 minutes a cron job, running via file_mover@locke's crontab, rotates the log files to a local buffer directory where they're retained for 7 days. The script also copies the files via nfs to the local NetApp [[nas1-a.pmtpa.wmnet]], which is mirrored offsite to [[nas1001-a.eqiad.wmnet]]. Finally, the NetApps are also nfs-mounted to grosley/aluminium where files are parsed by analytics scripts.<br> |
| − | + | ||
| − | Finally, the NetApps are also nfs-mounted to grosley/aluminium where files are parsed by analytics scripts.<br> | + | |
== udp2log proxy log collection == | == udp2log proxy log collection == | ||
| + | |||
<b>To enable</b><br> | <b>To enable</b><br> | ||
ssh to locke and uncomment fundraising-related lines in <code>/etc/udp2log/squid</code> to look like this: | ssh to locke and uncomment fundraising-related lines in <code>/etc/udp2log/squid</code> to look like this: | ||
| Line 32: | Line 31: | ||
</pre> | </pre> | ||
| − | == proxy log | + | == proxy log rotation and archiving == |
<b>To enable:</b><br> | <b>To enable:</b><br> | ||
| Line 48: | Line 47: | ||
</pre> | </pre> | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
== monitoring and debugging == | == monitoring and debugging == | ||
Revision as of 18:42, 6 September 2012
Banner impressions and landing page stats are collected from Squid logs via udp2log running on Locke. Every 15 minutes a cron job, running via file_mover@locke's crontab, rotates the log files to a local buffer directory where they're retained for 7 days. The script also copies the files via nfs to the local NetApp nas1-a.pmtpa.wmnet, which is mirrored offsite to nas1001-a.eqiad.wmnet. Finally, the NetApps are also nfs-mounted to grosley/aluminium where files are parsed by analytics scripts.
udp2log proxy log collection
To enable
ssh to locke and uncomment fundraising-related lines in /etc/udp2log/squid to look like this:
... ## Fundraising # Landing pages pipe 1 /a/squid/fundraising/lp-filter >> /a/squid/fundraising/logs/landingpages.log # Banner Impressions pipe 1 /a/squid/fundraising/bi-filter >> /a/squid/fundraising/logs/bannerImpressions.log ...
Then HUP udp2log:
awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log Sending SIGHUP to udp2log...
To disable
SSH into Locke, and comment fundraising-related lines in /etc/udp2log/squid.
Then HUP udp2log:
awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log Sending SIGHUP to udp2log...
proxy log rotation and archiving
To enable:
Enable this crontab entry for file_mover@locke:
*/15 * * * * /home/file_mover/scripts/rotate_fundraising_logs
To disable:
Comment out this crontab entry for file_mover@locke:
#*/15 * * * * /home/file_mover/scripts/rotate_fundraising_logs
monitoring and debugging
The cron script logs verbosely and locke:/var/log/syslog will show you actions and errors.
Under normal operation, you should see this sequence:
Sep 6 17:45:01 locke CRON[28592]: (file_mover) CMD (/home/file_mover/scripts/rotate_fundraising_logs) Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/landingpages.log to /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/bannerImpressions-sampled100.log to /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: reload udp2log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: rsync -ar /a/squid/fundraising/logs/buffer/ /a/squid/fundraising/logs/fr_archive/ Sep 6 17:45:02 locke rotate_fundraising_logs[28594]: done!
Note that /a/squid/fundraising/logs/fr_archive is the permanent storage location on the netapp.
