Fundraising Analytics/Impression Stats
(→udp2log proxy log collection) |
|||
| Line 2: | Line 2: | ||
== udp2log proxy log collection == | == udp2log proxy log collection == | ||
| + | udp2log is configured via two entries in <code>locke:/etc/udp2log/squid</code>: | ||
| − | |||
| − | |||
<pre> | <pre> | ||
| − | |||
| − | |||
# Landing pages | # Landing pages | ||
pipe 1 /a/squid/fundraising/lp-filter >> /a/squid/fundraising/logs/landingpages.log | pipe 1 /a/squid/fundraising/lp-filter >> /a/squid/fundraising/logs/landingpages.log | ||
# Banner Impressions | # Banner Impressions | ||
| − | pipe | + | pipe 100 /a/squid/fundraising/bi-filter >> /a/squid/fundraising/logs/bannerImpressions-sampled100.log |
| − | + | ||
</pre> | </pre> | ||
| − | + | To enable/disable, uncomment/comment these lines and then HUP udp2log: | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | |||
<pre> | <pre> | ||
awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log | awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log | ||
| − | |||
</pre> | </pre> | ||
Revision as of 18:48, 6 September 2012
Banner impressions and landing page stats are collected from Squid logs via udp2log running on Locke. Every 15 minutes a cron job, running via file_mover@locke's crontab, rotates the log files to a local buffer directory where they're retained for 7 days. The script also copies the files via nfs to the local NetApp nas1-a.pmtpa.wmnet, which is mirrored offsite to nas1001-a.eqiad.wmnet. Finally, the NetApps are also nfs-mounted to grosley/aluminium where files are parsed by analytics scripts.
udp2log proxy log collection
udp2log is configured via two entries in locke:/etc/udp2log/squid:
# Landing pages pipe 1 /a/squid/fundraising/lp-filter >> /a/squid/fundraising/logs/landingpages.log # Banner Impressions pipe 100 /a/squid/fundraising/bi-filter >> /a/squid/fundraising/logs/bannerImpressions-sampled100.log
To enable/disable, uncomment/comment these lines and then HUP udp2log:
awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log
proxy log rotation and archiving
To enable:
Enable this crontab entry for file_mover@locke:
*/15 * * * * /home/file_mover/scripts/rotate_fundraising_logs
To disable:
Comment out this crontab entry for file_mover@locke:
#*/15 * * * * /home/file_mover/scripts/rotate_fundraising_logs
monitoring and debugging
The cron script logs verbosely and locke:/var/log/syslog will show you actions and errors.
Under normal operation, you should see this sequence:
Sep 6 17:45:01 locke CRON[28592]: (file_mover) CMD (/home/file_mover/scripts/rotate_fundraising_logs) Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/landingpages.log to /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/bannerImpressions-sampled100.log to /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: reload udp2log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: rsync -ar /a/squid/fundraising/logs/buffer/ /a/squid/fundraising/logs/fr_archive/ Sep 6 17:45:02 locke rotate_fundraising_logs[28594]: done!
Note that /a/squid/fundraising/logs/fr_archive is the permanent storage location on the netapp.
