Fundraising Analytics/Impression Stats
(→udp2log proxy log collection) |
(→proxy log rotation and archiving) |
||
| Line 19: | Line 19: | ||
== proxy log rotation and archiving == | == proxy log rotation and archiving == | ||
| + | Log rotation, compression, and copy to netapp is handled by a cron job: | ||
| − | + | Enable/disable via this crontab entry for file_mover@locke: | |
| − | Enable this crontab entry for file_mover@locke: | + | |
<pre> | <pre> | ||
| + | # rotate and compress fundraising banner impression logs, and archive to netapp | ||
*/15 * * * * /home/file_mover/scripts/rotate_fundraising_logs | */15 * * * * /home/file_mover/scripts/rotate_fundraising_logs | ||
</pre> | </pre> | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
== monitoring and debugging == | == monitoring and debugging == | ||
Revision as of 18:51, 6 September 2012
Banner impressions and landing page stats are collected from Squid logs via udp2log running on Locke. Every 15 minutes a cron job, running via file_mover@locke's crontab, rotates the log files to a local buffer directory where they're retained for 7 days. The script also copies the files via nfs to the local NetApp nas1-a.pmtpa.wmnet, which is mirrored offsite to nas1001-a.eqiad.wmnet. Finally, the NetApps are also nfs-mounted to grosley/aluminium where files are parsed by analytics scripts.
udp2log proxy log collection
udp2log is configured via two entries in locke:/etc/udp2log/squid:
# Landing pages pipe 1 /a/squid/fundraising/lp-filter >> /a/squid/fundraising/logs/landingpages.log # Banner Impressions pipe 100 /a/squid/fundraising/bi-filter >> /a/squid/fundraising/logs/bannerImpressions-sampled100.log
To enable/disable, uncomment/comment these lines and then HUP udp2log:
awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log
proxy log rotation and archiving
Log rotation, compression, and copy to netapp is handled by a cron job:
Enable/disable via this crontab entry for file_mover@locke:
# rotate and compress fundraising banner impression logs, and archive to netapp */15 * * * * /home/file_mover/scripts/rotate_fundraising_logs
monitoring and debugging
The cron script logs verbosely and locke:/var/log/syslog will show you actions and errors.
Under normal operation, you should see this sequence:
Sep 6 17:45:01 locke CRON[28592]: (file_mover) CMD (/home/file_mover/scripts/rotate_fundraising_logs) Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/landingpages.log to /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/bannerImpressions-sampled100.log to /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: reload udp2log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: rsync -ar /a/squid/fundraising/logs/buffer/ /a/squid/fundraising/logs/fr_archive/ Sep 6 17:45:02 locke rotate_fundraising_logs[28594]: done!
Note that /a/squid/fundraising/logs/fr_archive is the permanent storage location on the netapp.
