Fundraising Analytics/Impression Stats
(→analytics processing script) |
m (→monitoring and debugging) |
||
| Line 32: | Line 32: | ||
The cron script logs verbosely and locke:/var/log/syslog will show you actions and errors. | The cron script logs verbosely and locke:/var/log/syslog will show you actions and errors. | ||
| + | |||
| + | Here's what's what in the following examples: | ||
| + | |||
| + | /a/squid/fundraising/logs/*log # active udp2log collection point | ||
| + | /a/squid/fundraising/logs/buffer/2012/*.log # freshly rotated log, before compression | ||
| + | /a/squid/fundraising/logs/fr_archive # netapp nfs mount (i.e. permanent archive location) | ||
Under normal operation, you should see this sequence: | Under normal operation, you should see this sequence: | ||
| Line 46: | Line 52: | ||
</pre> | </pre> | ||
| − | + | Things to watch out for include: | |
| − | + | # move/gzip errors due to local partition overrun, permissions snafu | |
| + | # nfs mount inaccessible | ||
| + | # udp2log HUP fails | ||
[[Category:Fundraising]] | [[Category:Fundraising]] | ||
[[Category:Fundraising_Analytics]] | [[Category:Fundraising_Analytics]] | ||
[[Category:Fundraising - Needs Updated]] | [[Category:Fundraising - Needs Updated]] | ||
Latest revision as of 19:04, 6 September 2012
Banner impressions and landing page stats are collected from Squid logs via udp2log running on Locke. Every 15 minutes a cron job, running via file_mover@locke's crontab, rotates the log files to a local buffer directory where they're retained for 7 days. The script also copies the files via nfs to the local NetApp nas1-a.pmtpa.wmnet, which is mirrored offsite to nas1001-a.eqiad.wmnet. Finally, the NetApps are also nfs-mounted to grosley/aluminium where files are parsed by analytics scripts.
Contents |
[edit] udp2log proxy log collection
udp2log is configured via two entries in locke:/etc/udp2log/squid:
# Landing pages pipe 1 /a/squid/fundraising/lp-filter >> /a/squid/fundraising/logs/landingpages.log # Banner Impressions pipe 100 /a/squid/fundraising/bi-filter >> /a/squid/fundraising/logs/bannerImpressions-sampled100.log
To enable/disable, uncomment/comment these lines and then HUP udp2log:
awjrichards@locke:~$ /home/file_mover/scripts/resetudp2log
[edit] proxy log rotation and archiving
Log rotation, compression, and copy to netapp is handled by a cron job running as user file_mover@locke:
# rotate and compress fundraising banner impression logs, and archive to netapp */15 * * * * /home/file_mover/scripts/rotate_fundraising_logs
[edit] analytics processing script
[To do: pgehres to document...]
[edit] monitoring and debugging
The cron script logs verbosely and locke:/var/log/syslog will show you actions and errors.
Here's what's what in the following examples:
/a/squid/fundraising/logs/*log # active udp2log collection point /a/squid/fundraising/logs/buffer/2012/*.log # freshly rotated log, before compression /a/squid/fundraising/logs/fr_archive # netapp nfs mount (i.e. permanent archive location)
Under normal operation, you should see this sequence:
Sep 6 17:45:01 locke CRON[28592]: (file_mover) CMD (/home/file_mover/scripts/rotate_fundraising_logs) Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/landingpages.log to /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: move /a/squid/fundraising/logs/bannerImpressions-sampled100.log to /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: reload udp2log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/bannerImpressions-sampled100-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: gzip /a/squid/fundraising/logs/buffer/2012/landingpages-20120906-174501.log Sep 6 17:45:01 locke rotate_fundraising_logs[28594]: rsync -ar /a/squid/fundraising/logs/buffer/ /a/squid/fundraising/logs/fr_archive/ Sep 6 17:45:02 locke rotate_fundraising_logs[28594]: done!
Things to watch out for include:
- move/gzip errors due to local partition overrun, permissions snafu
- nfs mount inaccessible
- udp2log HUP fails
