UDP based profiling

What and Where

• $wgProfiler = new ProfilingSimpleUDP; (in index or Settings)
• $wgUDPProfilerHost = '10.0.6.30'; (in Settings)
• running on professor:
  • /usr/udpprofile/sbin/collector svn root
    • listens on udp:3811 for profiling packets, provides xml dumps on tcp:3811
  • /usr/udpprofile/sbin/profiler-to-carbon
    • polls collector every minute, inserts count and timing (in ms) data into whisper db's
  • /opt/graphite/bin/carbon-cache.py
    • updates whisper db files for graphite
• graphite based web interface - uses labs ldap for auth
• aggregate report web interface
  • • svn root

Using The Graphite Dashboard

Finding Metrics

• The left sidebar of the graphite dashboard provides two drop down menus - "Metric Type", which is used for providing shortcuts or aliases to certain metrics (which are hardcoded in the dashboard.conf located in puppet/files/graphite) and Category. Then below, a hierarchical finder of everything under the chosen category. This is all straight forward, except what's shown when Category = * is limited to a single level of the hierarchy - you don't want this! If Metric Type == Everything, make sure to select a class in Category.

• The dashboard menu option allows sets of graphs to be saved as a named dashboard. Share provides a direct url to a saved dashboard, and Finder lists all shared dashboards.

Combining Metrics

• Just drag graphs on top of each other to combine.

Types of Metrics

• count - the number of calls made in the last minute. Note that for a few types of requests, mediawiki profiles 100% of requests, but most are at about 1.5%.
• tavg - average time in ms, based on everything collected in the sampling time - total-time/count
• tp50 - 50th percentile in ms, calculated from a bucket of 300 samples
• tp99 - 99th percentile
• tp999 - 99.9th percentile

Examples

• 99% ParserCache get times, with cluster deploys overlaid as vertical lines

• The url to generate this was - http://graphite.wikimedia.org/render?from=-24hours&until=now&width=800&height=600&target=ParserCache.get.tp99&target=drawAsInfinite(deploy.any)
  • Overlaying metrics is as simple as appending multiple target options. "&target=drawAsInfinite(deploy.any)" can be added to any graph for the deploy lines.