Swift/Other Processes

Swift Cleaner

• compares thumbnails on swift to those on ms5. deletes things on swift that don't match what's on ms5.
• two instances running on iron - full scan and incremental scan.
  • incremental takes about 2 hours
  • full takes about 2 weeks
• installed in /opt/swiftcleaner. canonical source in git:operations/software/swiftcleaner

Ganglia Logtailer

• ganglia-logtailer process running on all swift nodes (front and back ends)
• parses logs to gather performance and throughput information, feeds that into ganglia

swift-ganglia-report-global-stats

• runs on one server per cluster
• reports cluster-wide statistics to ganglia
  • object count, bytes used, change in # objects
  • these aren't related to a specific server but instead reflect cluster-wide stats

purgethumbs.py

• purges thumbnails from all media stores
• takes in a list of images to purge and attempts to purge all thumbnails for that list of images from all three locations: ms5, swift, and squid
• used to clean out unreferenced thumbnails generated by ariel's script

delete-containerless-objects

• script to crawl the object contents in a swift backend, compare this to a list of what's in the container, and delete objects that exist but aren't in the container list.
  • note that this shouldn't happen in the first place