Swift/Deploy Plan - Thumbnails
From Wikitech
< Swift
Contents |
order
- before user-visible deploy (pre-work)
- verify that LVS is working correctly
- mediawiki deploy (purge requests start flowing to swift)
- purge deleted objects
- on deploy (this triggers user-visible changes)
- swift cluster changes
- squid changes (end user thumbnail requests start flowing to swift)
- post-deploy
verify LVS
LVS should be answering to ms-fe.pmtpa.wmnet (10.2.1.27) on port 80 and passing it through to ms-fe1 and ms-fe2. Verify this is working as intended. [done]
mediawiki deploy
- Backport LocalFilePurgeThumbnails hook and deploy it [done]
- Set swift config in PrivateSettings.php [done]
- Set 'swiftThumb' MW udp logger [done]
- Create purgeDeletedFiles.php script [done]
- SVN up and sync SwiftCloudfiles MediaWiki extension [done]
- Enable and deploy SwiftCloudfiles [done]
- Copy swift.php to wmf-config and sync it [done]
- Include swift.php in CommonSettings.php and sync [done]
- Verify that purge requests are deleting the thumbnails from swift [done]
- upload an image, generate thumbnails, confirm they exist on swift, delete the image, confirm the thumbnails are deleted on swift
purge deleted objects
- Run "mwscriptwikiset purgeDeletedFiles.php all.dblist --starttime=20120126000000" on hume. [done]
swift cluster changes
- stop the thumbnail filler script (screen -x listen on fenari/~ben) done
- change
thumbhostin /etc/swift/proxy-server.conf fromupload.wikimedia.orgtoms5.pmtpa.wmnetdone - restart the proxy-server processes:
for i in 1 2 ; do ssh ms-fe$i.pmtpa.wmnet swift-init proxy restart; donedone
squid changes
- add the following to squid.conf done
## acls to send thumbnails to swift, from .4% to 100% # send only one shard (0.39%)to swift acl swift1_thumbs url_regex ^http://upload\.wikimedia\.org(/+)[^/][^/]*/[^/][^/]*/thumb/a/a2 # send 12.5% of shards to swift acl swift2_thumbs url_regex ^http://upload\.wikimedia\.org(/+)[^/][^/]*/[^/][^/]*/thumb/[ab]/ # send 25% of shards to swift acl swift3_thumbs url_regex ^http://upload\.wikimedia\.org(/+)[^/][^/]*/[^/][^/]*/thumb/[abcd]/ # send 50% of shards to swift acl swift4_thumbs url_regex ^http://upload\.wikimedia\.org(/+)[^/][^/]*/[^/][^/]*/thumb/[abcdef01]/ # send 75% of shards to swift acl swift5_thumbs url_regex ^http://upload\.wikimedia\.org(/+)[^/][^/]*/[^/][^/]*/thumb/[abcdef012345]/ # send 100% of shards to swift acl swift_thumbs url_regex ^http://upload\.wikimedia\.org(/+)[^/][^/]*/[^/][^/]*/thumb/
# swift - one shard cache_peer 10.2.1.27 parent 80 3130 originserver no-query no-digest connect-timeout=5 max-conn=200 login=PASS cache_peer_access 10.2.1.27 allow swift1_thumbs cache_peer_access 10.2.1.27 deny all
- confirm the squid changes are working done
- request a file in the a2 bucket
- increment the pixel count to force generation of a new thumbnail
- verify you are served the thumbnail
- verify the request hits swift (via log inspection or tcpdump) (note - on both ms-fe1 and ms-fe2)
- verify the request does *not* hit ms5 (via tcpdump)
- on subsequent days, change the cache_peer_access from
allow swift1_thumbsto </code>allow swift2_thumbs</code> and so on.
gradual rollout schedule
As indicated by the squid configs above, we've got rules for 0.4%, 12.5%, 50%, and 100%. These will be deployed Monday through Thursday.
- Monday: /a/a2/ done
- Tuesday: /a/* and /b/* done then /a-d/* done
- Wednesday: /a-f/* and /0/* and /1/* (not done) then /a-f/* and /0-5/* (not done)
- Thursday: all
