Server admin log/Archive 21

October 31

• 21:12 logmsgbot_: reedy synchronized php-1.21wmf3/includes/site/SiteObject.php
• 20:42 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed cbe526b06ec1876c933f8a04c2526b137642a879'
• 20:41 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed bf098d262dce290ac4b0cf70f74182c98aeca4d9'
• 20:09 Reedy: Restarting Jenkins
• 20:09 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Bug 41528 - need more memory for video thumbnails'
• 19:35 logmsgbot_: preilly Finished syncing Wikimedia installation... : update zero rated mobile access
• 19:34 mark: Restarted backend varnish instance on cp1024
• 19:33 K4-713: updated payments cluster to 5d8309e21
• 18:54 logmsgbot_: preilly Started syncing Wikimedia installation... : update zero rated mobile access
• 18:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia or wikivoyage to 1.21wmf3
• 18:40 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
• 18:39 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/ZeroRatedMobileAccess 'update post deploy'
• 18:38 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikinews to 1.21wmf3
• 18:31 cmjohnson1: rebuilding software raid for storage3 and provisioning
• 18:30 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ReaderFeedback
• 18:29 mutante: srv286 - installing php5-memcached and dist-upgrade
• 18:27 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikiquote and wikibooks to 1.21wmf3
• 18:25 mutante: srv286 - kill frozen apt-get update process, update sources, install package upgrades
• 18:18 mutante: mw18 - dpkg --configure -a to fix interrupted dpkg, configures php-luasandbox, install php5-memcached
• 18:17 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ReaderFeedback
• 18:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, closed and private to 1.21wmf3
• 18:01 logmsgbot_: reedy synchronized php-1.21wmf3/
• 18:00 logmsgbot_: reedy synchronized php-1.21wmf2/
• 17:56 mutante: now redirecting wikimania.wm to wikimania2013.wm
• 17:54 logmsgbot_: dzahn gracefulled all apaches
• 17:52 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/MobileFrontend 'update after deploy'
• 17:51 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
• 17:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
• 16:57 Reedy: Truncating wikibase.wb_changes
• 16:25 mark: Restarted backend varnish instance on cp1023
• 15:50 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgULSIMEEnabled = false'
• 15:49 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/UniversalLanguageSelector/ 'Update to master'
• 15:00 mark: Restarted backend varnish instance on cp1022
• 14:45 paravoid: putting ms-fe4 back in the pool
• 14:26 paravoid: temporarily depooling ms-fe4 to apply a temporary local hack
• 14:12 paravoid: putting ms-fe4 back in the pool
• 13:27 mark: Restarted cp1021 with persistent storage
• 13:26 paravoid: depooling ms-fe4 to upgrade to precise/swift 1.7.4
• 11:35 paravoid: putting ms-fe3 back in the pool
• 11:28 ^demon: restarted gerrit services on manganese & formey to pick up replication changes
• 07:59 paravoid: depooling ms-fe3 to upgrade to precise/swift 1.7.4
• 07:09 apergos: restarted lucene search on search1015 and search 1003
• 02:47 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Oct 31 02:47:40 UTC 2012
• 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 31 02:28:50 UTC 2012
• 02:05 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
• 02:01 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable TMH'
• 01:43 logmsgbot_: reedy synchronized wmf-config/
• 01:23 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disable TMH everywhere (and mwembed at the same time)'
• 01:15 K4-713: synchronized the payments cluster to 85e9007aef3e75
• 01:06 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/actions/mf-edit.js 'touch file'
• 00:58 logmsgbot_: reedy synchronized php-1.21wmf3/includes/job/JobQueueDB.php
• 00:57 logmsgbot_: awjrichards synchronized php-1.21wmf2/extensions/MobileFrontend/javascripts/actions/mf-edit.js 'touch file'
• 00:42 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'Update CentralNotice on meta and en.wiki to fix banner override bug'
• 00:30 logmsgbot_: maxsem Finished syncing Wikimedia installation... :

October 30

• 23:44 logmsgbot_: maxsem Started syncing Wikimedia installation... :
• 23:08 mutante: replaced wikipedia SSL cert with new one by GeoTrust
• 22:16 logmsgbot_: kaldari synchronized php-1.21wmf3/extensions/CentralNotice 'autoloading ApiCentralNoticeAllocations for CentralNotice'
• 22:15 LeslieCarr: failing overRE mastership on cr2-eqiad to upgrade re1
• 22:02 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'memcached logging group'
• 22:00 logmsgbot_: asher synchronized wmf-config/mc.php 'testing MemcachedPeclBagOStuff on test2wiki (serverlist fix)'
• 21:57 logmsgbot_: asher synchronized wmf-config/mc.php 'testing MemcachedPeclBagOStuff on test2wiki'
• 21:47 mutante: restarting nginx on ssl hosts for cert update
• 21:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
• 21:30 logmsgbot_: kaldari synchronized php-1.21wmf3/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
• 21:24 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
• 21:23 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
• 21:14 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
• 21:09 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.php 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
• 20:54 Jeff_Green: stopping mysql on db1013 for cloning
• 20:37 logmsgbot_: kaldari Started syncing Wikimedia installation... :
• 20:21 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'updating wgCentralBannerDispatcher for CentralNotice'
• 19:46 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'updating settings for CentralNotice banner loading'
• 18:31 binasher: completed centralnotice patch-bucketing.sql migration
• 18:24 mutante: giving sbernardin racktables permissions
• 17:53 mark: Installed Precise on new upload esams servers cp3003-cp3010
• 17:20 mutante: running sync-apache on srv199 only
• 17:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxAnimatedGifArea = 2.5e7'
• 17:15 logmsgbot_: reedy synchronized wmf-config/omgtestfile
• 17:06 Reedy: Running sync-common on srv199
• 17:01 Reedy: Running sync-common on srv194
• 16:54 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.21wmf3.php
• 16:38 apergos: stpped stray jobrunner on srv278
• 14:38 paravoid: powercycling srv229, dead for unknown reason so far
• 10:08 logmsgbot_: nikerabbit synchronized php-1.21wmf2/extensions/NewUserMessage/NewUserMessage.class.php 'I0f93ee53'
• 09:41 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Temporarily enable beta mappings for am wikis'
• 09:19 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam and Webfonts: bug 41460, bug 39200, bug 41359'
• 09:07 mark: Fixed wikidata.org language cnames issue
• 07:44 mark: Fixed IPv6 addresses on new esams bits servers
• 07:33 mark: Replaced 2 bits @ esams servers with 4 new servers cp3019-cp3022
• 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Oct 30 02:49:48 UTC 2012
• 02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 30 02:24:04 UTC 2012
• 01:09 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php
• 01:09 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
• 01:02 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/EventLogging 'Updating EventLogging on test2'
• 00:43 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
• 00:29 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed 18e51f3b06b84d1d5fbf47272d9ebfc5008dc879'
• 00:28 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed c1ac05640377f4f99cbe2a094e80d3d25d63b93d'
• 00:27 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/EventLogging 'Updating EventLogging on test2'

October 29

• 21:28 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
• 21:25 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Revert'
• 21:24 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Change wikidata search config even more'
• 21:20 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Change wikidata search config even more'
• 21:11 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Disable lucene on wikidatawiki for now'
• 20:44 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5
• 19:39 logmsgbot_: reedy synchronized live-1.5/
• 19:36 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf3
• 19:19 Reedy: Deleting php-1.20wmf11 files
• 19:16 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
• 18:25 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed 05118462e3ddbfc507772cc3c4e1117a8527e0b5'
• 18:19 Reedy: Run Wikibase/repo/maintenance/createBlacklistedItems.php on wikidatawiki
• 18:16 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor/ 'Update VisualEditor'
• 18:04 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Add wikidata to wgCrossSiteAJAXdomains'
• 17:54 Reedy: Ran patch-job_token.sql on wikidatawiki
• 17:52 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Turning SUL back on for wikidata.org'
• 17:51 logmsgbot_: catrope gracefulled all apaches
• 17:20 Reedy: Running \Wikibase\Utils::insertDefaultSites(); on wikidatawiki
• 17:08 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache'
• 17:07 logmsgbot_: reedy synchronized php-1.21wmf2/cache/interwiki.cdb 'Updating interwiki cache'
• 17:02 Reedy: Applied Wikibase/lib/sql/WikibaseLib.sql on wikidatawiki
• 16:50 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
• 16:44 logmsgbot_: reedy synchronized php-1.21wmf3/includes/WikiPage.php
• 16:43 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
• 16:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf3
• 16:38 logmsgbot_: reedy synchronized php-1.21wmf3/cache/l10n/ 'Sync'
• 16:30 logmsgbot_: mark synchronized wmf-config/CommonSettings.php 'Add new bits server IPs'
• 16:12 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
• 16:08 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
• 16:01 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
• 15:54 logmsgbot_: reedy synchronized wmf-config/
• 15:52 Reedy: Running sync-common on searchidx1001
• 15:51 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/PostEdit/
• 15:50 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EventLogging/
• 15:46 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
• 15:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf3
• 15:40 logmsgbot_: reedy synchronized php-1.21wmf3 'Initial file sync-out'
• 13:38 ^demon: flushed gerrit's account_byemail cache, error log was complaining
• 13:19 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Disabling wikidata for SUL until we fix SSL'
• 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Mon Oct 29 02:28:02 UTC 2012

October 28

• 22:37 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging 'Updating Extension:EventLogging (enabled only on test & test2)'
• 22:33 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php 'Enable Extension:EventLogging on test2'
• 21:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'arwiki logo'
• 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sun Oct 28 02:28:08 UTC 2012

October 27

• 21:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set geoservice to geoiplookup.wm.o'
• 20:51 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Comment out wgULSGeoService'
• 10:02 paravoid: repooling sq82 frontend in pybal
• 08:49 paravoid: reinstalling sq82 with lucid
• 07:41 apergos: depooled sq82 in pybal, squid3 was running on it again and it's still precise
• 02:33 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sat Oct 27 02:33:26 UTC 2012
• 00:25 mutante: restarting nginx on ssl hosts for changed server name for wikidata
• 00:17 mutante: reloading nginx on ssl1001

October 26

• 23:46 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/UniversalLanguageSelector/
• 23:23 Ryan_Lane: powercycling kaulen
• 23:09 cmjohnson1: tinkering with storage3
• 22:24 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set wgULSGeoService to false'
• 22:03 Ryan_Lane: repooing ssl1004
• 21:44 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgContentHandlerUseDB back to true. Mainpage fixed!'
• 21:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgContentHandlerUseDB to false for wikidatawiki'
• 21:27 cmjohnson1: imaging db42
• 20:59 Ryan_Lane: rebooting ssl1004
• 20:54 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that'
• 20:52 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disabling wikidata related extensions temporarily'
• 20:50 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
• 20:50 logmsgbot_: reedy synchronized images/sul/wikidata.png
• 20:43 logmsgbot_: reedy synchronized php-1.21wmf2/includes/content/ContentHandler.php
• 20:43 logmsgbot_: reedy synchronized php-1.21wmf2/cache/l10n/
• 20:43 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
• 20:37 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
• 20:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
• 20:28 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild localisation cache for wikidata related extensions
• 20:25 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/MwEmbedSupport 'deployed de1079e3eec4eae4f129d7bcffc71596f415ae74'
• 20:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikibaserepo on wikidatawiki'
• 20:13 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
• 20:13 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
• 20:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Temporary hacking in of ns' for wikidatawiki'
• 20:07 Reedy: Adding extensions/Wikibase/repo/includes/store/sql/Wikibase.sql to wikidatawiki
• 20:01 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
• 20:00 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed 31a1d21e8ebfb90ece5ef02075fc85f197a50e53'
• 19:59 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'live hacky hack'
• 19:55 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'live hacky hack'
• 19:53 cmjohnson1: rebooting sq82
• 19:52 logmsgbot_: reedy synchronized wmf-config/ 'gotta touch them all'
• 19:45 logmsgbot_: reedy synchronized wmf-config/
• 19:18 cmjohnson1: stopping puppet on brewster temporarily
• 19:17 logmsgbot_: root gracefulled all apaches
• 19:12 mutante: deleting files older than 3 days from /tmp/ in "mw" dsh group
• 19:05 mutante: deleting /tmp/mw-cache-1.20* from mw dsh group
• 18:57 Reedy: Cleaned /tmp on mw48
• 18:54 Reedy: Cleaned /tmp on mw34
• 18:51 Reedy: Cleaned /tmp on mw45
• 18:31 Jeff_Green: aluminium dist upgrade and reboot
• 18:27 Jeff_Green: erzurumi dist-upgrade and reboot
• 18:02 logmsgbot_: aaron synchronized php-1.21wmf2/thumb.php 'done'
• 17:58 logmsgbot_: aaron synchronized php-1.21wmf2/thumb.php 'improve path mismatch error message'
• 17:50 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php 'done'
• 17:44 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php
• 17:42 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php 'debug logging'
• 17:35 logmsgbot_: dzahn gracefulled all apaches
• 17:32 mutante: sync-apache new wikidata config
• 16:49 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php 'Disabling EventLogging on enwiki'
• 14:28 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules/ext.abuseFilter.tools.js 'Fixing JS for AbuseFilter'
• 14:28 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules/ext.abuseFilter.examine.js 'Fixing JS for AbuseFilter'
• 14:08 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules 'Updating AbuseFilter to f52f940e'
• 13:41 logmsgbot_: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/30070/'
• 13:07 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php 'Syncing I85721736'
• 12:46 paravoid: stopping squid3(!) on sq82, box was reprovisioned with precise but we don't support squid3/squid on precise yet
• 11:20 logmsgbot_: reedy synchronized wmf-config/
• 11:03 Reedy: Created WikiLove tables on urwiki
• 05:42 paravoid: powering off ms-fe1001 & ms-fe1002
• 02:38 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.db.php 'updating CentralNotice.db.php'
• 02:32 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Fri Oct 26 02:32:33 UTC 2012
• 01:37 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating CentralNotice'
• 01:21 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.db.php 'updating CentralNotice.db.php'
• 00:15 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php
• 00:14 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/ContributionTracking/ 'Updating ContributionTracking'
• 00:13 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage'
• 00:04 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'fixing bannerController'

October 25

• 23:55 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
• 23:49 cmjohnson1: installing base OS on sq82
• 23:20 logmsgbot_: kaldari Started syncing Wikimedia installation... :
• 22:58 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/E3Experiments
• 22:40 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/OpenSearchXml/ApiOpenSearchXml.php 'use slave DB for revision queries.'
• 22:36 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/MWSearch/MWSearch_body.php 'use slave DB for revision queries.'
• 22:30 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'done'
• 22:30 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'logging one more thing.'
• 22:24 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'done'
• 22:23 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'tweak wfGetCaller param a level'
• 22:22 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php
• 22:21 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'temp query debug logging'
• 22:18 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
• 21:55 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/AbuseFilter 'desc'
• 21:55 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5 'desc'
• 21:55 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
• 21:55 logmsgbot_: mlitn synchronized wmf-config 'desc'
• 21:46 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/AbuseFilter 'desc'
• 21:45 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5 'desc'
• 21:01 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Syncing I39c9693a'
• 21:00 logmsgbot_: demon synchronized wmf-config/wgConf.php 'Syncing I39c9693a'
• 20:30 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ 'Dark deploy Wikibase, Diff and ULS'
• 19:47 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/ContributionTracking/ 'Updating ContributionTracking to master'
• 19:44 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'adding new var for CentralNotice'
• 19:41 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/LandingCheck/ 'Updating LandingCheck to master'
• 18:36 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
• 18:36 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
• 18:31 Ryan_Lane: Put in a live-hack on labsconsole to remove m1.tiny from the list of instance types
• 18:29 cmjohnson1: installing base OS on srv194
• 18:17 Ryan_Lane: updated OpenStackManager on labsconsole to master version
• 18:16 binasher: flushed mobile varnish cache per preilly
• 18:09 Jeff_Green: restarted gmetad on nickel
• 17:53 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
• 17:52 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
• 16:56 paravoid: powercycling copper, load 700
• 14:57 notpeter: silencing wikivoyage checks in nagios until deployed
• 14:54 cmjohnson1: shutting down sq82 to replace /dev/sda
• 14:52 cmjohnson1: shutting down srv194 to replace disk
• 14:05 mark: Sending all non-European upload traffic to eqiad
• 12:01 mark: Sending US upload traffic to eqiad
• 12:01 logmsgbot_: demon synchronized wmf-config/wgConf.php 'Syncing out new prefixes for wikidata/wikivoyage'
• 10:56 mark: Sending Argentina upload traffic to eqiad
• 10:37 mark: Sending Brazil upload traffic to eqiad
• 10:24 mark: Sending Canadian upload traffic to eqiad
• 10:03 paravoid: powercycling nickel, OOM & unable to login
• 09:42 mark: Hacked Nagios back up
• 08:43 mark: Added georecords for wikidata-lb.wikimedia.org and wikivoyage-lb.wikimedia.org, their geomaps containing just a default entry pointing to eqiad
• 08:33 mark: Added LVS service IPs for wikidata and wikivoyage (pmtpa/eqiad) to DNS
• 07:12 mark: Stopping backend squid on sq82, sda I/O errors
• 00:34 paravoid: upgrading oggvideotools to 0.8a on all imagescalers, fixing regression on the lucid->precise upgrade
• 00:27 binasher: deploying updated squid mobile redirector
• 00:24 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php

October 24

• 23:12 binasher: restarting pdns on ns0
• 23:09 cmjohnson1: db78 loading base OS
• 23:06 cmjohnson1: authdns update adding db78.pmtpa.wmnet
• 22:53 cmjohnson1: authdns changes adding db63-db78 mgmt
• 21:37 logmsgbot_: aaron synchronized php-1.21wmf2/includes/EditPage.php 'deployed 51eede5af0fe0bb28d7abdf81a04292ba8a4864c'
• 20:48 logmsgbot_: demon synchronized php-1.21wmf2/includes/Title.php 'Syncing Ic2d3f0b8'
• 20:44 logmsgbot_: demon synchronized php-1.21wmf2/includes/EditPage.php 'Deploying I4c2055be'
• 19:59 logmsgbot_: dzahn gracefulled all apaches
• 18:49 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'rm debugging live hacks'
• 18:46 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
• 18:44 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
• 18:39 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
• 18:17 Reedy: Running sync-common on mw40
• 18:02 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf2
• 17:53 notpeter: upgrading srv223 and srv224 to precise
• 17:41 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/ZeroRatedMobileAccess 'update post deploy'
• 17:40 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess 'update post deploy'
• 17:38 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
• 17:26 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
• 17:25 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
• 17:21 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
• 17:10 cmjohnson1: db62 loading OS
• 16:14 cmjohnson1: shutting down db61 for rack relocation
• 16:03 cmjohnson1: removing h310 controller from db62 and adding original h700 card
• 14:21 notpeter: upgarding srv221 and srv222 to precise
• 13:39 cmjohnson1: installing base OS on db62
• 11:56 paravoid: putting ms-fe2 (upgraded to precise/swift 1.7.4) back into the pool
• 05:59 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php
• 02:48 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 24 02:48:06 UTC 2012
• 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 24 02:27:30 UTC 2012
• 00:50 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend
• 00:26 mutante: flushing varnish mobile cache
• 00:21 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/
• 00:18 logmsgbot_: maxsem synchronized php-1.21wmf2/extensions/MobileFrontend/
• 00:09 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly MobileFrontend and Zero deployment

October 23

• 23:25 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly MobileFrontend and Zero deployment
• 23:09 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
• 23:02 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
• 22:57 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php
• 22:57 logmsgbot_: reedy synchronized wmf-config/
• 22:54 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php 'debugging'
• 22:53 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php 'debugging'
• 22:13 logmsgbot_: mlitn Finished syncing Wikimedia installation... : updating ArticleFeedbackv5, PageTriage & E3Experiments
• 22:02 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
• 21:41 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
• 21:39 logmsgbot_: mlitn Started syncing Wikimedia installation... : updating ArticleFeedbackv5, PageTriage & E3Experiments
• 20:57 Ryan_Lane: restarting gmetad on nickel
• 20:56 Ryan_Lane: restarting gmetad
• 20:40 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
• 20:31 Jeff_Green: dist-upgrade & rebooting payments[1-4]
• 20:05 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
• 20:02 cmjohnson1: fixing db62 raid cnfg
• 19:10 logmsgbot_: asher synchronized wmf-config/db.php 'returning es10 to service'
• 18:40 binasher: shutting down es10 for dimm replacement
• 17:39 RobH: analytics1011 now offline and wiping as well
• 17:15 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 25MP'
• 17:11 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/PagedTiffHandler
• 16:36 cmjohnson1: db56 replacing disk 7 (slot6)
• 16:14 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/PagedTiffHandler/
• 16:08 logmsgbot_: reedy synchronized php-1.21wmf2/includes/db/Database.php
• 16:07 logmsgbot_: reedy synchronized php-1.21wmf2/includes/Message.php
• 16:06 cmjohnson1: db45 replacing disk2 (slot 1)
• 15:55 RobH: analytics1012-1022 are starting wipe and having network cable pulled
• 15:47 RobH: analytics1011-1022 will be dying today, byebye c2100
• 15:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
• 15:21 paravoid: manually patching swift on ms-be3 to comment-out the drop_cache() calls
• 14:57 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php 'Dont parse special pages'
• 14:50 paravoid: temporarily stopping puppet on brewster
• 14:49 mark: Inserted new varnish build 3.0.3plus~rc1-wm4 into precise-wikimedia, fixing duplicate header issue
• 14:49 notpeter: removing srv194 from apaches pool for drive error
• 14:49 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php 'Dont parse special pages'
• 14:32 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/PostEdit 'Update PostEdit to master'
• 12:53 mark: Inserted new varnish 3.0.3plus~rc1-wm3 packages into the precise-wikimedia APT repository, with a duplicate content-length header fix on streaming
• 11:20 paravoid: depooling ms-fe2, in preparation for ubuntu/swift upgrade
• 06:42 logmsgbot_: krinkle synchronized php-1.21wmf1/resources/mediawiki/mediawiki.searchSuggest.js 'Ibcc56f64'
• 06:40 logmsgbot_: krinkle synchronized php-1.21wmf2/resources/mediawiki/mediawiki.searchSuggest.js 'Ibcc56f64'
• 05:17 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/SpecialPage.php
• 02:55 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 23 02:55:38 UTC 2012
• 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 23 02:29:31 UTC 2012

October 22

• 23:04 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
• 23:00 LeslieCarr: switching RE mastership on cr2-eqiad
• 22:43 LeslieCarr: draining cr2-eqiad again for downgrade
• 22:37 logmsgbot_: asher synchronized wmf-config/db.php 'raising weight of db65'
• 22:33 LeslieCarr: switching cr2-eqiad routing engine mastership
• 22:19 logmsgbot_: asher synchronized wmf-config/db.php 'replacing db22 with db65'
• 22:06 LeslieCarr: undraining cr2-eqiad
• 22:01 Jeff_Green: stopping puppet on aluminium so fr-tech can local-test a config change
• 21:48 LeslieCarr: switch routing engine mastership on cr2-eqiad
• 21:23 LeslieCarr: commit full on cr2-eqiad which will restart all routing processes
• 21:20 cmjohnson1: replacing disk13 on tridge array
• 21:19 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/Vector
• 21:17 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
• 20:49 LeslieCarr: draining traffic from cr2-eqiad in preparation for maintenance and upgrade
• 20:21 Reedy: Running p[opulateFileArchiveSha1.php on enwiki
• 19:06 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
• 18:53 binasher: streaming hotbackup of db1038 to db65, preparing to replace to db22
• 18:28 logmsgbot_: reedy synchronized php-1.21wmf2/skins/CologneBlue.php
• 18:16 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf2
• 16:49 logmsgbot_: py gracefulled all apaches
• 16:47 Jeff_Green: stopping mysql on db1025 to clone it to db1013
• 16:02 logmsgbot_: reedy synchronized wmf-config/
• 16:01 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
• 15:48 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
• 15:43 notpeter: re-adding srv219 and srv220 to rendering pool
• 15:04 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
• 13:32 notpeter: removing srv219 and srv220 from rendering pool for upgarde to precise
• 09:35 mark: Built new varnish 3.0.3plus~rc1-wm2 packages and inserted them into the precise-wikimedia APT repository
• 02:51 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 22 02:51:32 UTC 2012
• 02:31 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Mon Oct 22 02:30:57 UTC 2012

October 21

• 22:50 logmsgbot_: tstarling synchronized php-1.21wmf2/bin/ulimit4.sh
• 22:39 Reedy: populateFilearchiveSha1.php finished on commonswiki
• 14:18 Reedy: Running mwscript populateFilearchiveSha1.php commonswiki
• 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 21 02:49:42 UTC 2012
• 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sun Oct 21 02:28:50 UTC 2012
• 01:35 paravoid: powercycling srv219, swap-died while trying to get a core dump
• 01:34 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 20MP'
• 01:33 paravoid: powercycling srv220/srv224, swapdeath

October 20

• 18:51 Reedy: Ran populateFilearchiveSha1.php on metawiki
• 18:21 paravoid: powercycling srv221 & srv222, both dead since ~5h ago
• 17:55 RobH_: updated otrs motd per requst by otrs admins (it referenced a now closed request for comment)
• 17:42 Reedy: Running foreachwiki cleanupUploadStash.php
• 03:13 K4-713: updated the payments cluster to 97e1ba74a39376
• 02:41 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sat Oct 20 02:41:10 UTC 2012
• 02:21 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sat Oct 20 02:21:41 UTC 2012
• 01:36 K4-713: updated the payments cluster to bb30dab91a7afe862
• 00:46 mutante: upgrade Bugzilla to 4.0.8 (security patch)

October 19

• 23:36 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 17MP'
• 23:31 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/SwiftCloudFiles
• 23:30 LeslieCarr: shutting down sfo-monitor1
• 23:27 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
• 23:15 binasher: returning srv194/200 to apache lvs pool
• 22:50 mutante: authdns-update, renaming travel-guide-lb to wikivoyage-lb
• 21:10 binasher: pulled srv194 and srv200 from lvs for memc testing
• 21:02 binasher: upgrading percona-toolkit to 2.1.5
• 20:18 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
• 19:33 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed 65e47037f6e9da2e81db2bc1374cdac375a82db5'
• 19:26 paravoid: powercycling srv223 too, locked up, nothing in serial console
• 19:25 paravoid: powercycling srv221, locked up, nothing in serial console
• 16:21 RobH: msw-a2-eqiad back online
• 15:54 RobH: msw-a2-eqiad is coming down for replacement per RT 3683. all mgmt connectivity (only) will be lost to items in A2-EQIAD
• 14:43 paravoid: putting ms-fe1 back into the pool
• 14:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.21wmf2
• 13:59 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.21wmf2
• 13:50 cmjohnson1: shutting down msbe1001-1012 to remove ssd's and package for dell return
• 13:48 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
• 04:15 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
• 02:59 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
• 02:57 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
• 02:47 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Fri Oct 19 02:47:49 UTC 2012
• 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Fri Oct 19 02:27:57 UTC 2012
• 02:27 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
• 02:23 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
• 02:10 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
• 01:59 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
• 01:56 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
• 01:53 K4-713: updated payments cluster to ff8a1e791eaff47313
• 01:53 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
• 01:50 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
• 01:46 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
• 01:45 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
• 01:44 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
• 01:20 logmsgbot_: kaldari Started syncing Wikimedia installation... :
• 00:26 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php
• 00:26 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/PostEdit
• 00:22 logmsgbot_: reedy synchronized wmf-config/
• 00:21 paravoid: disabling puppet on (the already depooled) ms-fe1
• 00:09 logmsgbot_: olivneh Finished syncing Wikimedia installation... :
• 00:04 logmsgbot_: olivneh Started syncing Wikimedia installation... :
• 00:03 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/PostEdit

October 18

• 23:42 logmsgbot_: reedy synchronized wmf-config/
• 23:31 logmsgbot_: spage Finished syncing Wikimedia installation... : updating 1.21wmf1 & wmf2 with E3Experiments (and 1.21wmf1 with PostEdit)
• 23:26 Reedy: populateFilearchiveSha1.php ran for all wikis on 1.21wmf2
• 23:16 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
• 23:14 logmsgbot_: spage Started syncing Wikimedia installation... : updating 1.21wmf1 & wmf2 with E3Experiments (and 1.21wmf1 with PostEdit)
• 23:13 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia.dblist to 1.21wmf2
• 23:11 Reedy: Added fa_sha1 to liquidthreads_labswikimedia
• 23:10 Reedy: Ran populateFilearchiveSha1.php on private, wiktionary, wikiversity and wikinews wikis
• 23:07 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php 'debugging hack'
• 23:04 Reedy: Added fa_sha1 to fdcwiki
• 23:00 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php 'debugging hack'
• 22:32 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed 15520f489b9adab8bcfd84e6b6a5d48280a85556'
• 22:05 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiQueryRevisions.php
• 21:26 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed ff03172cccad1ce7017c7dc44e508317cc73975f'
• 21:11 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
• 21:01 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php '$wgMaxImageArea to 15'
• 20:18 logmsgbot_: kaldari Started syncing Wikimedia installation... :
• 20:04 Reedy: Running scap-recompile on all mediawiki-installation
• 19:28 Jeff_Green: adjusted a few hostnames for silverpop
• 19:21 notpeter: created ulsfo rt queue
• 19:19 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
• 18:53 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
• 17:54 Jeff_Green: adjusted banner log filter to include non-fundraising banners
• 17:01 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of planned wikis to 1.21wmf2 minus commons and meta
• 16:53 K4-713: synchronized payments cluster to efcecc41c6f3c26
• 15:19 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiversity to 1.21wmf2
• 15:17 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
• 15:15 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews to 1.21wmf2
• 15:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private wikis to 1.21wmf2
• 14:19 notpeter: re-adding srv190 to rendering pool
• 02:50 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 18 02:50:46 UTC 2012
• 02:26 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Thu Oct 18 02:26:24 UTC 2012

October 17

• 22:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/specials/
• 20:58 mutante: nuking wmfsocial mailing list
• 20:48 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: revert test
• 20:47 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test
• 20:42 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Fix tetwiki
• 20:40 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf2
• 20:31 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki testwiki and test2wiki back to 1.21wmf2
• 20:23 ^demon: restarted gerrit service, was missing documentation from UI
• 19:33 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private and special wikis back to 1.21wmf1
• 19:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Increase $wgMaxImageArea to 2.5e7 on testwiki'
• 19:14 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Increase $wgMaxImageArea to 1.4e7'
• 18:54 binasher: fixed pc1 innodb_free_space logging on pc1 via "GRANT SELECT ON `parsercache`.`pc000` TO 'dbstats'@'localhost';" - (it had only been granted access to the objectcache table which no longer exists)
• 18:53 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments/lib/event/eventlog.js
• 18:44 cmjohnson1: racking and setting up replacements for ms-be6/7
• 18:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki back to 1.21wmf1
• 18:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, special and private to 1.21wmf2
• 18:08 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiEditPage.php
• 18:05 LeslieCarr: killed and restarted all varnish processes on cp1041
• 18:00 LeslieCarr: restarted varnish-frontend on cp1041
• 17:53 LeslieCarr: restarting varnish on cp1041
• 17:24 mark: Installed experimental Varnish package with range support in streaming mode on cp1021
• 16:50 logmsgbot_: midom synchronized wmf-config/InitialiseSettings.php
• 16:21 LeslieCarr: restarting varnish on cp1042 , gone into swapdeath spiral
• 16:15 logmsgbot_: reedy synchronized php-1.21wmf2/cache/l10n/
• 16:01 logmsgbot_: reedy synchronized php-1.21wmf1/cache/l10n/
• 15:59 notpeter: increasing weight on mw17-mw59 by 33.3%
• 15:46 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for educationprogram
• 15:36 Jeff_Green: switched donate.wikimedia.org to geodns
• 15:31 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/EducationProgram
• 15:23 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/EducationProgram/
• 14:34 logmsgbot_: midom synchronized wmf-config/InitialiseSettings.php
• 06:09 apergos: restarted lucene search on search1015
• 04:10 Tim: restarted swift frontends
• 03:41 Tim: seems to have worked, also truncating pc161-174 to be on the safe side
• 03:38 Tim: on pc1: truncated tables pc158, pc159, pc160 due to flood of "table full" errors in dberror.log indicating innodb space exhaustion
• 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 17 02:49:53 UTC 2012
• 02:26 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 17 02:26:34 UTC 2012

October 16

• 23:41 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/ 'https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-15%2616'
• 23:33 logmsgbot_: maxsem synchronized php-1.21wmf2/extensions/MobileFrontend/ 'https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-15%2616'
• 22:50 binasher: redeployed squid refresh_pattern no-cache tweaks
• 22:46 MaxSem: Running mw-update-l10n and friends to catch up newest MobileFrontend messages
• 21:45 paravoid: deploying squid configs, reverting a change that was too aggressive on caching content and broke https-by-default wikis
• 21:40 logmsgbot_: reedy synchronized php-1.21wmf2/includes/specials/SpecialUndelete.php
• 20:08 cmjohnson1: storage3 down for troubleshooting
• 19:57 Reedy: Ran namespaceDupes.php on azwikibooks
• 19:20 binasher: deployed squid refresh_pattern tweaks for testing
• 19:19 Jeff_Green: adding dns and geodns entries for donate-lb
• 18:58 cmjohnson1: shutting down db67 to add h800 card
• 14:57 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
• 14:56 logmsgbot_: hashar synchronized wmf-config/throttle.php '(bug 41069) Delhi 2012-10-17 workshop throttle rule'
• 14:56 logmsgbot_: hashar synchronized wmf-config/throttle.php '(bug 41069) Delhi 2012-10-17 workshop throttle rule'
• 12:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
• 09:06 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php '(bug 41015) Enable Narayam on hi.wiktionary'
• 02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 16 02:53:28 UTC 2012
• 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 16 02:28:10 UTC 2012
• 00:57 RoanKattouw: chmod -R g+w /home/wikipedia/common/php-1.21wmf2/.git
• 00:48 RoanKattouw: I mean 002
• 00:48 RoanKattouw: Created ~spage/.bashrc with 'umask 0002'
• 00:03 binasher: completed patch-job_token.sql migrations
• 00:00 binasher: running patch-job_token.sql migration against all wikis

October 15

• 23:04 mutante: power cycling cp1040
• 21:55 logmsgbot_: catrope synchronized php-1.21wmf2/extensions/VisualEditor/ 'f25aaf5bc4449ea8c1ac749a96487485b5f0c6b3'
• 21:54 logmsgbot_: catrope synchronized php-1.21wmf2/includes/resourceloader/ResourceLoaderFileModule.php '1f761e3bf349df8d48592f71ed4afff63471c6b4'
• 21:44 mwalker: resuming purge of CentralNotice cache (now working) with rate limit of 50hps. estimated completion in 3hrs.
• 21:29 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41016) Temporarily reenable wgUseNPPatrol on fi.wikipedia'
• 20:53 logmsgbot_: reedy synchronized php-1.21wmf2/resources/
• 20:53 logmsgbot_: reedy synchronized php-1.21wmf1/resources/
• 20:52 cmjohnson1: shutting down es10 for per rt3724
• 20:49 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ArticleFeedback
• 20:49 mutante: flushing mobile varnish cache
• 20:43 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ArticleFeedback
• 20:37 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
• 20:37 mwalker: rate limiting for CentralNotice purge is working too well... ie: script is not doing anything -- debugging
• 20:35 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/ 'touch files'
• 20:21 mwalker: Attempting to purge out another 500k pages with CentralNotice skin problem (this time not crashing the cluster by filtering out media files and rate limiting to no more than 50hps). Estimated run time is 3 hours.
• 20:20 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/ 'touch files'
• 20:10 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es10, third crash in three weeks'
• 20:07 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection/ 'Revert back to 9f37005e78a068821c45d9d313394c6728f55b4a'
• 20:05 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/Collection/ 'Revert back to 9f37005e78a068821c45d9d313394c6728f55b4a'
• 19:57 cmjohnson1: powering down srv266 troubleshooting per rt2896
• 19:04 binasher: restarted udp2log on oxygen with correct --recv-queue
• 18:58 logmsgbot_: reedy synchronized wmf-config/
• 18:55 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
• 18:49 RoanKattouw: Created fluorine:/a/mw-log/udp2log/resourceloader.log chmod 644 and owned by udp2log:udp2log as root per Reedy's request
• 18:19 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/CodeReview
• 18:11 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf2
• 18:07 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend
• 17:32 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgDebugLogGroups for resourceloader'
• 17:30 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
• 16:28 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MobileFrontend/
• 16:26 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend/
• 16:16 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable education program on enwiki'
• 16:06 logmsgbot_: reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.21wmf2
• 15:43 ottomata: stopping puppet on oxygen, looking for cause of recent packet loss
• 15:19 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MobileFrontend
• 15:17 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend
• 15:12 logmsgbot_: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.21wmf2
• 14:56 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf2
• 14:55 logmsgbot_: reedy synchronized wmf-config/
• 14:51 logmsgbot_: reedy synchronized php-1.21wmf2/ 'Initial sync'
• 13:35 Reedy: Killing php-1.20wmf11 l10n cache from apaches
• 13:34 Reedy: Killing php-1.20wmf11 l10n cache from apaches
• 13:31 Reedy: Pointing php to php-1.21wmf1
• 13:29 logmsgbot_: reedy synchronized wmf-config/throttle.php 'Updated from gitrepo'
• 13:28 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Remove extdist conditional for mount path'
• 11:49 mark: Deployed range_offset_limit 64 MB change (from 0) to all upload backend squids
• 11:24 paravoid: restarting pdns @ ns2
• 11:16 paravoid: switching European bits back to esams
• 09:48 paravoid: restarting all imagescaler apaches, did not recover after swift outage
• 09:24 paravoid: live-hacking swift on ms-be10, ms-be12, ms-be1 to remove fadvise calls
• 07:53 Tim: increasing worker count to 100 on all swift backends, via puppet
• 07:37 Tim: ms-be11 showed an immediate improvement in bandwidth out, but wchan still indicates that 48 is not enough, increasing to 100
• 07:33 Tim: experimentally doubling the worker count on ms-be11 since wchan indicates that the worker pool is exhausted
• 05:49 apergos: ms-be12 reporting errors on disk /dev/sdi, I've rt'ed it an unmounted it in the meantime
• 04:34 Tim: on ms-fe2: restarted swift-proxy due to mem leak
• 04:28 Tim: on ms-fe3: restarting swift-proxy due to swap
• 04:26 LeslieCarr: restarted swift proxy on ms-fe4 - it was in swap doom
• 02:25 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 15 02:25:12 UTC 2012

October 14

• 21:57 logmsgbot_: reedy synchronized php-1.21wmf1/includes/specials/SpecialUserlogin.php
• 14:26 logmsgbot_: reedy synchronized php-1.21wmf1/cache/l10n/
• 14:22 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 14 14:22:54 UTC 2012
• 02:00 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 13

• 21:39 logmsgbot_: preilly Finished syncing Wikimedia installation... : update zero rated mobile access
• 21:18 logmsgbot_: preilly Started syncing Wikimedia installation... : update zero rated mobile access
• 21:06 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
• 18:51 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to master'
• 18:41 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original again..'
• 18:38 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to f980f2380b830e712edfdbce065784d1bcba82fd'
• 18:36 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 4f898444ce5d9c2c9a5491f2303db15a3143a5c5'
• 18:34 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b'
• 18:26 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original'
• 18:25 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 8415a4564bf429b76e113368a3e5e1c19fb84a47'
• 18:23 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b'
• 18:21 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 9f37005e78a068821c45d9d313394c6728f55b4a'
• 16:34 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
• 16:27 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
• 13:17 logmsgbot_: reedy synchronized wmf-config/throttle.php
• 02:00 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 12

• 23:51 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'enabling redis debug log'
• 23:40 LeslieCarr: moving traffic back to esams via authdns-scenario
• 23:40 LeslieCarr: reenabled nagios alerts
• 23:20 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'enabling session multiwrites to redis'
• 23:16 LeslieCarr: rebooting esams core switch stack, expect reachability problems to esams
• 23:01 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'disabling session multiwrites to redis'
• 22:36 LeslieCarr: switching europe traffic to pmtpa via authdns-scenario
• 22:36 LeslieCarr: muted nagios
• 20:20 AaronSchulz: Confirmed all swift -> netapp sync scripts finished (s4 one kept dying to random http errors)
• 19:58 logmsgbot_: reedy synchronized wmf-config/
• 19:01 cmjohnson1: powering down search32 to replace cpu1
• 18:33 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'AFT (non v5) doesn't use clicktracking now'
• 18:28 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ArticleFeedback/
• 18:19 cmjohnson1: troubleshooting db42 w/IBM tech
• 17:46 cmjohnson1: pc3 shutting down to replace hdd's with ssd's rt3703
• 17:29 cmjohnson1: pc2 shutting down to replaced hdd w/ ssd rt3703
• 16:27 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Fix extdist mount related warnings'
• 16:14 cmjohnson1: running installer on storage3
• 15:54 paravoid: moving bits from esams to pmtpa (authdns normal scenario)
• 15:52 cmjohnson1: storage 3 replacing disk 12 and rebuilding raid
• 15:32 mark: Disabled csw1-esams:e8/2
• 14:37 mark: Rebooting cr1-esams
• 14:17 mark: Upgrading JunOS on cr1-esams to 11.4R5.5, preparing for reboot
• 13:59 mark: Activated aggregate generation on cr1-esams
• 13:49 mark: Moved AS38930 transit from csw1-esams:e8/4 to cr1-esams:xe-1/3/0
• 13:24 cmjohnson1: storage3 powering down
• 12:27 mark: Upgraded new stack member asw-oe11-esams to junos 11.4R5.5 and rebooted it
• 12:16 mark: Added new EX4200-48T asw-oe11-esams to csw2-esams stack
• 11:08 mark: Disabled csw2-oe12-esams:vcp-1
• 02:01 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed
• 01:21 mutante: moving ancient copy of bugzilla out of /h/w/htdocs (to /root/backup on fenari temp. just in case)
• 01:15 mutante: deleting old document root dirs on fenari. /home/wikipedia/htdocs/noc and noc.old (use /h/w/common/docroot/noc and mediawiki-config repo)
• 01:08 mutante: restarting Apache on fenari for noc.wm change
• 00:42 Jeff_Green: switched payments back to pmtpa and increased TTL to 5 min
• 00:37 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'And back to master'
• 00:31 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert due to error'
• 00:29 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Update collection to master'
• 00:09 cmjohnson1: storage3 being reimaged

October 11

• 23:19 mutante: manually copy index.html for noc from /common/docroot/noc to /h/w/htdocs/noc/
• 23:15 logmsgbot_: dzahn synchronized docroot/noc/index.html
• 23:13 logmsgbot_: dzahn synchronized docroot/noc/index.html
• 23:08 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
• 22:16 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
• 21:58 logmsgbot_: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master
• 21:36 logmsgbot_: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master
• 20:48 Jeff_Green: flipping payments.wikimedia.org to eqiad cluster to live-test
• 20:12 mutante: rebooting analytics1007
• 19:02 notpeter: stopping puppet on brewster
• 18:50 RobH: i need to pull virt1001-1003 out of nagios later, as they are now renamed
• 16:52 notpeter: restarting pdns on ns0
• 16:50 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
• 16:50 notpeter: dist-upgrade and reboot on oxygen
• 16:49 K4-713: updated the payments cluster to 4d0223ebb48f9b
• 16:39 RobH: authdns update for pc1001-1003 mgmt ip
• 15:35 RobH: working on analytics1007 per rt 3582
• 14:55 RobH: shutting down db1012 to remove evaluation ssd card
• 14:14 paravoid: apt: including php-parsekit, python-async, python-git, python-gitdb, python-smmap backports for precise
• 08:59 apergos: restarted swift proxy on ms-fe3 and ms-fe2 with some delay between. note I had to shoot stale processes from Sept on all these boxes as well
• 08:50 apergos: restarted ms-fe4 swift-proxy
• 04:13 mutante: activated W3 Total Cache performance plugin in Wordpress (blog.wikimedia.org)
• 04:12 mutante: marmontel (blog): fixed some file permissions in ./blog/wp-content for www-data the wordpress caching plugin needs to be re-activated
• 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 11 02:27:51 UTC 2012
• 01:55 LeslieCarr: moved traffic back to normal with authdns-scenario
• 01:48 LeslieCarr: reenabling nagios notifications
• 01:22 LeslieCarr: upgrading csw2-esams member 1
• 01:22 LeslieCarr: upgrading csw2-esams member 0
• 01:13 binasher: disabled nagios notifications
• 01:04 LeslieCarr: upgrading csw2-esams member 2
• 01:02 LeslieCarr: upgrading csw2-esams member 3
• 00:44 binasher: restarted pdns on ns2
• 00:44 LeslieCarr: restarting dns on ns2
• 00:33 LeslieCarr: draining esams via authdns-scenarios
• 00:11 logmsgbot_: reedy synchronized php-1.21wmf1/includes/api/ApiUpload.php

October 10

• 23:02 RobH: db1008.mgmt returned to service
• 23:02 RobH: msw-a2-eqiad has a bad set of ports, needs replacement per rt 3683
• 22:46 RobH: reboot seems to have restored system (is posting) but drac still not responding, possible bad cable, will test shortly
• 22:40 RobH: db1008 being cold reset (removing power cords)
• 22:05 mutante: apt-get dist-upgrade on marmontel (blog)
• 21:20 LeslieCarr: undraining esams via authdns-scenario normal
• 20:43 LeslieCarr: draining esams via authdns-scenario for upgrade
• 20:20 Jeff_Green: returning payments to pmtpa
• 20:14 Jeff_Green: flipped payments.wikimedia.org to the new eqiad cluster, live testing
• 20:06 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/javascripts/common/mf-application.js 'touchfile'
• 18:46 cmjohnson1: storage3 power cycling
• 18:24 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of wikipedia projects to 1.21wmf1
• 17:58 logmsgbot_: reedy synchronized php-1.20wmf12/cache/interwiki.cdb 'Update interwiki cache'
• 17:58 logmsgbot_: reedy synchronized php-1.21wmf1/cache/interwiki.cdb 'Update interwiki cache'
• 17:15 notpeter: created indices for fdcwiki on searchidx2 and searchidx1001
• 17:13 logmsgbot_: reedy synchronized docroot/fdc/
• 16:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Add meta as import source for fdcwiki'
• 16:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgServer for fdcwiki'
• 15:24 logmsgbot_: reedy ran sync-common-all
• 14:48 logmsgbot_: reedy synchronized s3.dblist
• 14:39 logmsgbot_: reedy synchronized wmf-config/
• 14:37 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
• 12:40 mark: Throttled SnapMirror originals transfer to 75 MB/s
• 12:20 mark: Increased SnapMirror TCP window size on nas1-a and nas1001-a for the originals snapmirror relationship, and restarted the initial transfer at the last checkpoint
• 06:28 apergos: restarted lucene search on search1016
• 03:01 binasher: completed fa_sha1 migration
• 02:41 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Wed Oct 10 02:41:09 UTC 2012
• 02:20 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 10 02:20:42 UTC 2012
• 00:02 binasher: started filearchive.fa_sha1 migration, starting with s1

October 9

• 23:34 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
• 23:32 logmsgbot_: tstarling synchronized php-1.21wmf1/includes/parser/ParserCache.php
• 23:31 logmsgbot_: tstarling synchronized php-1.20wmf12/includes/parser/ParserCache.php
• 23:30 logmsgbot_: reedy synchronized php-1.20wmf12/resources/startup.js 'touch'
• 23:30 logmsgbot_: reedy synchronized php-1.20wmf12/resources/jquery/jquery.js '1.8.2'
• 23:28 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
• 23:20 mutante: flushing mobile varnish cache
• 23:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Add outreachwiki to import sources of enwiki'
• 23:17 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/
• 23:11 logmsgbot_: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable sticky cookie fix on all wikis; Update logo asset path for mobile footer'
• 23:06 logmsgbot_: dzahn synchronized docroot/fdc/w
• 22:51 logmsgbot_: dzahn gracefulled all apaches
• 22:50 logmsgbot_: dzahn synchronized docroot/fdc/index.php
• 22:41 logmsgbot_: dzahn synchronized /docroot/fdc
• 22:38 logmsgbot_: reedy synchronized php-1.21wmf1/resources/startup.js 'touch'
• 22:26 logmsgbot_: reedy synchronized php-1.21wmf1/resources/jquery/jquery.js 'jQuery 1.8.2'
• 22:21 mutante: creating new doc root for fdc wiki /usr/local/apache/common/docroot/fdc
• 22:10 notpeter: readded mw61 to bits backends
• 21:53 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
• 21:29 cmjohnson1: swapping disk 12 out on storage 3
• 21:07 LeslieCarr: restarting apache2 on srv220
• 20:31 mutante: authdns-update on dobson, adding fdc.wikimedia.org DNS entry
• 19:36 Jeff_Green: added a couple temporary filters to oxygen for fr-tech
• 17:28 notpeter: reimaging mw61 with precise for bits test
• 16:29 Aaron|home: Set abbrvThreshold to 160 for thumbnails
• 16:27 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 160 for thumbnails.'
• 16:22 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 180 for thumbnails.'
• 16:18 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 200 for thumbnails.'
• 13:31 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Pre-emptively set $wgContentHandlerUseDB = false;'
• 11:35 mark: Setup sync SnapMirror relationship from nas1-a:originals to nas1001-a:originals and started transfer
• 11:29 mark: Increased maxfiles of nas1001-a:originals to 63753378
• 11:27 mark: Created 36 drive media1 raid-dp aggregate on nas1001-a, hosting new 40T volume 'originals'
• 10:59 paravoid: rolling restart of swift proxies (ms-fe2/3/4) because of the memory leaks
• 09:32 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 40366 - Narayam'
• 09:25 logmsgbot_: nikerabbit synchronized php-1.21wmf1/extensions/Translate/api/ApiGroupReview.php
• 02:50 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Tue Oct 9 02:50:29 UTC 2012
• 02:25 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 9 02:25:49 UTC 2012

October 8

• 20:47 Aaron|home: Started syncFileBackend.php (swift -> nas) on screens on hume (s4)
• 20:41 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/MobileFrontend.php 'Live hack to unbreak prop=extracts on 1.21wmf1'
• 20:36 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ConfirmEdit/FancyCaptcha.class.php 'Revert cache key change live hack'
• 20:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgOldChangeTagsIndex to false for all wikis running with new indexes'
• 19:22 Aaron|home: Started syncFileBackend.php (swift -> nas) on screens on hume (s1-s3,s5-s7)
• 19:13 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgOldChangeTagsIndex to false for outreachwiki'
• 19:07 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
• 19:05 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
• 18:44 Reedy: Updated liquidthreads_labswikimedia CategoryLinks schema with patch-categorylinks-better-collation.sql
• 18:36 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf1
• 16:43 logmsgbot_: reedy synchronized wmf-config/
• 11:49 apergos: manually cleaned up /tmp on mw49, forced a puppet run too (isn't there a cron job to clean up /tmp?), stopped job runner afterwards since it had been stopped already by someone
• 11:31 Reedy: Manually running sync-common on mw49
• 11:14 logmsgbot_: ariel synchronized wmf-config 'move from ms7 upload6 to nas1 upload7'
• 02:52 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Mon Oct 8 02:52:16 UTC 2012
• 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 8 02:27:45 UTC 2012

October 7

• 18:09 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'move captchadir to upload7'
• 10:11 logmsgbot_: maxsem synchronized wmf-config 'Raise creation throttle for India event even more'
• 02:48 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Sun Oct 7 02:47:58 UTC 2012
• 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 7 02:28:29 UTC 2012

October 6

• 15:56 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
• 14:14 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
• 03:11 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Sat Oct 6 03:11:10 UTC 2012
• 02:39 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sat Oct 6 02:39:31 UTC 2012
• 02:17 Reedy: Manually re-running localisation update
• 02:02 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 5

• 20:44 pp-pdf1: updated mwlib.zim to 0.3.0
• 20:02 notpeter: removing srv190 from imagescaling pool due to misbuilt pacakge
• 18:21 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ProofreadPage
• 17:22 Reedy: Live hacked 1.21wmf1 ConfirmEdit/FancyCaptcha.class.php cache key for directory list. Needs reverting after 24 hours or so
• 17:17 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ConfirmEdit/FancyCaptcha.class.php 'New memcached key'
• 17:06 logmsgbot_: reedy synchronized wmf-config/
• 14:00 notpeter: restarting puppet on brewster
• 13:54 mark: Rollback of upload-lb.eqiad; sending traffic back to pmtpa
• 13:48 notpeter: stopping puppet on brewster
• 13:42 logmsgbot_: ariel synchronized wmf-config 'rollback ms7 to nas1 upload6 to upload7 migration'
• 12:17 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgCacheEpoch to 20120908000000'
• 12:04 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Update wgCacheEpoch and wgThumbnailEpoch to the start of 2012 (+ 1 year)'
• 12:04 apergos: started one last rsync op upload/ from ms7 to nas1 with --update
• 11:46 mark: Moved /mnt/upload7/upload/* to /mnt/upload7, removed upload /mnt/upload7/upload dir
• 11:42 logmsgbot_: ariel synchronized wmf-config 'migrate media etc from ms7 upload6 to nas1 upload7'
• 11:41 apergos: running sync-dir wmf-config to migrate media etc from ms7 upload6 to nas1 upload7
• 02:46 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Fri Oct 5 02:45:59 UTC 2012
• 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Fri Oct 5 02:27:21 UTC 2012
• 01:03 mutante: updating a lot of mailing list descriptions per BZ:37537 / thehelpfulone

October 4

• 23:59 mutante: manually fixed brion's ssh key on fenari. session opened for user brion. welcome back :)
• 22:06 mutante: powercycling downed cp1044
• 21:40 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
• 21:28 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
• 20:41 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/ClickTracking/modules/jquery.clickTracking.js
• 20:18 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
• 18:09 logmsgbot_: catrope synchronized php-1.21wmf1/languages/messages/MessagesEn.php 'Ib8746ece3a34b5e577dc08596b0ff7b1b96b5d73'
• 18:08 logmsgbot_: catrope synchronized php-1.21wmf1/resources/mediawiki/mediawiki.feedback.js 'Ib8746ece3a34b5e577dc08596b0ff7b1b96b5d73'
• 16:16 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
• 15:30 logmsgbot_: reedy synchronized wmf-config/
• 14:02 Reedy: Manually ran sync-common on srv247
• 13:34 notpeter: removing srv191 from bits pool for test upgrade to precise
• 10:39 Tim: gave myself an account on sanger's exim so that I can send outbound mail through it
• 05:24 apergos: restarted lucene search on search1016 again
• 02:49 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Thu Oct 4 02:49:23 UTC 2012
• 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 4 02:28:12 UTC 2012
• 00:14 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection/Collection.body.php

October 3

• 22:47 mutante: reinstalling analytics1007
• 21:38 mutante: deleting education-ar mailman list per bug 40749
• 20:55 logmsgbot_: dzahn synchronized wmf-config/throttle.php 'IP cap lift for Grace Hopper event, bug 40669'
• 20:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia to 1.21wmf1
• 20:15 mutante: powercycling analytics1007
• 20:15 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikimedia and wiktionary to 1.21wmf1
• 20:13 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf1
• 20:12 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikiquote to 1.21wmf1
• 20:09 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, private and closed to 1.21wmf1
• 19:33 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
• 19:32 logmsgbot_: preilly synchronized php-1.20wmf12/extensions/MobileFrontend 'update for landing page'
• 19:29 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess 'update for landing page'
• 19:28 logmsgbot_: preilly synchronized php-1.20wmf12/extensions/ZeroRatedMobileAccess 'update for landing page'
• 15:43 Reedy: Close strategywiki, zawiktionary, zh_min_nanwikibooks
• 15:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'WikiLove on itwikiquote'
• 15:36 Reedy: Created wikilove tables to itwikiquote
• 15:18 Jeff_Green: test-deploying SPF DNS records
• 09:49 mark: Doubled number of inodes on nas1-a:originals from 31876689 to 63753378
• 08:38 apergos: stopped ms7 to ms8 replication
• 08:18 apergos: rebooting ms8 to unstick deadlocked zfs recv and zfs list processes
• 07:20 Tim: upgrading luasandbox on all apaches to version 1.1-1
• 06:28 apergos: restarted lucene search on search1016
• 05:22 logmsgbot_: tstarling synchronized php-1.21wmf1/cache/trusted-xff.cdb
• 05:19 logmsgbot_: tstarling synchronized php-1.20wmf12/cache/trusted-xff.cdb
• 03:11 logmsgbot_: tstarling synchronized wmf-config/db.php 'es10 back up'
• 03:03 Tim: on es10: started mysqld
• 02:51 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 3 02:51:35 UTC 2012
• 02:45 Tim: es10 not responding to ssh, rebooting
• 02:43 logmsgbot_: tstarling synchronized wmf-config/db.php
• 02:31 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Wed Oct 3 02:31:56 UTC 2012
• 02:09 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php 'API log'
• 01:35 logmsgbot_: krinkle synchronized live-1.5
• 01:29 logmsgbot_: tstarling synchronized live-1.5
• 01:25 Tim: on srv247, srv270, srv271, srv286, mw69: killing more stuck apt-get processes, will run apt-get upgrade
• 01:19 Tim: on srv233: killed frozen apt-get, running since August 17, running apt-get upgrade
• 01:17 logmsgbot_: krinkle Finished syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f
• 01:09 Tim: upgrading wikimedia-task-appserver on all apaches
• 01:06 Tim: on brewster: updated wikimedia-task-appserver
• 00:42 mutante: moved logmsgbot to -operations channel
• 00:34 logmsgbot: krinkle Started syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f
• 00:25 Reedy: Restarted Jenkins

October 2

• 23:45 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js 'Updating event format for OpenTask funnel'
• 23:44 logmsgbot: reedy synchronized php-1.21wmf1/extensions/Collection/
• 23:19 Ryan_Lane: deploying Damianz's OpenStackManager change for adding/removing bastion users on shell access on labsconsole
• 16:17 logmsgbot: asher synchronized wmf-config/db.php 'adding db64 to s3 in place of db25'
• 16:02 mark: Reclaimed space on brewster:/ - restarted squdi
• 15:19 mark: Sending ALL upload traffic to upload-lb.eqiad (Varnish)
• 14:57 mark: Sending traffic from India, Korea, China, Singapore, Thailand, Vietnam to upload-lb.eqiad (Varnish)
• 14:26 mark: Sending Mexican upload traffic to upload-lb.eqiad (Varnish)
• 14:12 mark: Sending Brazil upload traffic to upload-lb.eqiad (Varnish)
• 14:10 mark: Restarted pdns on linne
• 14:03 mark: Sending Japanese upload traffic to upload-lb.eqiad (Varnish)
• 13:57 notpeter: rebooting srv194-199 and mw55-59 to pick up kernel upgrades, and then re-adding them to apaches pool
• 13:10 notpeter: rebooting mw1-16 to pick up kernel upgrades before re-enabling them as (precise) jobrunners
• 13:05 notpeter: enabling srv190 in imagescaler pool at half-weight to test precise image-scaling
• 11:52 paravoid: apt: include new php5 built with libicu42 (precise-wikimedia/main), include libicu42
• 11:46 mark: Moving US upload traffic to upload-lb.eqiad (Varnish)
• 09:06 paravoid: squid deploy all: switching math & timeline to be served from swift
• 05:23 binasher: started replicating db64 from the s3 master
• 03:51 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js
• 03:28 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js
• 02:37 logmsgbot: LocalisationUpdate completed (1.21wmf1) at Tue Oct 2 02:37:57 UTC 2012
• 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Tue Oct 2 02:21:28 UTC 2012
• 01:56 Tim: send sessions to both the existing memcached and also redis, to fill redis and allow for a graceful complete switch later
• 01:55 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
• 01:41 Tim: restarting all apaches to get redis PHP extension
• 01:35 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
• 01:33 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
• 01:16 binasher: streaming hotbackup of db1019 to db64 (s3)
• 01:12 binasher: stopping mysql on db25 and powering down - logging lots of ecc chipkill errors and periodically unresponsive. building db64 as a perm replacement
• 01:00 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Updating MobileFrontend, per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-01
• 00:50 logmsgbot: asher synchronized wmf-config/db.php 'pulling db25, returning es10'
• 00:07 logmsgbot: awjrichards Started syncing Wikimedia installation... : Updating MobileFrontend, per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-01

October 1

• 22:10 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Eanble wgMFEnableDesktopResources on testwiki'
• 22:09 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Eanble wgMFEnableDesktopResources on testwiki'
• 21:40 logmsgbot: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 21:18 logmsgbot: preilly synchronized php-1.20wmf12/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 20:44 logmsgbot: reedy synchronized php-1.21wmf1/includes/specials/SpecialUserlogin.php 'bug 40668'
• 20:23 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
• 20:21 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
• 20:16 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
• 20:13 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
• 19:33 paravoid: pushing new rewrite.py to swift servers (r24303, r24514, r24576)
• 19:31 logmsgbot: reedy synchronized php-1.21wmf1/includes/job/Job.php
• 19:24 logmsgbot: reedy synchronized php-1.20wmf12/includes/job/Job.php
• 18:51 notpeter: stopping puppet on brewster
• 18:22 logmsgbot: asher synchronized wmf-config/db.php 'pulling es10, crashed over weekend, needs to catchup on repl'
• 18:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf1
• 15:52 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
• 15:40 ^demon: restarted gerrit on manganese.
• 15:36 ^demon: gracefully shut down gerrit until we figure out why manganese is swapping
• 14:18 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
• 14:08 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
• 14:06 mark: Added cp1029 and cp1030 to the frontend varnish upload pool in eqiad (PyBal)
• 13:59 logmsgbot: reedy synchronized php-1.21wmf1/
• 13:53 logmsgbot: reedy synchronized live-1.5/
• 13:53 logmsgbot: reedy synchronized wmf-config/
• 13:48 logmsgbot: reedy synchronized php-1.21wmf1
• 13:10 cmjohnson1: powering down db62 to replace raid controller card
• 12:38 logmsgbot: reedy synchronized live-1.5
• 12:34 Reedy: Make that, deleting wmf10 php source files
• 12:32 Reedy: Deleting wmf12 php source files
• 12:30 Reedy: Deleting wmf11 localisation cache
• 11:11 mark: Added cp1029 and cp1030 to the Varnish upload.eqiad pool (10 mins ago)
• 09:17 mark: Rebooting cp3001, both ports in LACP block
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Mon Oct 1 02:25:14 UTC 2012

September 30

• 23:58 MaxSem: Ran mwscript extensions/WikimediaMaintenance/migrateWlmUploads.php testwiki --do-it>wlm.log
• 19:26 logmsgbot: maxsem synchronized wmf-config 'Remove WLM banner / https://gerrit.wikimedia.org/r/#/c/25771/'
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sun Sep 30 02:23:14 UTC 2012

September 29

• 10:39 mark: Started rsync of /export/upload (respecting /etc/rsync.includes) to /mnt (nas1-a:vol/originals/) on ms7 in a screen session
• 10:35 mark: Manually mounted nas1-a:/vol/originals on ms7, fenari
• 10:33 mark: Created 40 TB volume 'originals' inside 36-drive aggregate 'media1' on nas1-a
• 04:07 mutante: reinstalled cp1031 (new mainboard, thx RobH)
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sat Sep 29 02:24:45 UTC 2012
• 01:59 mutante: restarting cp1031 after hardware replacement
• 01:24 mutante: checking/fixing puppet runs on cp103x servers

September 28

• 23:54 binasher: rebooting db1047 to new kernel
• 21:28 apergos: disbaled puppet and stopped swift processes on ms-be6 again. noticed four drives in "Unconfigured" state in megacli output after disk replacement,don'tknow more details about how that went.
• 20:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 20:04 mark: Destroyed nas1001-a:images volume, containing aggregate test0, and started zeroing drives
• 20:02 mark: Destroyed test0 aggregate on nas1-a, zeroing disks
• 19:55 mark: Destroyed volume nas1-a:images
• 19:54 MaxSem: And one more autocreation + re-running the migration script
• 19:51 mark: Destroyed snapmirror relationship between nas1001-a:images -> nas1-a:images and deleted related snapshots
• 19:51 MaxSem: Autocreated Commons accounts for 3 testwiki users to migrate their images
• 19:24 MaxSem: running mwscript extensions/WikimediaMaintenance/migrateWlmUploads.php testwiki --do-it
• 19:03 logmsgbot: reedy synchronized wmf-config/
• 19:01 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 18:55 logmsgbot: reedy synchronized wmf-config/
• 18:38 logmsgbot: reedy synchronized wmf-config/
• 18:04 binasher: stopping mysql on db1047 for upgrades
• 16:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgCopyUploadProxy and enable copyupload on testwiki'
• 16:17 logmsgbot: reedy synchronized php-1.20wmf12/includes/upload/UploadFromUrl.php
• 16:16 logmsgbot: reedy synchronized php-1.20wmf12/includes/DefaultSettings.php
• 15:46 RobH: done with db62. it was in the ubuntu installer when I took it over, so its just sitting now
• 14:53 RobH: db62 being pulled for drac7 work
• 14:39 Jeff_Green: rebooting aluminium
• 13:11 mark: Moved BGP transit sessions to AS1257 from cr2-eqiad (over equinix exchange) to cr1-eqiad (dedicated link)
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Fri Sep 28 02:26:27 UTC 2012

September 27

• 23:13 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/Vector/modules/ext.vector.footerCleanup.css
• 22:50 logmsgbot: olivneh synchronized wmf-config
• 22:05 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
• 22:00 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update PageTriage and ArticleFeedbackv5 to master
• 21:32 logmsgbot: bsitu Started syncing Wikimedia installation... : Update PageTriage and ArticleFeedbackv5 to master
• 20:48 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning off moodbar on testwiki'
• 20:09 RobH: authdns-update for new misc servers mgmt in eqiad
• 19:10 cmjohnson1: search32 going down to swap cpu's (dell request)
• 18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
• 17:42 cmjohnson1: bring db62 down to swap disk controller card per ct/asher
• 17:16 logmsgbot: demon synchronized wmf-config/throttle.php 'Syncing I8c44415b: Account creation throttle for ptwikiversity event'
• 16:42 cmjohnson1: ms-be6 powering down
• 16:37 RobH: cp1031 mainboard replaced per rt3614, will need reinstall
• 15:59 RobH: mc1006 memory repaired rt3613
• 15:53 RobH: mc1002 repaired rt3612
• 15:33 RobH: mc1002 (rt3612) & mc1006 (rt3613) offline for memory repair/swap
• 15:20 RobH: authdns-update for wikimediafoundation.info
• 15:06 Jeff_Green: indium dist-upgrade & reboot
• 13:58 Jeff_Green: dist-upgrade & reboot grosley
• 10:55 apergos: ignore ms-be6 messages, I'm trying to get into the bleeping lsi raid util
• 04:04 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/postEditFeedback.js
• 02:36 Jeff_Green: powercycle fenari, it's nonresponsive even via drac terminal

September 26

• 20:42 RobH: mc1011 drac fixed
• 20:39 RobH: mc1009, 1010 drac fixed
• 20:30 RobH: cp1031 will remain offline until replacement parts arrive tomorrow rt3614
• 18:44 RobH: correction rt 3613
• 18:43 RobH: mc1006 testing done, single bad dimm, support ticket filed rt3614
• 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 285 remaining wikis to 1.20wmf12
• 18:06 RobH: mc1006 offline for memory troubleshooting per rt 3613
• 16:52 RobH: mc1002 offline, memory bad per rt3612, new memory will arrive tomorrow
• 16:14 RobH: carbon rebooting for console redirection issues, any eqiad based installs will fail during this downtime
• 15:57 Jeff_Green: dist-upgrade & reboot most payments boxes
• 15:21 RobH: bast1001 link speed fixed, done working on it
• 15:19 RobH: troubleshooting bast1001 link speed per 3414
• 15:12 cmjohnson1: sq37 has fatal error...powering down to replace disk controller card
• 13:56 cmjohnson1: ms-be8 powercycling
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 26 02:48:22 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Wed Sep 26 02:23:25 UTC 2012
• 01:00 mutante: reinstalling cp1030 with precise
• 00:34 mutante: fixing puppet on singer..finally

September 25

• 23:25 binasher: ran "varnishadm param.set nuke_limit 300" on all mobile varnish front and back instances to match new default config
• 23:20 binasher: repooled cp1044
• 23:16 binasher: depooling cp1044 from lvs for testing
• 22:44 binasher: stopping puppet on cp1044, experimenting with varnish lru / nuke params to solve allocation failures
• 22:28 mutante: generating locales on singer, add ro UTF-8 locale, run ro.planet
• 21:50 Jeff_Green: dropped apparently-unused IP alias on loudon, deprecated from old haproxy install
• 21:50 Jeff_Green: apt-update and reboot loudon
• 21:36 Jeff_Green: replaced iptables ruleset for loudon
• 21:02 notpeter: stopping puppet on brewster
• 20:49 mutante: sync-common-file new favicon.ico for wikidata.org
• 20:49 logmsgbot: dzahn synchronized docroot/www.wikidata.org/favicon.ico 'updating wikidata favicon per Lydia'
• 17:57 Jeff_Green: restarted udp2log on locke to clean up ~20 defunct procs
• 17:56 notpeter: temp stopping puppet on brewster
• 17:27 mutante: adding otto to ops group in LDAP
• 14:39 cmjohnson1: ms-be7 shutting down to check jumpers
• 14:35 cmjohnson1: ms-be8 shutting down to check jumpers
• 12:14 mark: Powercycled carbon, has been down for 4d
• 05:58 Tim: updated Scribunto for bug 40480
• 05:45 paravoid: upgrading all PDF boxes for USN-1581-1/CVE-2012-4405
• 05:45 paravoid: upgrading all imagescalers for USN-1581-1/CVE-2012-4405

September 24

• 19:44 RobH: db1012 card install done, system powered down for asher
• 19:36 RobH: db1012 shutting down for pci card install per rt3558
• 19:14 logmsgbot: reedy synchronized php-1.20wmf12/extensions/TimedMediaHandler 'Updating to master'
• 18:54 pgehres: added new columns to fundraising contribution_tracking table
• 18:20 notpeter: temp stopping puppet on brewster
• 18:15 logmsgbot: hashar synchronized wmf-config/filebackend-wmflabs.php 'beta: disable commons as a default foreign file repo'
• 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf12
• 17:03 cmjohnson1: powering up and toying with ms-be8 for h/w checks
• 16:35 RobH: mc1001-1008 memory upgrade complete
• 16:35 cmjohnson1: ms-be6 shutting down for h/w testing
• 15:51 RobH: mc1001-mc1008 are having memory upgrades installed. hosts already offline.
• 13:53 pp-pdf1: restart all services
• 13:52 pp-pdf1: updated mwlib.epub to 0.14.2
• 13:52 pp-pdf1: updated mwlib.rl to 014.1
• 13:52 pp-pdf1: update mwlib to 0.14.1
• 13:51 pp-pdf1: updated timelib to 0.2.4
• 13:50 pp-pdf1: updated pip to 1.2.1
• 13:50 pp-pdf1: updated distribute to 0.6.28
• 13:17 LeslieCarr: switching unnumbered GRE tunnel to numbered GRE Tunnel between cr2-knams and cr2-eqiad for routability reasons
• 12:45 LeslieCarr: restarting powerdns on ns2
• 09:34 logmsgbot: hashar synchronized docroot/bits/static-master/ '(bug 40419) extension assets not available on beta (aka add docroot/bits/static-master'
• 09:23 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40436) Namespaces configuration for se.wikipedia'
• 09:19 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40186) Favicon for es.wiktionary'
• 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 24 02:53:05 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Mon Sep 24 02:27:55 UTC 2012
• 01:53 Tim: on hume: running updateCollation.php with --force for bug 40462
• 01:50 Tim: depooled precise appservers and stopped job runners on precise servers

September 23

• 23:26 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'Reduce wgMaxGeneratedPPNodeCount to 1.5M'
• 17:29 binasher: powercycled cp1043
• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 23 02:45:21 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sun Sep 23 02:26:13 UTC 2012

September 22

• 09:26 mark: Restarted lucene on search1015
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 22 02:46:32 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sat Sep 22 02:27:27 UTC 2012

September 21

• 20:55 notpeter: removing srv190 from apaches pool to use to test build as an imagescaler
• 20:38 logmsgbot: asher synchronized wmf-config/db.php 'moving ES writes from es1 (cluster23) to es2 (cluster24) and es3 (cluster25)'
• 20:29 logmsgbot: asher synchronized wmf-config/db.php 'db.php with new es shards added, but not being written to'
• 20:29 ^demon: restarted gerrit like 10m ago to deploy I03de7c74 (apache timeout increase)
• 20:17 Jeff_Green: apt-get upgrade on manganese, restart apache for adjusted 5->10m gerrit timeout
• 20:15 mutante: analytics1007 seems dead. powercycle but never see any console output on mgmt at all
• 14:49 hashar: labs: cleaned out a 4GB file out of labs-nfs1:/export/home/deployment-prep
• 02:49 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 21 02:49:27 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Fri Sep 21 02:25:33 UTC 2012
• 01:39 logmsgbot: krinkle synchronized php-1.20wmf12/extensions/LiquidThreads 'I5a098a42'
• 01:37 logmsgbot: krinkle synchronized php-1.20wmf12/skins/common/preview.js 'Ia9673a75'
• 00:16 binasher: configured pt-heartbeat on es2 and es3 shards

September 20

• 23:18 logmsgbot: kaldari synchronized php-1.20wmf12/extensions/PageTriage/modules/ext.pageTriage.defaultTagsOptions/ext.pageTriage.defaultTagsOptions.js
• 23:17 logmsgbot: kaldari synchronized php-1.20wmf11/extensions/PageTriage/modules/ext.pageTriage.defaultTagsOptions/ext.pageTriage.defaultTagsOptions.js
• 22:58 Ryan_Lane: restarting gerrit
• 22:36 notpeter: stopping puppet on brewster
• 22:20 logmsgbot: aaron synchronized wmf-config/filebackend.php
• 22:18 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled short thumbnails for testwiki'
• 22:15 logmsgbot: olivneh synchronized php-1.20wmf11/extensions/E3Experiments/experiments
• 22:02 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:32 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 20:32 ottomata: reinstalling analytics1002-analytics1010 with Ubuntu precise
• 20:25 logmsgbot: aaron synchronized php-1.20wmf12/includes/filebackend/SwiftFileBackend.php
• 20:25 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php
• 20:13 mutante: bugs.mediawiki.org|bugzilla.wikipedia.org|bugzilla.mediawiki.org all redirect to https:// rather than http now
• 20:12 logmsgbot: dzahn gracefulled all apaches
• 20:10 mutante: sync-apache and pushing out redirects.conf change for Bugzilla redirects to https
• 20:01 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 19:50 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php 'deployed 8001d1ebf5b79b5a6ea359646284347d31e5442b'
• 19:38 logmsgbot: aaron synchronized php-1.20wmf12/includes/filebackend/SwiftFileBackend.php 'deployed 4246f85ac2a68a19b9fa8f4954c5ff7aabea2922'
• 19:33 logmsgbot: py synchronized wmf-config/lucene.php
• 19:32 notpeter: pushing all search traffic back to eqiad
• 19:18 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 18:58 binasher: ran scap on mw58 and returning to the lvs pool
• 17:08 logmsgbot: aaron synchronized php-1.20wmf12/thumb.php 'deployed fd26ac3603c544c9f2273408b14034cd22b56a52'
• 17:04 notpeter: temp stopping puppet on brewster
• 10:41 hashar: UploadWizard has been broken in some cases following the release of wmf12. Restored by reverting a change. See bug 40380 for details.
• 10:13 logmsgbot: hashar synchronized php-1.20wmf12/resources/mediawiki.language '(bug 40380) inexistent language files are loaded'
• 08:12 logmsgbot: hashar synchronized php-1.20wmf12/includes/EditPage.php '(bug 40098) Wrong edit summary generation when adding a new discussion section'
• 08:11 logmsgbot: hashar synchronized php-1.20wmf11/includes/EditPage.php '(bug 40098) Wrong edit summary generation when adding a new discussion section'
• 08:03 hashar: Jenkins: installing "Gerrit Trigger plugin" 2.6 which replaces our 2.5.3 snapshot
• 05:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
• 05:40 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
• 05:26 logmsgbot: tstarling synchronized php-1.20wmf12/includes/parser/Parser.php 'node limit log'
• 05:23 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/Preprocessor_DOM.php 'node limit'
• 05:22 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/Parser.php 'node limit'
• 05:22 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/ParserOptions.php 'node limit'
• 05:13 Tim: fixing common-local on mw58, has wrong permissions
• 05:11 logmsgbot: tstarling synchronized php-1.20wmf11/includes/DefaultSettings.php 'node limit'
• 03:23 Ryan_Lane: purged search data from dataset2 on diederik's request
• 02:44 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 20 02:44:42 UTC 2012
• 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Thu Sep 20 02:20:57 UTC 2012

September 19

• 22:48 binasher: pulling mw58 from lvs pool and deploy groups for temporary testing
• 22:07 logmsgbot: olivneh synchronized php-1.20wmf11/extensions/E3Experiments
• 21:50 notpeter: temp stopping puppet on brewster
• 21:28 apergos1: ms-be10: after chris installed the disks, filesystems were created ok by puppet and mounted but on a couple of reboots, there were boot messages about two not identical lists of disk not present or not ready; waiting a little bit allowed them to come up but it's not great.
• 20:56 Jeff_Green: restarting gmetad on nickel
• 20:32 Reedy: Deleting php-1.20wmf10 localisation cache from mediawiki-installation
• 20:00 Reedy: Deleting php-1.20wmf9 from mediawiki-installation
• 19:43 logmsgbot: reedy synchronized php-1.20wmf12/extensions/CategoryTree 'Category tree to master'
• 18:12 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Swapping all non-pedias to 1.20wmf12
• 18:07 cmjohnson1: ms-be10 powering down to replace 12 disk drives
• 17:38 logmsgbot: py synchronized wmf-config/lucene.php
• 17:30 logmsgbot: py synchronized wmf-config/lucene.php
• 17:26 logmsgbot: aaron synchronized php-1.20wmf12/thumb.php 'deployed 7be7e496ae2d6e4b082278ab4ace8c1b4b08e433'
• 17:24 notpeter: moving search traffic to pmtpa for eqiad search package upgrades, starting with en
• 15:22 cmjohnson1: updating authdns. adding 12 new misc servers mgmt ip's
• 13:31 mark: Deactivated subnet for vlan 104 (Tampa)
• 12:28 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39264) Add Tudalen: and Indecs: namespaces to cy.wikipedia'
• 12:24 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 38840) Namespaces configuration on uz.wikipedia'
• 12:18 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 38840) Namespaces configuration on uz.wikipedia'
• 12:13 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39206) Namespaces configuration for se.wikipedia'
• 12:02 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40163) Try to fix ltwiki import source for betawikiversity'
• 06:01 paravoid: depooling ms-fe1; broken after precise upgrade; pending further investigation
• 04:46 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation caches for pagetriage
• 04:04 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation caches for pagetriage
• 02:48 logmsgbot: reedy synchronized php-1.20wmf12/extensions/PageTriage
• 02:48 logmsgbot: reedy synchronized php-1.20wmf11/extensions/PageTriage
• 02:42 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Wed Sep 19 02:42:01 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 19 02:22:47 UTC 2012
• 01:27 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
• 01:17 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php

September 18

• 20:17 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'gerrit 23795 disable anon edits on wikimania2012 wiki'
• 20:13 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39652) Fix "autoreviewer" restriction level on ptwiki'
• 20:09 logmsgbot: hashar: ran "mwscript updateArticleCount.php --wiki guwikisource --update" for bug 40173
• 20:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40173) Set wgArticleCountMethod for gu.wikisource'
• 20:04 logmsgbot: hashar synchronized wmf-config/CommonSettings-wmflabs.php 'gerrit 17365 beta: Removes transcoding stub'
• 19:50 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cleanup sep11wiki left over'
• 19:50 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'cleanup sep11wiki left over'
• 19:50 logmsgbot: hashar synchronized closed.dblist 'cleanup sep11wiki left over'
• 19:48 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Cleaning Berlin hackathon tutorial configuration.'
• 19:45 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39942) Disables patrol on fi.wikipedia'
• 19:43 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39905) Create interface editor group on pt.wikipedia'
• 19:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'gerrit 24207 enable EPUB collection format on ALL wiki
• 19:34 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'ran stylize.php on InitialiseSettings.php gerrit 23998'
• 19:13 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Removing live hack to enable EPUB format on all wiki, should use $wmgCollectionUseEpub instead (see our internal mailing list)'
• 19:01 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40186) New logo for es.wiktionary'
• 17:16 logmsgbot: hashar synchronized wmf-config/InitialiseSettings-wmflabs.php 'fix AFTv5 on beta'
• 17:07 notpeter: putting srv190 and mw56-59 back into apaches pool
• 16:52 cmjohnson1: search32 going down to check if it's DIMM error or possible board or cpu1 error
• 15:41 cmjohnson1: srv249 powercycling to check drac configuration
• 14:08 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40122) disable GlobalBlocking on fishbowl, private wikis'
• 13:53 logmsgbot: hashar synchronized php-1.20wmf11/includes/Sanitizer.php 'bug 40306 - Only convert align to float for table'
• 13:52 logmsgbot: hashar synchronized php-1.20wmf11/tests/phpunit/includes/SanitizerTest.php 'bug 40306 - Only convert align to float for table'
• 13:48 apergos1: ms-fe1 back in swift frontend pool, upgrade to precise complete.
• 13:41 hashar: gerrit 23798 manually deleted /apache/common-local/docroot/wikipedia.org/{million,test}.txt files from all mediawiki-installation hosts
• 12:53 logmsgbot: hashar synchronized php-1.20wmf12/includes/Sanitizer.php 'bug 40306 - Only convert align to float for table'
• 12:53 logmsgbot: hashar synchronized php-1.20wmf12/tests/phpunit/includes/SanitizerTest.php 'bug 40306 - Only convert align to float for table'
• 12:22 apergos1: reboot ms-fe1, upgrading to precise
• 05:59 Aaron|home: Last of the math/ file copy scripts has finished
• 02:42 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Tue Sep 18 02:42:15 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 18 02:26:08 UTC 2012
• 00:20 Ryan_Lane: adding an autofs direct map to LDAP, pointing at the gluster storage for testing home directory move

September 17

• 23:43 paravoid: adding a modified ixgbe.ko to precise-installer's initrd.gz; adding ixgbe-dkms to apt
• 23:30 notpeter: starting puppet on brewster
• 23:17 notpeter: stopping puppet on brewster
• 21:45 apergos: (from 5 hours ago) last of the swift proxy services restarted, it had reached 8.6gb of swap use. good for a few more days now
• 20:52 Jeff_Green: stopped puppet on aluminium for config testing
• 20:45 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'enable EPUB'
• 18:55 notpeter: removing srv190 and mw55-59 from apaches pool for upgrade to precise
• 18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf12
• 18:47 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation cache
• 18:20 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation cache
• 18:13 logmsgbot: aaron synchronized php-1.20wmf12/extensions/timeline/Timeline.php
• 18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enabled html5 on all wikis'
• 18:07 binasher: granted 'research' read access to user rollup/filter dbs on dbs 42,1047
• 18:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf12 and rebuild localisation cachae
• 16:36 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf12 and rebuild localisation cachae
• 16:21 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ArticleFeedbackv5/ 'Bring in logging related fixes etc'
• 16:17 logmsgbot: reedy synchronized php-1.20wmf12 'Initial sync'
• 15:31 Reedy: Deleted 1.20wmf10 l10n cache
• 15:28 Reedy: Pointing php to php-1.20wmf11
• 08:21 apergos1: swap was using 7gb on ms-fe1... restarted swift-proxy over there. wlll prolly restart it on ms-fe2 through 4 over the next day or two
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 17 02:26:30 UTC 2012

September 16

• 14:53 mark: powercycled srv256
• 14:53 mark: powercycled srv250
• 14:49 mark: cleared profiling
• 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 16 02:20:02 UTC 2012

September 15

• 19:04 mark: Power cycled srv212, 211, 226, 273, 237
• 19:04 logmsgbot: tstarling synchronized wmf-config/mc.php 'swapping out other slow servers'
• 18:53 logmsgbot: tstarling synchronized wmf-config/mc.php 'swap out srv237 for srv249'
• 18:46 logmsgbot: tstarling synchronized php-1.20wmf11/includes/objectcache/MemcachedClient.php 'memcached profiling'
• 18:37 mark: Reduced MaxRequestsPerChild from 300 to 4 in apache.conf
• 18:29 LeslieCarr: logs stopped on flourine between 18:11 and 18:27 UTC
• 14:03 LeslieCarr: powercycling srv259 (unresponsive)
• 13:57 LeslieCarr: rebooting srv270 (unresponsive)
• 13:36 LeslieCarr: rebooting srv288
• 11:54 logmsgbot: andrew synchronized wmf-config/throttle.php 'Fix undeclared variable error'
• 09:56 logmsgbot: reedy synchronized wmf-config/throttle.php 'Bug 40263 - Request to lift IP address for mass creation - Philippines SFD 2012'
• 08:52 thedj: someone should deploy Planets update, or add blogs to the old config because otherwise WLM has passed before the blogs are added to the stream.
• 04:04 LeslieCarr: started ircd using the instructions here - http://wikitech.wikimedia.org/view/IRCD on ekr�em
• 03:55 LeslieCarr: powercycling ekrem
• 02:29 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 15 02:29:57 UTC 2012

September 14

• 15:32 mark: Powercycled sodium
• 15:18 logmsgbot: hashar synchronized docroot/noc/conf/wikiversions.dat 'publish wikiversions.dat on noc.wm.org'
• 04:43 Aaron|home: Copying missing timeline files from swift to nfs on hume
• 04:36 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline/Timeline.php 'Switched to doOperations().'
• 03:24 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline 'deployed 6e7076f35b38c16579342c69e4f19f8df4ed0ab6'
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 14 02:22:36 UTC 2012

September 13

• 23:04 logmsgbot: dzahn gracefulled all apaches
• 23:04 mutante: apache-graceful-all, pushing out redirect for wikimedia.biz
• 21:02 mutante: fixing UID for spage on stat1 (/etc/passwd and files), was used twice
• 20:34 logmsgbot: reedy synchronized php-1.20wmf11/extensions/Collection/
• 18:47 RobH: authdns-update for wikimedia.biz
• 17:05 mutante: flushing mobile varnish cache
• 17:03 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable wlm banner'
• 17:02 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/stylesheets/banner.css 'Touching banner.css'
• 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 13 02:20:30 UTC 2012
• 01:51 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
• 01:46 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
• 01:44 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/stylesheets/banner.css 'Update timestamp on banner.css'
• 01:41 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
• 01:32 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
• 01:01 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 're-disable wlm banner on enwiki'
• 00:58 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/
• 00:55 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Easier toggle of MF banner'
• 00:54 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Easier toggle of MF banner'
• 00:34 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Disabling wlm banner for mobile frontend'
• 00:18 logmsgbot: catrope synchronized php-1.20wmf11/extensions/Vector 'c9c0c3098c5bc6c15362332a1ed18e8545b1ab07'
• 00:17 logmsgbot: catrope synchronized php-1.20wmf11/skins/common/shared.css 'f9242f450ae16cf4f0a99e6fa78248e13ccb4ee0'
• 00:12 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline/Timeline.php 'debug logging'
• 00:07 logmsgbot: maxsem Finished syncing Wikimedia installation... : WLM app banner - MF, WikimediaMessages and config

September 12

• 23:52 logmsgbot: reedy synchronized php-1.20wmf11/extensions/MobileFrontend/
• 23:38 logmsgbot: maxsem Started syncing Wikimedia installation... : WLM app banner - MF, WikimediaMessages and config
• 23:09 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'migrate favicons from upload to bits'
• 23:08 logmsgbot: hashar synchronized docroot/strategyapps/favicon.ico
• 23:07 logmsgbot: hashar synchronized docroot/strategy/favicon.ico
• 23:06 logmsgbot: hashar synchronized docroot/steward/favicon.ico
• 23:06 logmsgbot: hashar synchronized docroot/outreach/favicon.ico
• 23:06 logmsgbot: hashar synchronized docroot/checkuser/favicon.ico
• 23:06 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline 'deployed f79a67ac344801acc57edbc95158bf5ecc87481b'
• 23:03 logmsgbot: hashar synchronized docroot/bits/favicon
• 23:02 logmsgbot: hashar synchronized docroot/bits/favicon
• 22:24 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled auto-resyncing for file backends.'
• 22:12 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disabled directory operations on local-temp zones.'
• 22:12 logmsgbot: reedy synchronized closed.dblist
• 21:48 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/FileBackendMultiWrite.php 'deployed 6a6a0427160e1493617789ce320020b21d0836dd'
• 21:48 logmsgbot: aaron synchronized php-1.20wmf11/languages/messages/MessagesEn.php 'deployed 6a6a0427160e1493617789ce320020b21d0836dd'
• 21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Subpagesortkey on ptwikibooks and enwikibooks'
• 21:13 RoanKattouw_: On ms7: find /export/upload/math ! -user apache -exec /root/fixownership2 \{\} \;
• 21:10 logmsgbot: reedy synchronized php-1.20wmf11/includes/api/ApiQueryLogEvents.php
• 20:51 logmsgbot: reedy synchronized php-1.20wmf11/extensions/TimedMediaHandler/
• 20:50 logmsgbot: reedy synchronized php-1.20wmf11/extensions/MwEmbedSupport/
• 19:05 notpeter: reimaging mw1-16
• 19:00 hashar: jenkins: hashar installed a snapshot of Gerrit Trigger plugin
• 18:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else over to 1.20wmf11
• 18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf11
• 18:09 notpeter: adding mw59 to apaches pool
• 18:03 logmsgbot: reedy synchronized php-1.20wmf10
• 17:58 logmsgbot: reedy synchronized php-1.20wmf11
• 17:53 notpeter: re-adding srv194-199 to apaches pool
• 17:38 notpeter: removing mw1-16 from apaches pool to reimage them as precise jobrunner boxes
• 14:46 paravoid: restarted swift on ms-fe1-4, proxy-server was leaking memory and had been swapping
• 02:52 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Sep 12 02:52:12 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 12 02:27:16 UTC 2012

September 11

• 22:29 notpeter: removing srv195-199 from apaches pool for upgarde to precise
• 20:51 RobH: authdns-update per rt 1326
• 18:31 apergos: second reboot had whines of device not present/not ready for /dev/sdn1 and /dev/sdh1 (note the list is not the same as the previous list), waited a little and it booted up with all disks mounted
• 18:26 apergos: first reboot of msbe10 showed complaints of not ready on devices sdi, k, n but they did eventually mount when the box came up fully. rebooting a second time to see what happens
• 18:13 apergos: rebooting ms-be10 after recreation and remount of swift filesystems (except those on ssds), see if they stick
• 17:55 apergos: new degradedarray event on ms-be6, booooo
• 17:54 apergos: rebooting ms-bs10 to see if we can get some reasonable device names, in prep for trying the same trick on it as on ms-be6 and 7
• 17:43 Reedy: Running sync-common on mw8
• 17:42 Reedy: mw8 seems to have out of date mw files
• 17:42 apergos: rebooting ms-be7 after recreating all filesystems (except those on the ssds) and remounting them, to see if the disks stay visible
• 17:38 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ConfirmEdit/FancyCaptcha.class.php
• 17:31 apergos: puppet disabled on ms-be7 via /etc/default/puppet, swift processes shut down while we test
• 17:10 apergos: one more reboot of ms-be6 after raid repaired, see if it and all mounts stay ok
• 15:57 apergos: rebuilding the degraded raid array on ms-be6
• 15:15 apergos: stopping swift processes on ms-be6 til we decide what's next; all disks came up and were mounted (but we still do have degraded raid array, not so awesome)
• 15:12 apergos: redid other ms-be6 filesystems so labels would be correct with disk layout without ssds. reboot again to see if it comes up properly
• 14:44 apergos: manually recreated filesystems and mounted missing drives on ms-be6 (without ssds cabled), seems to have worked without errors. rebooting to see what happens.
• 14:18 apergos: stopped swift processes on ms-be6 and disabled puppet (/etc/default/puppet) so it won't restart them. still doing testing of hw over there so we need the box up.
• 04:46 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ConfirmEdit/ '9643ccd3964827651e8a23a19a92ba3d240e60ee'
• 02:42 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Sep 11 02:41:59 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 11 02:22:02 UTC 2012
• 01:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf10
• 01:02 paravoid: adjusting spence's firewall to include all eqiad subnets

September 10

• 22:53 Ryan_Lane: adding all current labsconsole users to mediawiki shell group
• 22:47 logmsgbot: maxsem synchronized php-1.20wmf11/extensions/MobileFrontend/ 'MF update'
• 22:46 logmsgbot: maxsem synchronized php-1.20wmf10/extensions/MobileFrontend/ 'MF update'
• 22:40 paravoid: swift: weight 66->100 for ms-be9, 11, 12
• 21:40 Ryan_Lane: added salt and dependent packages to lucid-wikimedia and precise-wikimedia repos
• 21:39 logmsgbot: maxsem synchronized php-1.20wmf11/extensions/MobileFrontend/api/ApiMobileView.php 'Bug 39489'
• 19:42 mutante: push change for glwiki logo to cluster (gerrit 23175)
• 19:42 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
• 18:32 RobH: authdns-update for wicipediacymraeg.org
• 18:01 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
• 17:08 Jeff_Green: adjusted locke:/etc/udp2log/squid to fix pattern match for fundraiser banners
• 02:44 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Sep 10 02:44:38 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 10 02:26:50 UTC 2012

September 9

• 17:41 logmsgbot: aaron synchronized php-1.20wmf11/includes/upload/UploadFromChunks.php
• 17:40 logmsgbot: aaron synchronized php-1.20wmf11/includes/upload/UploadFromChunks.php 'deployed 9824650d70a0b2362e35eff3a5ba632d923404f6'
• 17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filerepo/FileRepo.php 'deployed 9824650d70a0b2362e35eff3a5ba632d923404f6'
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Sep 9 02:47:05 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 9 02:26:48 UTC 2012
• 01:13 Jeff_Green: restarted udp2log on locke

September 8

• 07:54 hashar: restarted Jenkins to fix up a plugin
• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Sep 8 02:45:17 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 8 02:27:32 UTC 2012

September 7

• 22:22 mutante: creating new gerrit project for the wikibugs IRC bot as a .deb (operations/debs/wikibugs)
• 21:43 mutante: for more reference, mw8 is down for DIMM errors (RT-3499)
• 21:42 RoanKattouw: For reference, the dsh groups Rob removed mw8 from are mediawiki-installation and apaches
• 21:39 RobH: pulled mw8 out of dsh groups since its down
• 19:59 cmjohnson1: loading new OS on mw8 after mother board swap
• 16:02 logmsgbot: hashar synchronized wmf-config/ext-wmflabs.php 'gerrit 22172 - Deploying OnlineStatusBar on beta'
• 15:21 logmsgbot: py synchronized wmf-config/lucene.php
• 15:17 notpeter: moving pool2,3,4 and prefix search traffic to eqiad
• 15:14 logmsgbot: py synchronized wmf-config/lucene.php
• 15:07 notpeter: moving en search traffic from pmtpa to eqiad
• 13:26 logmsgbot: hashar: bug 38036 cp /mnt/upload6/wikipedia/favicon-si.ico /mnt/upload6/wiktionary/favicon-si.ico
• 11:07 mark: Sending Canadian upload traffic to upload-lb.eqiad (Varnish)
• 07:12 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'change cortado location back again'
• 06:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'change cortado location'
• 04:26 Tim: professor has stopped responding to ssh and is not reporting to ganglia, rebooting
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Sep 7 02:37:41 UTC 2012
• 02:19 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 7 02:19:43 UTC 2012
• 01:02 AaronSchulz: Started copying math/ files into swift
• 00:58 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
• 00:49 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Removed excess math global'
• 00:28 logmsgbot: aaron synchronized wmf-config/ 'Write math files to swift and NFS'
• 00:19 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed 5aa6ab06fd4e0e76bd6a64e468c9eacb0f46f680'
• 00:06 Ryan_Lane: remove cn from uniqueness check in LDAP. it's a global check and causes problems
• 00:04 RobH: authdns-update to support wicipediacymraeg.org

September 6

• 23:23 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '...and anonnotice too (5f7cdae1da72895bcdb485ee7983944099f5af15)'
• 23:18 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'wgForceUIMsgAsContentMsg for bewikimedia (c1ef8fc89889a9802845c5f73381285a75a79eb3)'
• 21:56 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:03 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 19:53 ottomata: stopping puppet on oxygen. (puppet upgraded udp-filter before I was ready, I have to make sure ip filtering still works before I can turn it back on)
• 19:38 logmsgbot: reedy synchronized php-1.20wmf11/includes/media/FormatMetadata.php
• 18:40 RobH: authdns-update for ersch ip and removing old decom servers from pdns templates
• 18:29 RobH: installing ersch as poolcounter server
• 18:17 notpeter: stopping search indexing on searchidx1001 and searchidx2 to sync to eqiad
• 17:49 RoanKattouw: Pushing master of operations/mediawiki-config.git into gerrit from fenari
• 17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php 'deployed 3b2518d0284267100a61445b5d811940c8891395'
• 17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/FileBackend.php 'deployed 3b2518d0284267100a61445b5d811940c8891395'
• 16:14 mutante: srv219 - apt-get auto-remove unneeded packages, running puppet
• 16:11 mutante: srv219 - apt-get -f install, removing cm-super, dist-upgrading,
• 16:05 mark: Sending upload traffic from Argentina to upload-lb.eqiad (Varnish with streaming and persistence patches)
• 14:55 cmjohnson1: ms-be6 going down to remove ssd drives
• 14:23 notpeter: reimaging search1001, 1002, 1003
• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Sep 6 02:45:50 UTC 2012
• 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 6 02:28:18 UTC 2012
• 01:39 mutante: renaming mailing list chaptercommittee-l to affcom , rebuilding archives...
• 01:27 Ryan_Lane: enabling $wgEmailConfirmToEdit on labsconsole
• 01:21 Ryan_Lane: enabling ConfirmEdit with FancyCaptcha on labsconsole
• 01:09 Ryan_Lane: upgrading OATHAuth to 795cef09cab6ecb0e9ded35df06b2877ccc22c1a on labsconsole
• 00:19 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable LQT on lqtlabs'
• 00:19 Ryan_Lane: added SpamBlacklist to labsconsole
• 00:12 Ryan_Lane: added scribunto to labsconsole
• 00:12 RoanKattouw: Added rev_sha1 to revision table on liquidthreads_labswikimedia
• 00:09 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'and wgCanonicalServer too'
• 00:07 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wgServer for liquidthreads_labswikimedia'

September 5

• 23:59 logmsgbot: catrope rebuilt wikiversions.cdb and synchronized wikiversions files:
• 23:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '$wgDisplayTitle=false; on bewikimedia (114ff229f4eb15c1dc88b6da238c7aeea987c025)'
• 23:21 RobH: done with local hacks on brewster, resuming puppet
• 23:19 logmsgbot: catrope synchronized php-1.20wmf11/extensions/VisualEditor 'Update VisualEditor (55b78f34db13c0abd43bcdcc8798a8c3425f14ba)'
• 23:06 RobH: stopping puppet on brewster for local hack testing on ms-be6
• 23:00 logmsgbot: reedy synchronized php-1.20wmf11/includes/api/ApiQuery.php
• 22:36 mutante: renaming mailing list "wsor" to "wmfresearch"
• 22:34 RobH: blog update successful, all plugins also updated.
• 22:31 RobH: blog software is outdated by a few versions, updating.
• 22:12 apergos: shutdown ms-be6, prep for puppet changes for reinstall w/o ssds
• 22:09 Ryan_Lane: pulling in change 22690 to labsconsole
• 20:45 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 20:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 20:15 logmsgbot: reedy synchronized php-1.20wmf11/includes/media/FormatMetadata.php
• 20:12 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed c48c6c189ce576186457bb916f42f56e9d35d6f6'
• 20:11 Jeff_Green: rebooting storage3
• 19:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf11
• 19:56 mutante: added new Nagios monitor to Watchmouse
• 19:53 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 19:43 mutante: deleting Ben as a Watchmouse contact
• 19:41 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak the site'
• 19:39 logmsgbot: catrope synchronized php-1.20wmf11/includes/ImagePage.php 'Fix for bug 40018 (ac25a9d1127848269b6c9766262c23f961b35c5b)'
• 19:39 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 19:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2 back to wmf11
• 19:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2 back to wmf10 for a few mins..
• 19:23 logmsgbot: krinkle synchronized docroot/default/index.html 'Ia7ca3eaf'
• 19:23 logmsgbot: krinkle synchronized docroot/default/wmf_logo_with_text.png 'Ia7ca3eaf'
• 19:00 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'revert all back to old cortado.jar'
• 18:59 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 18:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 18:48 logmsgbot: reedy synchronized wmf-config
• 18:40 Reedy: commonswiki to stay on 1.20wmf10 for the meantime
• 18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf10 for testing
• 18:32 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Go back to older cortado to check thumbnail isue'
• 18:27 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Use an updated version of cortado for testwiki and test2wiki'
• 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia to 1.20wmf11
• 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wiktionary to 1.20wmf11
• 18:19 logmsgbot: reedy synchronized php-1.20wmf11/includes/Timestamp.php '0000...'
• 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikiversity and wikimedia to 1.20wmf11
• 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Misc wikis to 1.20wmf11
• 17:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Enable CORS for the API (347f386e2144688bcd4c80d24bd9f3be7d135d74)'
• 17:51 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable CORS for the API (347f386e2144688bcd4c80d24bd9f3be7d135d74)'
• 17:28 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
• 17:28 logmsgbot: aaron synchronized php-1.20wmf11/includes/filerepo/file/File.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
• 17:27 logmsgbot: aaron synchronized php-1.20wmf11/includes/StreamFile.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
• 15:54 Jeff_Green: updated MOTD for OTRS.
• 15:01 cmjohnson1: search32 still borked ..going down for troubleshooting
• 14:48 logmsgbot: reedy synchronized php-1.20wmf11/extensions/CheckUser 'CheckUser to master'
• 04:10 apergos: running d/d8 container consistency check fr ms7 from bast1001 in screen session as ariel
• 02:51 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 5 02:51:43 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Sep 5 02:26:57 UTC 2012
• 02:17 RobH: nagios crashed, fixing
• 01:05 Ryan_Lane: enabling DynamicSidebar extensions on labsconsole

September 4

• 23:34 AaronSchulz: Started copying timeline/ files into swift
• 23:34 logmsgbot: reedy synchronized wmf-config/
• 23:30 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Write timeline files to swift and NFS'
• 22:45 logmsgbot: catrope synchronized php-1.20wmf11/extensions/VisualEditor/ 'Update VisualEditor (59101ea3c93021c22da9796bf1715603967693fe)'
• 22:43 apergos: running heads on commons d/d7 on bast1001 as ariel from screen, data consistency check
• 22:07 notpeter: adding srv266 and srv281 back to apaches pool (to see if they break...)
• 21:04 logmsgbot: py synchronized wmf-config/lucene.php
• 21:01 notpeter: moving en search traffic back to pmtpa
• 20:46 logmsgbot: py synchronized wmf-config/lucene.php
• 20:45 notpeter: moving en search back to eqiad
• 20:41 Ryan_Lane: added oliver mail alias for okeyes
• 20:37 cmjohnson1: srv291 shutting down for troubleshooting
• 19:09 apergos: swift consistency check (heads of various objects) running from bastion1001 as ariel in screen session, about 10 HEADS/sec, if this starts to impact swift just shoot it
• 18:38 cmjohnson1: srv281 going down for troubleshooting
• 17:54 RobH: helium being reinstalled to poolcounter server per RT#3407
• 17:41 Ryan_Lane: pulled latest update of branch 1.20wmf10 on labsconsole
• 17:22 cmjohnson1: search32 shutting down to replace processor
• 16:26 notpeter: temp stopping puppet on brewster
• 16:26 notpeter: reimaging searchidx1001
• 15:26 logmsgbot: reedy synchronized wmf-config/ 'Staging for subpagesortkey'
• 15:24 logmsgbot: reedy synchronized wmf-config/
• 15:22 logmsgbot: reedy synchronized php-1.20wmf11/extensions/SubpageSortkey/
• 14:29 logmsgbot: reedy synchronized php-1.20wmf11/resources/startup.js 'touch'
• 14:28 logmsgbot: reedy synchronized php-1.20wmf10/resources/startup.js 'touch'
• 14:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 14:23 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery/jquery.js 'Revert back to 1.7.2 for wmf10 wikis'
• 14:19 logmsgbot: reedy synchronized php-1.20wmf11/resources/jquery/jquery.js 'Revert back to 1.7.2 for wmf11 wikis'
• 12:55 mark: Pushed varnish 3.0.3plus-rc1 packages into the precise-wikimedia APT repository
• 02:51 paravoid: powering down ms-be6, broken hardware
• 02:39 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 4 02:39:34 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Sep 4 02:24:01 UTC 2012
• 01:28 logmsgbot: tstarling synchronized wmf-config/flaggedrevs.php

September 3

• 18:34 logmsgbot: reedy synchronized live-1.5/
• 18:26 Reedy: Deleting php-1.20wmf8 from mediawiki-installation
• 18:25 Reedy: Deleting 1.20wmf9 l10n cache from mediawiki-installation
• 18:22 Reedy: Repointed php symlink from wmf9 to wmf10
• 18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf11
• 17:48 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild 1.20wmf11 localisation cache
• 17:42 paravoid: powering down mw8; broken DIMM (#3499), has been flapping for days
• 17:32 paravoid: killing power via ipmi on ms-be7; broken hardware, stale data, should not be on rotation
• 17:18 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild 1.20wmf11 localisation cache
• 17:16 paravoid: swift: distributing new ring; disable ms-be7, ms-be8, ms-be11 sdi, increase ms-be9/11/12 weight from 31 to 66
• 17:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf11 and rebuild localisation cache
• 15:59 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf11 and rebuild localisation cache
• 15:44 logmsgbot: reedy synchronized php-1.20wmf11/cache/
• 15:44 logmsgbot: reedy synchronized wmf-config/
• 15:42 Reedy: sync-docroot
• 15:41 logmsgbot: reedy synchronized php-1.20wmf11 'Initial sync-out'
• 15:09 notpeter: reinstalling all eqiad search frontends as precise
• 13:55 hashar: Sent one live hack made to 1.20wmf10 : https://gerrit.wikimedia.org/r/#/c/22461/
• 13:52 hashar: Rebased local 1.20wmf10 directory on top of its current origin/1.20wmf10 ( 472ec09 )
• 13:50 hashar: installed Gerrit commitmsg hook in /h/w/c/php-1.20wmf10/.git/hooks
• 13:36 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery/jquery.js 'jQuery 1.8.1'
• 09:43 hashar: manually cleaning out /tmp on gallium. It is being filled by faulty tests not cleaning the files they are creating.
• 02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Sep 3 02:18:28 UTC 2012

September 2

• 21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Sep 2 02:25:51 UTC 2012

September 1

• 02:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Deploy 22ceabed668b96cd88b48f6f7482c86a53645fb2'
• 02:49 logmsgbot: catrope synchronized wmf-config/filebackend.php 'Deploy 988ea2f71d84e1f62269f6e3d72eaf7b03206906'
• 02:49 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Deploy 988ea2f71d84e1f62269f6e3d72eaf7b03206906'
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Sep 1 02:25:35 UTC 2012

August 31

• 22:04 Ryan_Lane: removing attribute uid from the attribute uniqueness plugin. It's the naming attribute and it's already ensured to be unique. It causes problems when it's also in the plugin.
• 21:54 Ryan_Lane: enabling attribute uniqueness plugin on virt0 and virt1000 for attributes: cn, uid, uidNumber
• 21:51 mutante: running rm -rf /var/lib/testswarm on gallium in a screen ..takes forever
• 21:51 Ryan_Lane: adding equality index to uidNumber on virt0 and virt1000 opendj
• 21:35 mutante: ms-be6 - just for the record, still could not mount same disks (sdn1, sdl1, sdi1, sdf1) after firmware upgrade
• 20:32 cmjohnson1: ms-be6 down for firmware upgrade
• 20:28 mutante: deleting /var/lib/testswarm on gallium (testswarm is gone)
• 12:11 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '5c84fca $wgDBerrorLogInUTC -> $wgDBerrorLogTZ'
• 12:11 logmsgbot: hashar synchronized live-1.5/robots.php 'f0bf7a5 - disallow robots on beta'
• 12:11 hashar: forgot to pull before sync :-)
• 12:11 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '5c84fca $wgDBerrorLogInUTC -> $wgDBerrorLogTZ'
• 12:10 logmsgbot: hashar synchronized live-1.5/robots.php 'f0bf7a5 - disallow robots on beta'
• 04:02 logmsgbot: tstarling synchronized php-1.20wmf10/tests/parser/parserTests.txt
• 04:01 logmsgbot: tstarling synchronized php-1.20wmf10/includes/specials/SpecialBlock.php
• 04:01 logmsgbot: tstarling synchronized php-1.20wmf10/includes/api/ApiFormatBase.php
• 04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/Linker.php
• 04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/User.php
• 04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/AuthPlugin.php
• 03:59 logmsgbot: tstarling synchronized php-1.20wmf10/includes/DefaultSettings.php
• 02:59 paravoid: swift: removing ms-be8 sdc1, sdg1, sdk1, sdl1, sdn1 from ring; broken hardware (again)
• 02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Aug 31 02:18:02 UTC 2012

August 30

• 22:54 logmsgbot: maxsem synchronized php-1.20wmf10/includes/api/ApiMain.php 'WLM API logging https://gerrit.wikimedia.org/r/#/c/21748/'
• 22:27 Ryan_Lane: ran a maintenance script on labsconsole to update all instance pages
• 22:05 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:45 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:17 notpeter: removing srv194 from apache pool
• 21:13 logmsgbot: olivneh synchronized php-1.20wmf10/extensions/E3Experiments/experiments
• 20:39 cmjohnson1: srv291 shutting down to reseat DIMM
• 20:39 cmjohnson1: srv291 shutting down to reseat DIMM
• 20:28 logmsgbot: krinkle synchronized live-1.5/static-1.20wmf10 'Creating static aliases for wmf10'
• 20:25 Krinkle: Created static-1.20wmf10 and extensions,resources,skins subdir symlinks in /home/wikipedia/commons/live-1.5, was not done by the script apparently
• 20:07 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery
• 20:02 mutante: reinstalling cp1024-cp1028
• 19:28 Ryan_Lane: finished upgrading labsconsole to 1.20wmf10
• 19:17 notpeter: all search traffic now pointed back to pmtpa
• 19:17 logmsgbot: py synchronized wmf-config/lucene.php
• 18:58 Ryan_Lane: upgrading labsconsole to 1.20wmf10
• 18:57 logmsgbot: py synchronized wmf-config/lucene.php
• 18:41 paravoid: deploying squid config all for thumb_handler.php addition to thumb_php
• 18:31 logmsgbot: preilly synchronized wmf-config/lucene.php
• 17:47 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/SwiftFileBackend.php 'deployed faec65ea23a8d17f08488950fccb0911f7b63086'
• 17:33 apergos: and rebooting ms-be7 again in the slim hope that we get back the missing drives... meh
• 17:32 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/SwiftFileBackend.php 'deployed 3bc61905b704096789376927b6cc3d7ca78b0670'
• 17:32 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FSFileBackend.php 'deployed 3bc61905b704096789376927b6cc3d7ca78b0670'
• 16:45 notpeter: removing srv281 from apaches pool because it's a broken piece of garbage.
• 16:23 apergos: power cycle ms-be7 so it picks up the replaced drive with the right device name
• 15:46 cmjohnson1: mw8 shutting down for DIMM troubleshooting
• 14:09 RobH: authdns-update for helium/potassium (poolcounter servers)
• 10:53 hashar: gerrit-wm irc bot is apparently no more sending anything to #mediawiki :( IRC user has been inactive since midnight UTC. Opened bug 39797
• 03:35 Tim: restarting apaches to get php.ini change of pcre.backtrack_limit
• 02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Aug 30 02:18:56 UTC 2012
• 01:36 K4-713: updated the payments cluster to 95907e42e57
• 01:10 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed fix for bogus var'
• 00:48 mutante: starting nagios on spence

August 29

• 23:02 logmsgbot: catrope synchronized wmf-config/filebackend.php 'Deploy d5b73b216317c705f58d1a1eef8bc0db21a90314'
• 21:25 logmsgbot: krinkle synchronized php-1.20wmf10/includes/EditPage.php '(bug 39584) Deploy MediaWiki:Previewnote bug fix (I2144f026d3b301edb521bf)'
• 21:12 logmsgbot: krinkle synchronized php-1.20wmf10/resources/ 'Update jquery.ui and jquery.effects to jQuery UI 1.8.23 (I35ed086e9a8c6d)'
• 20:29 Ryan_Lane: changing apache config and wiki location on virt0
• 20:27 logmsgbot: krinkle synchronized php-1.20wmf10/resources/jquery/jquery.js '(8dc8ee84) jQuery 1.8'
• 18:41 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Move last remaining wiki to 1.20wmf10, so long 1.20wmf9
• 18:38 Ryan_Lane: deleting global role groups from LDAP. They aren't needed in keystone's LDAP DIT
• 18:00 mutante: reinstalling cp1022 and up
• 17:56 notpeter: returning srv281 to apaches pool
• 17:36 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Switching all 'pedias to 1.20wmf10
• 17:11 notpeter: removing srv281 once again and reimaging
• 17:09 notpeter: returning srv281 to apaches pool
• 17:01 cmjohnson1: swapping out disk7 (dev/sdg) on ms-be7
• 16:36 logmsgbot: root Finished syncing Wikimedia installation... :
• 16:28 notpeter: removing srv281 from apaches pool
• 16:20 logmsgbot: root Started syncing Wikimedia installation... :
• 16:11 logmsgbot: mark synchronized wmf-config/mc.php
• 16:10 notpeter: enabling srv194 and srv281 in pybal for apache/precise live test
• 16:04 mark: Fixed serial console on hume
• 16:00 mark: Powering up fenari
• 15:49 mark: Powering up hume
• 15:48 mark: Final rsync finished.
• 13:37 mark: Mounted /home read-only on nfs1, started final rsync to nas1-a
• 13:28 mark: Changed /home mount from nfs1 to nas1-a on srv193 and spence
• 13:05 mark: Shutdown fenari and hume
• 12:48 hashar: reloading Jenkins to fix up Ext-TitleBlacklist misconfiguration
• 12:42 hashar: enabling Jenkins on TitleBacklist extension (job: Ext-TitleBlacklist )
• 08:55 logmsgbot: nikerabbit synchronized php-1.20wmf9/includes/Linker.php
• 08:54 logmsgbot: nikerabbit synchronized php-1.20wmf10/includes/Linker.php
• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 29 02:45:36 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Aug 29 02:27:21 UTC 2012

August 28

• 23:45 LeslieCarr: there are some network issues with one of our ipv6 carriers, causing some european ipv6 users to have trouble reaching our US sites.
• 23:17 Ryan_Lane: fixed scripts for management home directories and gluster shares for labs by changing the project search filter to match keystone's LDAP DIT
• 21:44 logmsgbot: bsitu synchronized php-1.20wmf10/extensions/PageTriage/tools/cleanupPageTriagePageTags.php 'Fix the cleanup cron'
• 21:06 K4-713: updated the payments cluster to 9bcd8961fecb5
• 20:59 mutante: git pull and sync-file InitialiseSettings.php to cluster to push namespace fix on ar.wikimedia
• 20:59 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
• 20:52 logmsgbot: mlitn Finished syncing Wikimedia installation... : Updating ArticleFeedbackv5
• 20:16 logmsgbot: mlitn Started syncing Wikimedia installation... : Updating ArticleFeedbackv5
• 19:55 logmsgbot: catrope synchronized php-1.20wmf10/extensions/ProofreadPage 'Deploy 65fcb5984fd7d52ecd1887b248fda608b9dc4987'
• 19:53 logmsgbot: catrope synchronized php-1.20wmf10/tests/phpunit/includes/RevisionStorageTest.php 'Deploy e1cfe5ada99f3ed4594c5cfc33906538f4e02bdf'
• 18:52 mutante: rebooting cp1021 into PXE
• 17:41 paravoid: include debs for swift 1.5 and friends to apt (lucid/precise)
• 16:15 Jeff_Green: added payments100[1-4] & pay-lvs100[12] to nagios nsca config, restarted nagios on spence
• 14:03 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'ttmserver part 2'
• 14:02 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ttmserver part 1'
• 13:14 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Syncing 95c87a0b for AFT (mlitn said its ok)'
• 13:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Syncing f58063db'
• 13:11 logmsgbot: demon synchronized .gitignore 'Syncing .gitignore'
• 12:29 mark: Built varnish 3.0.3plus~rc1 packages and deployed them on cp1041 (mobile)
• 10:02 logmsgbot: nikerabbit synchronized php-1.20wmf10/extensions/Translate/ 'Picking up Solarium'
• 09:23 logmsgbot: nikerabbit synchronized php-1.20wmf10/extensions/wikihiero/ 'RTL fix'
• 04:01 Ryan_Lane: rebooting virt5
• 03:56 Ryan_Lane: rebooting virt4
• 03:41 Ryan_Lane: upgrading virt1 to precise
• 03:38 Ryan_Lane: upgrading virt4 to precise
• 03:37 Ryan_Lane: upgrading virt5 to precise
• 03:06 Ryan_Lane: bringing virt7 instances up
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 28 02:48:54 UTC 2012
• 02:45 Ryan_Lane: bringing virt8 instances up
• 02:37 logmsgbot: tstarling Finished syncing Wikimedia installation... :
• 02:30 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Aug 28 02:30:01 UTC 2012
• 02:24 Ryan_Lane: force running puppet on virt7
• 02:24 Ryan_Lane: force running puppet on virt8
• 02:17 Ryan_Lane: rebooting virt8
• 02:17 Ryan_Lane: rebooting virt 7
• 02:08 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 01:51 Ryan_Lane: upgrading virt8 to precise
• 01:48 logmsgbot: tstarling synchronized php-1.20wmf10/includes/GlobalFunctions.php
• 01:48 logmsgbot: tstarling synchronized php-1.20wmf10/thumb.php
• 01:47 logmsgbot: tstarling synchronized php-1.20wmf10/includes/upload/UploadFromUrl.php
• 01:46 Ryan_Lane: upgrading virt7 to precise
• 01:32 Ryan_Lane: bringing all instances up on virt6
• 01:01 Ryan_Lane: attempting to bring up some instances on virt6

August 27

• 23:31 Ryan_Lane: restarting virt6
• 23:08 Ryan_Lane: upgrading virt6 to precise. This action will cause downtime for a while.
• 22:49 mutante: fixed gmq.planet.wm updates
• 22:34 Ryan_Lane: rebooting virt3
• 22:24 Ryan_Lane: upgrading virt3 to precise
• 22:23 Ryan_Lane: disabling nova-compute on virt2
• 21:01 dschoon: ${whoami}
• 21:00 Ryan_Lane: enabling OpenStackManager and LdapAuthentication on labsconsole temporarily for testing
• 21:00 Ryan_Lane: enabling nova-compute on virt2 for testing instance creation
• 20:47 ottomata: starting puppet back up on emery
• 20:32 logmsgbot: aaron synchronized php-1.20wmf9/extensions/MobileFrontend 'deployed f2551fa97ba686101745d5cb97ef4678838cd3bb'
• 20:29 logmsgbot: aaron synchronized php-1.20wmf10/extensions/MobileFrontend 'deployed 75104cf6d3bc6d528d2b85e09761005081fb5cf4'
• 20:20 paravoid: deploying swift origs to all pmtpa backend squids
• 20:09 Ryan_Lane: force running puppet on virt2
• 20:04 Ryan_Lane: rebooting virt2
• 19:54 paravoid: deploying swift origs on sq51 (1h ago), sq52, sq53
• 19:54 Ryan_Lane: upgrading virt2 to precise
• 19:38 Ryan_Lane: upgrading OpenStackManager on labsconsole
• 19:33 Ryan_Lane: adding keystone services and endpoints
• 19:32 Ryan_Lane: upgrading nova, glance and keystone database schemas
• 19:18 ottomata: temporarily stopping puppet on emery to test log2udp AFT clicktracking relay to vanadium
• 19:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf10
• 18:41 Ryan_Lane: force running puppet on virt0
• 18:36 Ryan_Lane: restarting virt0
• 18:09 Ryan_Lane: upgrading virt0 to precise
• 17:54 Ryan_Lane: running a fresh backup of opendj on virt0
• 17:54 Ryan_Lane: backing up all databases on virt0
• 17:46 Ryan_Lane: stopping rabbitmq on virt0
• 17:43 Ryan_Lane: disabling openstack services on the virt infrastructure, that is
• 17:43 Ryan_Lane: disabling all openstack services
• 17:42 Ryan_Lane: disabling OpenStackManager and LdapAuthentication on labsconsole
• 17:37 Ryan_Lane: disabling puppet on all virt nodes
• 17:33 paravoid: rolled back sq51's config; depooled sq51 (10' ago)
• 17:18 paravoid: modified sq51 to serve origs from swift; added sq51 back to the pool
• 16:45 paravoid: depooling sq51 backend
• 15:55 paravoid: upgraded swift on ms-be12
• 15:54 paravoid: upgraded swift on ms-be5
• 15:47 paravoid: upgraded swift on ms-be11
• 14:51 paravoid: upgraded swift on ms-be3
• 14:40 paravoid: upgraded swift on ms-be4
• 14:30 paravoid: upgraded swift on ms-be10; (still depooled, no effect)
• 09:38 mark: Decommissioning bayes, shut it down
• 02:52 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 27 02:52:23 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Aug 27 02:27:16 UTC 2012
• 02:16 Tim: on spence: fixed /apache symlink, was pointing to a non-standard location, breaking sync scripts
• 01:32 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
• 01:32 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/Scribunto.i18n.php
• 01:31 Tim: updating Scribunto
• 01:31 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/Scribunto.namespaces.php

August 26

• 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 26 02:53:09 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Aug 26 02:27:02 UTC 2012

August 25

• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sat Aug 25 02:46:16 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Aug 25 02:27:08 UTC 2012

August 24

• 23:08 notpeter: temp stopping puppet on iron
• 23:02 maplebed: upgrading swift backend ms-be9
• 22:15 maplebed: upgrading swift backend ms-be7
• 21:54 maplebed: upgrading swift back end ms-be2
• 21:30 mutante: removing decom'ed srv206 and srv217 from ALL dsh groups
• 21:24 maplebed: upgrading swift backend host ms-be1
• 21:22 mutante: breaking test.wp :) ..and rolling back
• 21:11 maplebed: upgraded swift backend host ms-be8
• 20:10 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php (bug 34817)
• 19:57 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
• 19:01 RoanKattouw: (That was me)
• 19:01 logmsgbot: Restarted logmsgbot, was broken again
• 19:00 ottomata: starting puppet back up on brewster
• 18:58 mutante: sync-file ./wmf-config/InitialiseSettings.php
• 18:57 mutante: git pull in /h/w/common/wmf-config
• 18:32 ottomata: temporarily stopping puppet on brewster to test out partman change for analytics1023-1027
• 14:57 RobH: restart puppeted on brewster
• 14:48 RobH: analytics1023-1027 rotating down for removal of extra nic
• 14:28 RobH: stopping puppet on brewster to do a local nonpuppetized test change
• 13:08 paravoid: repooling ms-fe1/2 with all new swift
• 12:40 apergos: on ms5, running from screen session as root: tossing non-standard thumb sizes > 100 px for commons/x/xx to see what space that gives us
• 12:32 paravoid: depooling ms-fe1/2 for the 1.5 upgrade
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 24 02:47:46 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Aug 24 02:26:10 UTC 2012
• 02:08 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 01:55 logmsgbot: kaldari Started syncing Wikimedia installation... :

August 23

• 23:56 logmsgbot: kaldari synchronized php-1.20wmf10/extensions/UploadWizard
• 23:49 K4-713: updated paypal IPN to r2028
• 22:40 maplebed: putting ms-fe3 and 4 into rotation running swift v1.5.0-3
• 22:40 K4-713: updated production civicrm to r2024
• 22:06 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:53 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:33 logmsgbot: aaron synchronized php-1.20wmf10/extensions/SwiftCloudFiles 'deployed de4a83af7750750fcaa49e81b7faa771c8601546'
• 21:32 logmsgbot: aaron synchronized php-1.20wmf9/extensions/SwiftCloudFiles 'deployed de4a83af7750750fcaa49e81b7faa771c8601546'
• 20:42 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/experiments/trackEdits.js
• 20:40 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/lib/e3.js
• 20:35 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 20:12 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 18:44 maplebed: rebooting ms-be3
• 18:34 apergos: deleting about 200k thumb dirs and their contents from ms5, unused on any project, covering june and july uploads
• 18:06 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disable auth caching for testwiki'
• 17:40 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FSFileBackend.php 'debug logging.'
• 17:21 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Make testwiki use ms-fe4'
• 17:09 logmsgbot: aaron synchronized wmf-config/filebackend.php
• 16:54 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Only write thumbnails to the master backend.'
• 16:41 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FileBackendMultiWrite.php 'deployed 36c0c1edb6e713a797e7fde52f716ca17f548866'
• 16:40 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed 36c0c1edb6e713a797e7fde52f716ca17f548866'
• 11:46 paravoid: depooling ms-fe3 to stage 1.5 upgrade
• 11:27 paravoid: depooling ms-fe4 to stage 1.5 upgrade
• 04:21 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/MediaTransformOutput.php 'deployed ed9d124c37f57ab7d7c9678908af54363290b1f4'
• 04:20 logmsgbot: aaron synchronized php-1.20wmf10/includes/media/MediaTransformOutput.php 'deployed ed9d124c37f57ab7d7c9678908af54363290b1f4'
• 04:12 logmsgbot: aaron synchronized php-1.20wmf10/includes/media/MediaTransformOutput.php
• 03:49 logmsgbot: aaron synchronized php-1.20wmf10/includes/StreamFile.php
• 03:38 logmsgbot: aaron synchronized php-1.20wmf10/includes/StreamFile.php 'debug logging'
• 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 23 02:53:10 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Aug 23 02:27:31 UTC 2012
• 00:04 logmsgbot: asher synchronized wmf-config/db.php 'returning db35 to s5'

August 22

• 23:54 binasher: rebooting db35 for kernel upgrade
• 23:50 binasher: new s5 master repl pos - MASTER_LOG_FILE='db45-bin.000187', MASTER_LOG_POS=807116168
• 23:49 logmsgbot: asher synchronized wmf-config/db.php 'setting s5 to writeable'
• 23:47 logmsgbot: asher synchronized wmf-config/db.php 'setting s5 to read-only and switching master to db45, return db39 to s3'
• 23:47 binasher: preparing to rotate the s5 master to db45 (last of these for now)
• 23:36 mutante: check_job_queue check fixed after Tim synced spence. zhwiki has large queue: zhwiki (118614). Nagios did not talk because it was a transition from one critical to another
• 23:26 Tim: added spence to the mediawiki-installation dsh group
• 23:25 binasher: rebooting db39 for kernel upgrade
• 23:19 binasher: new s3 master repl pos: MASTER_LOG_FILE='db34-bin.000864', MASTER_LOG_POS=466194762
• 23:18 logmsgbot: asher synchronized wmf-config/db.php 'setting s3 to writeable'
• 23:16 logmsgbot: asher synchronized wmf-config/db.php 'setting s3 to read-only, switching master to db34'
• 23:15 binasher: preparing to rotate the s3 master to db34
• 23:14 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Restored math directory to its intended value.'
• 23:13 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Restored math directory to its intended value.'
• 23:06 K4-713: updated production civicrm to r2013
• 22:42 mutante: rebooting srv208,srv238
• 22:35 mutante: re-enabling puppet on srv208 and srv238, dist-upgrading while at it
• 22:26 K4-713: updated production civicrm to r2011
• 22:10 mutante: srv208,srv238 - puppet running but it's not. looks like puppet is disabled using puppet agent --disable, on purpose?
• 21:39 logmsgbot: catrope synchronized php-1.20wmf10/resources 'jQuery revert'
• 21:03 mutante: adding Bugzilla https monitor to watchmouse
• 20:53 mutante: dist-upgrading kaulen, new kernel (not rebooted yet), fixed SSL cert path in puppet, restart puppet
• 20:37 mutante: stopped puppet on kaulen. manually fixed path to SSL cert so Apache would start. Bugzilla back for now
• 20:32 mutante: powercycling kaulen. OOM death.
• 20:29 mutante: kaulen unresponsive, bugzilla affected
• 20:26 logmsgbot: aaron synchronized php-1.20wmf9/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed 8fea21b10eaba28773c30c6952fc792f08bf09a8'
• 20:24 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Re-enabled swift token caching.'
• 20:20 logmsgbot: aaron synchronized php-1.20wmf10/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed 8fea21b10eaba28773c30c6952fc792f08bf09a8'
• 19:17 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php
• 19:13 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'bit logic is fun'
• 19:11 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'debug logging'
• 19:08 mutante: reinstalling zirconium to use LVM
• 19:05 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'debug logging'
• 18:55 logmsgbot: Restarted ircecho on fenari, logmsgbot was broken --Roan
• 18:52 RoanKattouw: 1.20wmf10 deployed to all non-Wikipedia projects around 20:04 UTC
• 17:44 mutante: installing package upgrades on bast1001
• 17:19 paravoid: stopped supervisord & puppet on silver/zhen per preilly's request
• 17:00 apergos: oh, for people following along, shot the delete job. apparently even deletes is enough to push ms5 over the edge, yay
• 16:50 apergos: removing thumbs on ms5 unused on any project, for june-july uploads. running in screen as root
• 16:37 ^demon: bugzilla went down, but came back up. magic?
• 16:35 Jeff_Green: restarted ns1 b/c it fell down after authdns-update
• 16:33 Jeff_Green: adding dns for pay-lvs100[12]
• 16:32 paravoid: deploy new squid config that removes ms5 from a swift fallback for thumbs
• 16:06 cmjohnson1: ms-be6 powering down for memory testing
• 15:32 notpeter: rebooting srv193, upgrade to precise
• 11:38 mark: Removed /homewmf NFS mount on kaulen
• 06:14 binasher: new s1 repl position: MASTER_LOG_FILE='db63-bin.000049', MASTER_LOG_POS=646903534
• 06:14 logmsgbot: asher synchronized wmf-config/db.php 'returing db38 as an s1 slave'
• 06:02 binasher: shutting down mysql on db38 and rebooting for kernel upgrade
• 05:50 logmsgbot: asher synchronized wmf-config/db.php 'setting s1 writeable on new master'
• 05:49 logmsgbot: asher synchronized wmf-config/db.php 'setting s1 to read-only, switching master to db63'
• 05:47 binasher: preparing to switch s1 master to db63 (upgrade to 96GB ram and precise) in a minute
• 05:29 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'enabling Scribunto on mediawiki.org'
• 05:07 Tim: on ms7, running a script to remerge the split math directories back into /mnt/upload6/math, using rsync
• 02:55 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Aug 22 02:55:22 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 22 02:27:55 UTC 2012
• 00:19 Tim: running updateCollation.php for ptwiki in a screen on hume
• 00:15 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'setting $wgCategoryCollation=uca-default on ptwiki'
• 00:00 binasher: completed user.user_email index migrations

August 21

• 23:46 maplebed: deployed change to swift proxy rewrite to only send thumb 404s to the image scalers gerrit change 20916
• 23:44 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled token/url caching for swift.'
• 23:26 Tim: updated email address for enwiki user account "Giano"
• 23:17 binasher: running user.user_email index migration on s6-7
• 23:16 binasher: running user.user_email index migration on s5
• 23:14 binasher: truncated enwiki.flagged(images|templates)
• 23:10 binasher: running user.user_email index migration on s4
• 22:44 logmsgbot: asher synchronized wmf-config/db.php 'returning db52 as an s2 slave'
• 22:24 binasher: s2 master swap - new repl position MASTER_LOG_FILE='db54-bin.000302', MASTER_LOG_POS=870283287
• 22:23 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 to writeable'
• 22:22 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 to read-only for master swap to db54'
• 22:09 logmsgbot: reedy synchronized php-1.20wmf10/extensions/EducationProgram
• 21:59 binasher: running purgeParserCache.php for all dbs
• 21:26 binasher: resumed s3 user.user_email migrations
• 20:34 Jeff_Green: hand-tweezed spence:/etc/nagios/nsca_payments.cfg b/c puppet is broken for spence, restarted nagios on spence
• 20:15 Jeff_Green: added hosts boron/indium to nagios's nsca config
• 19:39 logmsgbot: reedy synchronized php-1.20wmf10/resources/ 'jQuery UI 1.8.23'
• 19:26 paravoid: rolling back sq51 switch to originals
• 18:52 paravoid: putting sq51 (modified with originals pointing to swift) back into the backend pool
• 18:33 RobH: authdns-update for mc1009-1016 mgmt
• 18:16 RobH: db1047 back online, presends sdb and sdc at 3597.3 GB each
• 18:11 RobH: db1047 disk shelf added, two raid10 arrays of 12 disks each, booting now into OS.
• 18:00 RobH: db1047 shutting down
• 17:58 RobH: db1047 mysql shutting down (going to add new disk shelf to server)
• 16:44 AaronSchulz: Running setZoneAccess.php for all wikis
• 16:40 apergos: shot morebots so the restarter script would restart it
• 05:41 binasher: halting s3 user migration on hiwiktionary, will resume in the morning (pst)
• 04:02 binasher: adding user.user_email index on s3
• 03:44 binasher: adding user.user_email index on s2
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Aug 21 02:37:10 UTC 2012
• 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 21 02:21:32 UTC 2012
• 01:47 binasher: adding user.user_email index to enwiki via osc
• 00:32 maplebed: removed ms-be6 from rotation due to bad memory and repeated kernel panics
• 00:23 mutante: fenari - install a couple lib* package upgrades
• 00:10 logmsgbot: preilly synchronized php-1.20wmf10/extensions/MobileFrontend 'weekly deploy'
• 00:09 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'weekly deploy'

August 20

• 23:48 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 23:46 logmsgbot: tstarling synchronized php-1.20wmf9/maintenance/benchmarks/obama.php
• 23:18 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 22:50 logmsgbot: reedy synchronized php-1.20wmf10/extensions/VisualEditor/
• 22:49 logmsgbot: reedy synchronized php-1.20wmf10/includes/GlobalFunctions.php
• 22:15 logmsgbot: reedy synchronized php-1.20wmf10/skins/
• 21:33 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling file uploads on testwiki for regular user accounts (exception to bug 12556)'
• 20:26 Jeff_Green: fixing typo in pdns-templates/wmnet and reloading nameservers
• 18:57 logmsgbot: catrope synchronized php-1.20wmf10/extensions/VisualEditor/ 'Actually push out VE fixes'
• 18:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Update VE config for Parsoid prefix change'
• 18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki and mediawikiwiki to 1.20wmf10
• 18:22 paravoid: cleaning sq51's cache, poisoned after swift changes staging
• 16:06 logmsgbot: reedy synchronized php-1.20wmf10/cache/l10n/
• 16:04 mark: Upgrading JUNOS on asw2-a5-eqiad to 11.4R2.14
• 15:49 mark: Upgrading JUNOS on asw2-d3-sdtpa to 11.4R2.14
• 15:47 logmsgbot: reedy Started syncing Wikimedia installation... :
• 15:38 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.20wmf10 and rebuild localisation cache for it..
• 15:01 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.20wmf10 and rebuild localisation cache for it..
• 14:46 logmsgbot: reedy synchronized php-1.20wmf10/
• 14:39 logmsgbot: reedy synchronized wmf-config/
• 14:38 logmsgbot: reedy synchronized php-1.20wmf10/
• 14:36 logmsgbot: reedy synchronized live-1.5/
• 14:30 Reedy: Removing php-1.20wmf7 from mediawiki-installation group
• 14:26 logmsgbot: reedy synchronized php-1.20wmf10/ 'staging 1.20wmf10 directory'
• 09:49 paravoid: powering off mw8, faulty (#3425), has been flapping a lot
• 09:24 paravoid: restarted gmond in cp3002
• 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 20 02:28:20 UTC 2012

August 19

• 20:54 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 19 02:24:13 UTC 2012

August 18

• 03:46 Tim: copying old logs from bast1001 to fluorine
• 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sat Aug 18 02:28:03 UTC 2012

August 17

• 23:17 Ryan_Lane: added cn=essextest project to labs LDAP for essex testing. this project isn't compatible with diablo
• 22:16 notpeter: rebooting/reimaging zhen and silver for upgrade to precise
• 22:01 mutante: rebuilding wikitech wiki l10n cache
• 22:00 mutante: rebuilding wikitech-l archives
• 20:15 Reedy: Pointing php to php-1.20wmf9
• 20:13 Reedy: Deleting php-1.20wmf8/cache/l10n from mediawiki-installation
• 17:48 RobH: once all apaches update, will migrate old logs from nfs to fluorine
• 17:48 RobH: apache logs confirmed hitting fluorine, all will transition over as they puppet update
• 17:24 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
• 17:24 logmsgbot: robh synchronized wmf-config/CommonSettings.php
• 17:23 RobH: moving apache logs back to fluorine
• 17:00 cmjohnson1: shutting down search23 to repair DIMM
• 16:59 cmjohnson1: mc1 shutting down to repair DIMM
• 16:31 mark: Enabled all 3 new 802.3ad aggregated links for asw-d1-sdtpa to asw-d3-sdtpa
• 16:15 RobH: authdns-update to correct typo, nothing really broke badly, i r lucky
• 15:54 RobH: fluorine now coming down for reinstall, all log files backed up to nfs
• 15:47 mark: Created static 802.3ad trunks on csw1-sdtpa for asw-d1-sdtpa to asw-d3-sdtpa uplinks, deployed with only the existing port enabled
• 15:44 mark: Created static 802.3ad trunks on asw-d1-sdtpa and asw-d2-sdtpa for uplinks
• 15:31 RobH: puppet is taking forever on nagios, as soon as it completes, will remove fluorine from decom and rerun
• 15:11 RobH: authdns-update for fluorine ip update
• 15:04 RobH: lvm restructure
• 15:04 RobH: all data copied, all traffic moved, fluorine coming down for reinstallation into internal vlan and lvs restructure
• 15:03 RobH: all log traffic appears to have moved back to nfs1, copying fluorine logs back to nfs for reformat
• 14:53 mark: Setup static 802.3ad trunk on asw-d3-sdtpa with one port up
• 14:40 cmjohnson1: swapping disk3 db44
• 14:27 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
• 14:27 logmsgbot: robh synchronized wmf-config/CommonSettings.php
• 13:48 logmsgbot: robh ran sync-common-all
• 13:34 RobH: moving apache logs from pushing to fluorine back to nfs1 for now
• 05:12 Tim: queued refreshlinks jobs for 1330 commonswiki image description pages with coord templates which weren't refreshed in the previous manual attempt, probably due to failed jobs
• 03:34 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/DjVu.php 'deployed 3ae632c3c9111b16f019574bcafec58fa3ee408d'
• 03:34 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/SVG.php 'deployed 3ae632c3c9111b16f019574bcafec58fa3ee408d '
• 03:16 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php
• 03:14 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php
• 03:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php 'Profiling'
• 03:08 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php 'Profiling'
• 02:42 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/SwiftFileBackend.php 'Profiling'
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 17 02:37:38 UTC 2012
• 01:12 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/TempFSFile.php 'deployed 08ea8adc95a4edba0b1f2f13c34f2a57725f1a58'
• 01:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/file/File.php 'deployed 08ea8adc95a4edba0b1f2f13c34f2a57725f1a58'
• 00:13 Tim: copying MW log archives from NFS to fluorine via rsync on fenari

August 16

• 23:39 Tim: on fluorine: chown'd log files and restarted udp2log to get new uid
• 23:39 binasher: reslaved es1002 after conversion to innodb
• 23:31 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/Experiments.hooks.php
• 23:26 mutante: fix date/NTP on sq36
• 23:14 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 23:12 mutante: killing nagios-wm. it stopped talking even though stuff gets written to --infile
• 22:50 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 22:25 logmsgbot: preilly Finished syncing Wikimedia installation... : update mobile frontend
• 22:04 logmsgbot: preilly Started syncing Wikimedia installation... : update mobile frontend
• 22:03 notpeter: restarting puppet on nfs1/2
• 21:54 notpeter: restarting puppet on locke
• 21:51 notpeter: restarting puppet on emery
• 21:39 notpeter: restarting puppet on oxygen
• 21:37 cmjohnson1: shutting ms-be6 down for hardware testing/replacing
• 21:26 notpeter: stopping puppet on nfs2 and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
• 21:20 notpeter: stopping puppet on nfs1 and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
• 21:12 maplebed: ms-be6 down for bad disk repair
• 20:58 Ryan_Lane: attempting to upgrade virt1000 to precise
• 20:47 cmjohnson1: ms-be6 shutting down to replace 2 disks
• 20:38 notpeter: stopping puppet on locke and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
• 20:12 RobH: ms-be1004 is ready to be installed into service
• 19:57 RobH: ms-be1004 down for ssd install
• 19:44 notpeter: stopping puppet on emery and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
• 19:34 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Set $readFromSwiftShared = true;'
• 19:30 logmsgbot: aaron synchronized wmf-config/filebackend.php
• 19:29 notpeter: stopping puppet on oxygen and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
• 19:23 logmsgbot: aaron synchronized wmf-config/filebackend.php
• 19:17 binasher: running purgeParserCache.php again
• 19:05 RobH: puppet restarted on brewster
• 18:58 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Fixed container mapping for shared repo'
• 18:28 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Use NFS for reads for shared-multiwrite'
• 18:14 logmsgbot: aaron synchronized wmf-config/swift.php 'Disabled redundant thumb copy/purge hooks.'
• 18:01 RobH: authdns-update for new es1005-1010
• 17:59 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Make internal reads come from swift for all wikis'
• 17:50 apergos: powercycled ms-be3 ... in theory
• 16:52 cmjohnson1: shutting down mw18 and mw19 to replace DIMM
• 16:29 ottomata: Stopping puppet on brewster again. Sigh. PARTMAAAAN!
• 16:22 logmsgbot: reedy synchronized php-1.20wmf9/includes/QueryPage.php
• 16:07 cmjohnson1: shutting down mw8 to replace bad DIMM
• 15:58 ottomata: starting puppet on brewster. (Woo, partman looking better!)
• 14:15 ottomata: stopping puppet on brewster to continue partman troubleshooting for analytics dells
• 13:26 mark: Enabled and started SIS deduplication on home_pmtpa on nas1-a
• 12:05 mark: Removed OSPF metric on xe-5/2/1.0 on cr2-eqiad, to move eqiad->pmtpa traffic to the lower latency link
• 11:27 pp-pdf2: restarted all services
• 11:27 pp-pdf1: restarted all services
• 11:27 pp-pdf3: restarted all services
• 11:26 pp-pdf3: updated mwlib.epub to 0.14.1
• 11:26 pp-pdf2: updated mwlib.epub to 0.14.1
• 11:25 pp-pdf1: updated mwlib.epub to 0.14.1
• 10:50 mark: Setup semi-sync snapmirror from nas1-a:home_pmtpa to nas1001-a:home_pmtpa
• 03:57 mutante: nagios back up after adding missing monitor groups misc_pmtpa in appserver role (srv194)
• 02:56 K4-713: synchronized payments cluster to 37e31eddf5a4
• 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 16 02:28:33 UTC 2012
• 02:12 Ryan_Lane: pushed in large puppet change for ldap, openstack, gerrit and ldap pdns to make it more modular and to add support for eqiad region
• 02:10 Ryan_Lane: fixed arecord issues with labsconsole by adding an exception handling live hack for the jobs
• 01:21 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'remove wikimedia philippines event exemption'
• 00:40 mutante: installing package upgrades on singer
• 00:36 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed c51a9a288b6dd5c0023a77f324c04707b23501c6'
• 00:24 binasher: running "mwscript purgeParserCache.php --wiki=$db --age=1814400" instead
• 00:19 binasher: running mwscript purgeParserCache.php --wiki=enwiki --age=1209600
• 00:02 RoanKattouw: find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec /root/fixownership2 \{\} \; where fixownership2 = chown -h apache $1
• 00:02 RoanKattouw: Restarting the find-chown, this time with -h so symlinks are handled correctly (for some reason there are a bunch of broken symlinks with weird characters out there...)

August 15

• 23:52 binasher: powercycled srv259
• 23:44 mutante: brought nagios back up by removing "misc_pmtpa" host and servicegroups from srv194 manually
• 23:35 mutante: restarted pdns-recursor on ns0
• 23:11 mutante: authdns-update, moving zirconium to public IP and removing selenium
• 22:33 logmsgbot: reedy synchronized php-1.20wmf9/extensions/OpenSearchXml/
• 21:19 RoanKattouw: find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec chown apache \{\} \;
• 21:17 RoanKattouw: chown on ms7 has finished, running another find to find remaining bad files
• 20:57 K4-713: synchronized payments cluster to 79275e152b76f
• 20:26 ottomata: staring puppet on brewster. PARTMAN IS THE ENEMY. will continue the fight tomorrow.
• 20:22 mutante: http://en.wikipedia.org/w/ now redirects to Main_page on all languages. It was a 403 before.
• 20:20 mutante: apache-graceful-all
• 20:15 mutante: sync-apache pushing out new redirects from /w/ to Main_Page
• 20:04 ottomata: stopping puppet on brewster to experiment with partman on new analytics dells
• 19:29 mark: Removed OSPF/OSPFv3 metric 60 on cr1-sdtpa:xe-1/1/0 (eqiad link)
• 19:17 mark: Moved VRRP mastership from cr2-pmtpa to cr1-sdtpa by reassigning VRRP priorities, relieving the sdtpa-pmtpa link
• 18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remaining wikis over to 1.20wmf9
• 17:11 maplebed: ms-be1 dropped offline yesterday; powercycling.
• 16:11 mark: Setup SnapMirror replication from nas1-a:fr_archive to nas1001-a:fr_archive
• 16:10 mark: Created 4 TB volumes 'fr_archive' in aggregate prod1 on both nas1-a nad nas1001-a
• 16:10 mark: Increased aggregate 'prod1' by 4 drives on both nas1-a and nas1001-a
• 16:06 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 15 16:06:26 UTC 2012
• 15:40 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 15 15:40:21 UTC 2012
• 14:33 cmjohnson1: authdns-update for es9 and es10
• 13:46 RobH: authdns-update for wikipedia.co.za
• 12:15 mark: Started rsync of /home to nas1-a:/vol/home_pmtpa on nfs1
• 12:11 mark: Restricted NFS mounting of /vol/root on all NetApps
• 11:39 mark: Destroyed aggregate 'labs' on nas1-a
• 10:00 Tim: deploying bugfix version of wmerrors
• 09:51 Tim: moved old MW debug logs out of /home/wikipedia/logs to avoid confusion, live logs are now on fluorine
• 05:22 Tim: moving MediaWiki logs to fluorine
• 05:17 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
• 05:15 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
• 04:38 Tim: added rate limit exemption for 27.108.200.111
• 04:38 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
• 03:00 Tim: built a udplog package for precise and uploaded it
• 02:35 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'enabling Scribunto and CodeEditor on test2wiki'
• 02:28 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
• 02:15 logmsgbot: tstarling Finished syncing Wikimedia installation... :
• 02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 01:46 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 01:00 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 00:47 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 00:39 Tim: running scap to for extension-list update

August 14

• 23:53 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Move VE Parsoid from cadmium to wtp1'
• 21:25 logmsgbot: kaldari synchronized php-1.20wmf9/extensions/PageTriage/tools/cleanupPageTriage.php 'adding PageTriage maintainence script'
• 21:05 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 20:45 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 20:38 mutante: killed duplicate nagios-wm
• 20:20 cmjohnson1: db44 swapping out disk1 rt 3424
• 20:08 ottomata: installed precise and puppetized stat1001
• 19:26 cmjohnson1: search23 shutting down to replace DIMM A2 rt 3423
• 19:00 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Deploying https://gerrit.wikimedia.org/r/19399'
• 18:52 RoanKattouw: Running /root/fixownership < /root/badownershipfiles in a screen on ms7
• 18:14 notpeter: stopping puppet on brewster for partman experimentation
• 17:34 logmsgbot: catrope synchronized php-1.20wmf9/.gitmodules
• 17:33 RoanKattouw: Yesterday's find on ms7 is done, found 1.8 million files with bad ownership. Will run a batch chown on them soon
• 17:32 logmsgbot: catrope synchronized php-1.20wmf9/extensions/UploadWizard 'Deploy 9d9189458ad8d966d952d295918847f00e304cb5'
• 09:17 logmsgbot: nikerabbit synchronized php-1.20wmf9/extensions/Translate/ 'I18ndeploy: Translate bugs'
• 07:46 logmsgbot: tstarling Finished syncing Wikimedia installation... :
• 07:07 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 06:48 Tim: running scap to push out CodeEditor and Scribunto sources, not enabled yet
• 05:45 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/ 'Update E3Experiments to master'
• 03:34 Tim: restarting all apaches to get LuaSandbox extension
• 03:26 Tim: on mw8: removed broken package php5-memcached
• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Aug 14 02:45:30 UTC 2012
• 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 14 02:28:13 UTC 2012
• 01:06 Tim: restarting apaches to get new wmerrors configuration

August 13

• 23:53 Tim: uploaded a new version of the php5-wmerrors package for lucid and precise
• 23:18 mutante: removed #wikimedia-tech from /usr/local/bin/start-nagios-bot
• 23:01 mutante: killing nagios-wm process which isn't supposed to be on #wikimedia-tech anymore
• 21:58 cmjohnson1: running authdns-update for labsdb1 & 2
• 21:19 RoanKattouw: ...piping result into ms7:/root/badownershipfiles
• 21:18 RoanKattouw: ms7# find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache
• 21:17 RoanKattouw: Searching for files with wrong ownership on ms7 using
• 21:13 RobH: reinstalling virt1004 to see if the raid issue re-appears
• 20:55 notpeter: updating root auth keys file on ms7 by hand.
• 20:36 logmsgbot: catrope synchronized php-1.20wmf9/resources/jquery/jquery.tablesorter.js 'Deploy 1fafaef3aa157f49cbbf4b9cffb03544bcf72a08'
• 20:04 preilly: updating zero for custom banner colors
• 20:03 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
• 20:03 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
• 19:42 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
• 19:41 preilly: update to zero for banner text link
• 19:41 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
• 19:36 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
• 19:32 RobH: powering off analytics1011-1014 to remove the add on nics (disrupting installer)
• 18:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf9
• 17:21 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Switched backend reads to swift for testwikis and mw.org'
• 15:18 paravoid: reprovisioning srv281 with a new fs layout
• 13:05 mark: Doing NDU of IOM3 shelf firmware on nas1001-a
• 12:59 mark: Doing NDU of IOM3 shelf firmware on nas1-a
• 12:47 mark: Setup NTP on all NetApps
• 12:24 mark: Restored SnapMirror relationship between nas1-a and nas1001-a, 70 days lagged, transfer initiated
• 12:04 paravoid: apt: include php-memcached 2.0.1-6~wmf+lucid2 @ lucid-wikimedia
• 11:58 paravoid: apt: include php5 5.3.10-1ubuntu3.2+wmf1 @ precise-wikimedia
• 11:08 paravoid: apt: remove wikidiff2/0.0.1wm1 (obsolete), copysrc php-wikidiff2 {lucid,precise}-wikimedia
• 11:08 paravoid: apt: remove php-wikidiff2/1.1.0-2 from lucid-wikimedia/universe (lucid-wikimedia/main has 1.1.2-1)
• 10:49 apergos: starting initial rsync from ms8 to ms10 (should take approximately forever)
• 06:26 Tim: uploaded czmq and zpubsub packages to brewster for Ori
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Aug 13 02:37:53 UTC 2012
• 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 13 02:20:27 UTC 2012

August 12

• 03:56 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Aug 12 03:56:33 UTC 2012
• 03:38 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 12 03:38:33 UTC 2012
• 03:19 Reedy: Running localisation update manually in screen session on fenari
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

August 11

• 02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

August 10

• 21:54 Ryan_Lane: live hacked nova on virt2 with parts of nova git commit 4584e552a653904c36cf04cb295a7bf09d2def28
• 21:22 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
• 21:22 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
• 21:21 preilly: update to zero needed to support carrier testing
• 21:21 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero'
• 21:21 preilly: update to zero needed to support carrier testing
• 21:20 logmsgbot: preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero'
• 18:42 Jeff_Green: adding frack.eqiad.wmnet hosts to DNS (wmnet and 10..in-addr.arpa)
• 18:11 K4-713: Payments cluster updated to 133b2b6fb5
• 17:07 notpeter: temporarily stopping puppet on brewster
• 16:43 cmjohnson1: enabling pxe boot for 10GB Nic cards mc1-16
• 16:32 mark: Started OnTap upgrade to 8.1 on nas1-a and nas1-b
• 14:39 mark: OnTap upgrade on nas1001-a and -b complete
• 14:11 mark: Started OnTap upgrade to 8.1 on nas1001-b
• 13:55 mark: Started OnTap upgrade to 8.1 on nas1001-a
• 13:10 mark: Upgraded SP firmware to 1.3 on nas1001-a and nas1001-b
• 11:55 mark: nas1001-b disk fw upgrade completed
• 11:51 mark: nas1001-a disk firmware upgrade completed. Started disk firmware upgrade on nas1001-b
• 11:46 mark: Started disk firmware update on nas1001-a
• 02:39 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Aug 10 02:39:57 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 10 02:22:22 UTC 2012
• 01:07 Tim: on virt0: enabling mysql general log to debug "detached session" errors on nova
• 00:05 Tim: added myself as a labsconsole sysop, added myself to the packaging project, created a "php" instance for building php-related packages

August 9

• 22:05 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Added swift to multiwrite backend for all remaining wikis'
• 22:02 RobH: db1046 back online
• 21:40 RobH: shutting down db1046 to migrate its position in the rack
• 21:27 logmsgbot: olivneh synchronized php-1.20wmf8/extensions/E3Experiments/experiments/communityClicks.js
• 21:20 binasher: resumed enwiki replication in eqiad
• 21:13 binasher: stopping mysql on db1017 for a minute (and all enwiki eqiad replication with it)
• 20:52 logmsgbot: kaldari synchronized wmf-config/extension-list 'Removing deprecated CustomUserSignup extension'
• 20:52 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Removing deprecated CustomUserSignup extension and turning on Curation Toolbar for en.wiki'
• 20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Removing deprecated CustomUserSignup extension'
• 20:09 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 19:48 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'VE bugfix'
• 19:37 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 19:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php
• 19:22 binasher: rebooting db1009, db1010 for upgrades
• 19:20 binasher: rebooting db1011 for upgrade
• 19:17 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak links in VE'
• 18:01 cmjohnson1: re-imaging db68-77
• 16:54 cmjohnson1: loading OS on db65-67
• 16:06 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
• 16:05 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
• 14:53 RobH: cp1001 returned to service, resolving rt 3212
• 14:30 RobH: cp1001 squid services going offline for troubleshooting
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Aug 9 02:37:33 UTC 2012
• 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 9 02:19:57 UTC 2012

August 8

• 21:30 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 20:49 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 20:02 binasher: streaming a hotbackup of db1017 to db1050
• 19:54 ottomata: updating DNS entries for analytics1011-1027, they are all in row-c
• 18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia to 1.20wmf9
• 18:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix $wgVisualEditorParsoidPrefix'
• 18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf9
• 18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiquote over to 1.20wmf9
• 18:35 binasher: completed metawiki centralnotice schema migration for fundraising
• 18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikisource to 1.20wmf9
• 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, fishbowl, closed and private wikis to 1.20wmf9
• 18:25 Reedy: special, fishbowl, closed and private wikis to 1.20wmf9
• 18:16 apergos: shutting down ms10, going to move it to public ip
• 17:21 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
• 17:20 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
• 16:58 mutante: freeing disk space on wikitech
• 16:53 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
• 16:52 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
• 16:46 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero'
• 16:45 logmsgbot: preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero'
• 16:40 cmjohnson1: swapping out main board on search32
• 15:21 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/.git 'Sync .git metadata so Special:Version is accurate'
• 14:43 logmsgbot: reedy synchronized wmf-config/
• 11:49 mark: Upgraded Observium to latest SVN
• 02:40 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 8 02:40:06 UTC 2012
• 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 8 02:21:44 UTC 2012
• 01:04 mutante: a couple package upgrades on bast1001

August 7

• 23:22 logmsgbot: catrope synchronized php-1.20wmf8/resources/mediawiki/mediawiki.js 'Deploying 6b6466f948d29520e1e3ab2592b940ce52415300'
• 23:19 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'Update VisualEditor'
• 22:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Curation Toolbar'
• 22:47 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings.php for Curation Toolbar'
• 22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disabled echo due to bug 39085'
• 22:12 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
• 22:05 binasher: streaming a hotbackup of db1001 to db1049 (new enwiki snapshot host in eqiad)
• 21:48 logmsgbot: reedy synchronized php-1.20wmf9/extensions/EducationProgram/
• 21:45 Reedy: Updated EducationProgram schema on both enwiki and test2wiki
• 20:15 logmsgbot: reedy synchronized php-1.20wmf9/includes/api/ApiQuery.php
• 16:23 LeslieCarr: moving bits traffic from pmtpa to eqiad
• 15:45 mark: Moving text squid traffic from pmtpa to eqiad
• 15:08 first reports of sites being very slow
• 15:07 nagios-wm: Apache HTTP on ... is CRITICAL: CRITICAL - Socket timeout after 10 seconds
• 14:35 Jeff_Green: starting otrs dump on db49
• 07:11 paravoid: powercycled srv272, swapdeath
• 03:09 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 7 03:09:28 UTC 2012
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Aug 7 02:37:34 UTC 2012
• 00:53 mutante: planet (en and fr) configs Updated to revision 115645.
• 00:32 Tim: on cp3002: set tcp_tw_recycle back to zero

August 6

• 23:49 mutante: wikitech back to old version after failed upgrade attempt
• 23:48 mutante: test
• 22:27 mutante: dist-upgrading wikitech instance
• 22:19 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/backend/SwiftFileBackend.php
• 22:14 binasher: rebooting db1028, upgrading
• 22:13 binasher: rebooting db1027, upgrade to precise
• 22:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/SwiftFileBackend.php 'deployed 0a5c71d4dd6405502b1d1d0a02b5d0927d519986'
• 21:59 binasher: rebooting db1026, upgrading to precise
• 21:54 mutante: saving space on wikitech linode - gzipping old .sql dumps and stuff
• 21:32 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'Updating VisualEditor'
• 21:18 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix Parsoid URL'
• 21:09 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor 'Updating VisualEditor'
• 20:23 logmsgbot: catrope synchronized php-1.20wmf9/cache/l10n/ 'Sync out l10n cache again now that it is properly rebuilt'
• 19:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf9
• 19:45 RoanKattouw: Rebuilt ExtensionMessages-1.20wmf9.php and then l10n cache for wmf9 to pick up addition of FundraiserLandingPage.i18n.magic.php , somehow wasn't picked up during initial scap
• 19:44 RobH: asw-c8-eqiad PEM1 power reseated, cleared alarm rt 3204
• 19:39 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.20wmf9.php 'Updated'
• 19:30 RobH: db1047 back online
• 19:18 RobH: db1047 shutting down
• 19:15 RobH: db1047 mysql and system shutdown per rt 3084 for bad memory swap
• 19:14 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 19:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf9
• 19:11 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
• 18:20 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
• 18:12 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
• 18:06 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed b55e9652fce3051d621356c5c87c47f44515a367'
• 18:02 logmsgbot: aaron synchronized wmf-config/filebackend.php
• 17:59 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed b55e9652fce3051d621356c5c87c47f44515a367'
• 17:11 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Adding Swift to local multiwrite backend for testwikis+mw.org'
• 16:57 cmjohnson1: powering down mc1 to reseat DIMM
• 16:50 logmsgbot: reedy synchronized .
• 16:33 logmsgbot: reedy synchronized php-1.20wmf9/
• 15:08 paravoid: switched European bits traffic back to esams
• 14:50 Tim: on cp3002: temporarily set net.ipv4.tcp_tw_recycle=1
• 14:41 Tim: on cp3002 increasing tcp_max_tw_buckets
• 14:33 Tim: on cp3001: date -s `"date"` to fix leap second issue
• 13:26 Nemo_bis: Reedy says: Status: Down - routing issues to Tampa !Wikipedia !Wikimedia
• 13:17 Nemo_bis: more or less everything at !Wikimedia / !Wikipedia seems down
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Aug 6 02:27:00 UTC 2012

August 5

• 21:54 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Update per rearrangements etc'
• 21:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'more cp updates to $wgSquidServersNoPurge'
• 20:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add some eqiad internal ips for squids to wgSquidServersNoPurge'
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Aug 5 02:24:31 UTC 2012

August 4

• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sat Aug 4 02:27:51 UTC 2012

August 3

• 23:50 logmsgbot: reedy synchronized wmf-config/secure.php 'fix bits usages'
• 21:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38988 - Enable Extension:Collection for Wikimedia Sverige'
• 18:58 maplebed: updated DNS to fix *.svc.{esams,eqiad}.wmnet forward resolution
• 18:49 andrewbogott: turned off nova-compute on virt1, virt2, virt3, virt4
• 18:44 maplebed: updating DNS adding swift eqiad LVS address
• 17:42 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler 'Updated to master (0207962155b810b11100bc6d05c3562949e7c1a9)'
• 07:58 hashar: GlusterFS upgrade of labs project storage seems to have been completed. Fixed the issues I had with it on 'beta' project, yeah!!!!!
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Aug 3 02:25:37 UTC 2012
• 01:54 domas: bumped memcached timeout to be slightly above min_rto, 200ms->250ms, should eliminate most of stupid memcached errors because of datacenter network failures
• 01:53 logmsgbot: midom synchronized wmf-config/mc.php

August 2

• 22:48 Ryan_Lane: upgrading glusterfs for project storage to 3.3
• 22:05 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Moved all wikis to multiwrite backend'
• 22:05 mutante: dist-upgrading sodium
• 21:47 hashar: added Dsc to the Gerrit 'integration' group. David now got access to the various integration/* repositories.
• 20:34 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo/modules/base/ext.echo.base.css
• 20:30 logmsgbot: asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent back to false, no change observed to current issue'
• 20:12 logmsgbot: asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent = true as an experiment'
• 20:06 Andrew: Leaving Echo deployment there for now.
• 20:04 Andrew: Rolled back LQT updates, they seem to break other notifications. We'll figure out the bugs with that in another window.
• 20:02 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
• 19:56 Andrew: deployed Echo to mediawiki.org
• 19:56 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
• 19:55 logmsgbot: andrew synchronized wmf-config/CommonSettings.php
• 19:55 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo
• 19:54 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
• 19:44 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
• 19:44 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo
• 19:43 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
• 19:43 Andrew: Deploying Echo to test2wiki instead
• 18:38 Andrew: Turned Echo back on on testwiki
• 18:35 Andrew: Stealing E2's deployment window to have another crack at Echo
• 16:44 paravoid: sync-apache/apache-gracefull-all for BZ #38905 (ShortUrl on !Wikipedia)
• 16:20 RobH: authdns-update for smokeping
• 13:05 logmsgbot: nikerabbit synchronized php-1.20wmf8/extensions/Translate/specials/ 'Translate bug fixes'
• 04:45 logmsgbot: tstarling synchronized php-1.20wmf8/skins/common/commonElements.css
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Aug 2 02:27:17 UTC 2012
• 02:11 Ryan_Lane: deployed OATHAuth 8992e6f541cd7adf8111ccd87f25a481d3759b33 on labsconsole
• 02:01 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
• 01:26 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/
• 01:16 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php
• 00:55 logmsgbot: asher synchronized wmf-config/db.php 'moving watchlist/special contrib queries to db63, preparing to decom db12'
• 00:47 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php
• 00:43 logmsgbot: asher synchronized wmf-config/db.php 'adding db63 to s1 at a low weight'

August 1

• 23:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 23:27 mutante: rebuilding wikitech-l archives
• 23:16 logmsgbot: andrew Finished syncing Wikimedia installation... :
• 22:59 Andrew: pulling the plug on Echo deployment due to replication-lag related bug.
• 22:57 logmsgbot: andrew Started syncing Wikimedia installation... :
• 22:32 Andrew: Running scap to deploy Echo to test
• 22:31 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made mw.org use the multiwrite backend.'
• 22:23 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'logging for /home being used for thumbs for testwiki'
• 22:14 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed read-only flag from testwiki backends.'
• 22:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made testwikis use multiwrite backend.'
• 21:27 Andrew: Added Echo tables to testwiki
• 21:11 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Configured $wgTimedTextForeignNamespaces.'
• 21:08 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler/handlers/TextHandler/TextHandler.php 'Deployed a247b6264c99cff66e4e3ae3428a46f971fab3a2'
• 21:07 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler/TimedMediaHandler.php 'Deployed a247b6264c99cff66e4e3ae3428a46f971fab3a2'
• 21:01 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
• 20:48 logmsgbot: andrew synchronized wmf-config/CommonSettings.php
• 20:47 Andrew: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
• 20:47 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
• 20:46 Andrew: syncing {Common,Initialise}Settings.php
• 20:46 Andrew: Added configuration skeleton for Echo in {Common,Initialise}Settings.php
• 20:46 Andrew: Created tables for Echo on mediawikiwiki
• 19:16 MaxSem: Ran extensions/GeoData/updateIndexGranularity.php on testwiki
• 19:04 binasher: started hotbackup of db1017 to db63
• 19:04 mark: Ran reprepro --delete clearvanished on brewster to cleanup removed repositories karmic-wikimedia and oneiric-wikimedia
• 19:03 mutante: installing a couple lib upgrades on fenari
• 18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything else over to 1.20wmf8
• 18:04 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler 'Updating to 4b7c77d63d7798a024c7f8b4ba01e2f3e7203550'
• 17:37 maplebed: pushing dns typo correction
• 10:32 Tim: on dobson: restarted ntpd
• 10:30 Tim: on linne: restarted ntpd in an attempt to fix the leap indicator
• 02:49 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Aug 1 02:49:33 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 1 02:25:43 UTC 2012
• 01:53 logmsgbot: catrope synchronized php-1.20wmf8/cache/l10n
• 01:05 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayer.js 'touch'
• 00:33 Tim: leap second event plus one month is causing an apparent 1s step in time reported by linne/dobson as seen by some clients, causing nagios errors etc. Will step.

July 31

• 23:12 logmsgbot: aaron Finished syncing Wikimedia installation... :
• 22:32 logmsgbot: aaron Started syncing Wikimedia installation... :
• 22:19 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disable ogghandler for wikis using TMH.'
• 22:18 logmsgbot: catrope synchronized php-1.20wmf7/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php Deploying 3d45bc25577f33bc220e17a7671d391977f9fbf9 and 4c7db40395b6b2d273e08f6b0e59c5a74cb5ed34'
• 22:17 RobH: authdns-update for ms-be1006 and ms-be1012
• 22:04 binasher: added memcached_1.4.14-0wmf1_amd64 to precise-wikimedia
• 21:55 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
• 21:54 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
• 21:49 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
• 21:49 logmsgbot: aaron Finished syncing Wikimedia installation... :
• 21:45 logmsgbot: aaron Started syncing Wikimedia installation... :
• 21:08 logmsgbot: aaron Finished syncing Wikimedia installation... :
• 21:03 logmsgbot: aaron Started syncing Wikimedia installation... :
• 20:09 logmsgbot: mlitn Finished syncing Wikimedia installation... :
• 19:35 logmsgbot: mlitn Started syncing Wikimedia installation... :
• 19:26 RobH: authdns-update for wtp1 info
• 17:40 AaronSchulz: Initial swift originals migration for s1-s7 done
• 17:29 maplebed: finished swift deploy - image scaling requests now go straight to the rendering cluster
• 16:16 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'Recognize thumb_handler for testwiki and /home check.'
• 16:09 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Deployed handlerUrl for thumb zone.'
• 16:00 maplebed: beginning swift deploy to make thumbnail requests bypass ms5 and go straight from swift to the rendering cluster
• 14:33 mark: Repooled cp1042
• 14:33 mark: Reinstalled cp1042 with Precise
• 13:16 Tim: compiling phpllvm tests on bast1001
• 12:19 mark: Depooled cp1042 for reinstall with Precise
• 08:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 08:45 logmsgbot: reedy synchronized flaggedrevs.dblist 'Disable FR on eswikibooks'
• 02:54 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 31 02:54:35 UTC 2012
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Jul 31 02:27:34 UTC 2012

July 30

• 23:49 binasher: adding revised enwiki.aft_article_feedback af_user_id_user_ip index via osc
• 23:48 binasher: dropping enwiki.aft_article_feedback af_user_id_user_ip index
• 21:48 logmsgbot: reedy synchronized php-1.20wmf7/extensions/Collection/.git 'Just for saper..'
• 20:51 binasher: adding enwiki.aft_article_feedback af_user_id_ip index
• 20:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 20:30 logmsgbot: reedy synchronized php-1.20wmf8/extensions/Collection
• 20:29 logmsgbot: reedy synchronized php-1.20wmf7/extensions/Collection
• 19:26 logmsgbot: reedy synchronized .
• 19:22 logmsgbot: reedy synchronized php-1.20wmf8/extensions/UploadWizard/
• 18:25 notpeter: removing srv194 from apaches pool due to logging issues
• 18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf8
• 17:42 notpeter: re-adding srv194 to pmtpa apaches pool for basic testing of precise
• 17:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove duplicate routing code'
• 16:55 logmsgbot: reedy synchronized wmf-config/
• 16:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable shorturl on tawikis'
• 16:31 Reedy: Created ShortUrl table on numerous tawiki* wikis
• 16:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ShortUrl on tawiki, hiwiki and orwiki'
• 16:26 Reedy: Add ShortUrl table to hiwiki, orwiki and tawiki
• 16:05 paravoid: sync-apache/apache-graceful-all for RT #2121
• 16:02 Reedy: Created FlaggedRevs tables on eswikiboooks
• 15:58 logmsgbot: reedy synchronized wmf-config/
• 15:53 Reedy: Created FlaggedRevs tables on eswiki
• 08:11 hashar: gallium/jenkins: deployed job for the WLMMobile nightly builds
• 07:40 hashar: gallium/jenkins: updating Android SDKs
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 30 02:45:58 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Jul 30 02:23:05 UTC 2012
• 01:00 Tim: installed mytop on fenari at faidon's request
• 00:52 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'docs'
• 00:49 Tim: killed all AFTv5 queries another few times
• 00:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabled properly'
• 00:45 Tim: killed ArticleFeedback queries on db12
• 00:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabled ArticleFeedback since it caused an overload on db12 and general site slowness'
• 00:32 Tim: on hume: kill -CONT
• 00:28 Tim: on hume: stopped populateRevisionSha1.php with kill -STOP due to excessive (800s) lag on db12

July 29

• 23:13 logmsgbot: reedy synchronized php-1.20wmf8/includes/WebRequest.php
• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 29 02:45:34 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Jul 29 02:23:12 UTC 2012

July 28

• 06:29 logmsgbot: catrope synchronized php-1.20wmf8/extensions/Vector/modules/ext.vector.collapsibleNav.css 'touch'
• 06:29 logmsgbot: catrope synchronized php-1.20wmf7/extensions/Vector/modules/ext.vector.collapsibleNav.css 'touch'
• 02:51 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 28 02:51:29 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sat Jul 28 02:25:46 UTC 2012

July 27

• 21:16 RobH: updated dns for mgmt on wtp1.mgmt
• 20:42 cmjohnson1: setting db66
• 20:23 RobH: authdns-udpate for yttrium/wlm.w.o per rt 3221
• 20:12 cmjohnson1: setting up tmh2 rt3298
• 19:51 cmjohnson1: setting up tmh1 rt 3298
• 19:40 RobH: setting up yttrium per rt 3221
• 19:32 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
• 19:31 logmsgbot: aaron synchronized php-1.20wmf7/includes/DefaultSettings.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
• 19:31 logmsgbot: aaron synchronized php-1.20wmf7/includes/filerepo/FileRepo.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
• 19:26 logmsgbot: aaron synchronized php-1.20wmf8/thumb.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
• 19:25 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/FileRepo.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
• 19:24 logmsgbot: aaron synchronized php-1.20wmf8/includes/DefaultSettings.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
• 19:00 RobH: allocating potassium and zirconium to rt 3342 (misc db use) and shutting them down until they are installed.
• 18:37 RobH: another authdns-update, forgot to update asset tags to servernames for mgmt
• 18:33 RobH: authdns-update for tmh1/2
• 14:07 hashar: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
• 14:07 hashar: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
• 14:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'gerrit 16817 - fix nycwikimedia wgImportSources'
• 13:40 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Deploying I39d1ca7c: import sources for nycwikimedia'
• 12:51 Reedy: Created tgs_lang indexes on wikis that have translate installed
• 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Fri Jul 27 02:53:30 UTC 2012
• 02:31 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Jul 27 02:31:05 UTC 2012

July 26

• 22:26 logmsgbot: kaldari synchronized php-1.20wmf7/extensions/E3Experiments/Experiments.hooks.php 'syncing Experiments.hooks.php for wmf7'
• 22:25 logmsgbot: kaldari synchronized php-1.20wmf8/extensions/E3Experiments/Experiments.hooks.php 'syncing Experiments.hooks.php for wmf8'
• 22:00 logmsgbot: reedy synchronized wmf-config/
• 21:55 logmsgbot: reedy synchronized php-1.20wmf7/LocalSettings.php
• 21:48 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning E3Experiments on for en.wiki'
• 21:38 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:09 logmsgbot: kaldari synchronized php-1.20wmf8/extensions/E3Experiments 'sync E3Experiments stuff for en.wiki'
• 21:02 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on E3Experiments for en.wiki'
• 19:40 RobH: changed to calcium instead, as yttrium is a 610 and i only need a 310
• 19:37 RobH: claiming yttrium for smokeping install
• 19:23 RobH: authdns update for new services
• 18:59 logmsgbot: reedy synchronized php-1.20wmf8/extensions/WikiEditor
• 16:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 16:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 16:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Special -'
• 16:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'enable shorturl on testwiki and test2wiki'
• 16:42 Reedy: Created ShortUrl tables on test2wiki
• 13:44 hashar: deployment-prep rsync finished for both apache and upload6. Remounting and restarting apaches
• 13:23 cmjohnson1: search32 has hardware issues..powering down
• 08:43 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php '(bug 32516) Enable Narayam on pawiki.'
• 08:38 logmsgbot: nikerabbit synchronized php-1.20wmf7/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/16721'
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 26 02:46:34 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Jul 26 02:24:01 UTC 2012
• 00:21 logmsgbot: catrope synchronized php-1.20wmf8/skins/common/commonElements.css 'Deploy 0cc26386cec1955acf2e1f7341eaaa466ea220b2 and 162a25e96cf48cbd2545f6c3d54f0c20f1bcd520'
• 00:08 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Undo testing change'

July 25

• 23:46 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
• 23:30 binasher: completed abuse_filter_log migration for all wikis
• 22:54 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php
• 22:29 LeslieCarr: fixed puppetmaster on stafford (bad stafford, no cookie!)
• 22:20 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'url extraction fix'
• 22:15 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'debug logging'
• 22:08 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'url regex tweak'
• 22:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily set $wmgArticleFeedbackv5OversightEmails to my work e-mail for debugging'
• 22:05 binasher: started second step of abuse_filter_log migration via osc, indexes for new columns
• 21:56 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'debug logging'
• 21:08 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Re-implementing wgNoticeFundraisingUrl override for wmf7 wikis'
• 21:02 logmsgbot: aaron synchronized live-1.5/thumb_handler.php 'entry point to thumb.php'
• 20:43 binasher: started abuse_filter_log migration via osc. first step, adding afl_rev_id, afl_log_id columns. running in cluster order starting with s1 (enwiki)
• 20:30 logmsgbot: reedy synchronized php-1.20wmf8/includes/objectcache/MemcachedClient.php 'Bring in reversion to shut up warnings'
• 20:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all none wikipedia projects to 1.20wmf8
• 20:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 19:49 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/ 'Revert back to 0eeafeab44fb7a177ebf929c313999855656701f'
• 18:25 logmsgbot: reedy synchronized php-1.20wmf8/extensions/Quiz/
• 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, closed and special to 1.20wmf8
• 18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews to 1.20wmf8
• 18:02 binasher: streaming hot backup of db1033 to db63, intended db12 replacement and first precise enwiki db
• 17:10 mark: Added asw-c-eqiad to Torrus, RANCID, Observium
• 16:52 cmjohnson1: srv266 still working on it...going down
• 15:46 logmsgbot: reedy synchronized wmf-config/
• 15:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 15:33 Reedy: Created wikilove tables on svwikinews
• 15:24 Reedy: Created FlaggedRevs tables on cawikinews
• 15:23 logmsgbot: reedy synchronized wmf-config/
• 14:37 cmjohnson1: srv266 shutting down for attempted h/w repairs
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Jul 25 02:46:06 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 25 02:23:44 UTC 2012

July 24

• 23:23 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Synchronizing updates to MobileFrontend per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-06-24
• 22:52 logmsgbot: awjrichards Started syncing Wikimedia installation... : Synchronizing updates to MobileFrontend per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-06-24
• 22:26 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Fixing footer logo for enwiki on mobile view'
• 20:10 binasher: upgrading versions of xtrabackup and percona toolkit on all coredbs
• 19:45 logmsgbot: mlitn Finished syncing Wikimedia installation... :
• 19:10 logmsgbot: mlitn Started syncing Wikimedia installation... :
• 18:30 RoanKattouw: root@fenari:~# chmod g+w /home/wikipedia/common/php-1.20wmf8/cache/l10n/
• 18:30 RoanKattouw: root@fenari:~# chown l10nupdate:wikidev /home/wikipedia/common/php-1.20wmf8/cache/l10n
• 18:04 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
• 16:28 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
• 16:08 RobH: srv278 hardware worked on by chris, placing back in service to see if its going to stay fixed
• 14:30 cmjohnson1: correcting enviromental leads on powestrips in sdtpa
• 14:06 paravoid: force puppet run on all LVS servers for the ganglia change
• 09:10 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 36104 - Narayam on bnwikisource'
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Jul 24 02:48:21 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 24 02:26:52 UTC 2012
• 00:03 binasher: upgrading db1045 to precise for testing

July 23

• 23:11 AaronSchulz: Started originals migrations for remaining wikis
• 22:07 binasher: deploying new mobile redirector to eqiad squids
• 20:44 cmjohnson1: ms-be10 taking off-line for some hardware testing
• 20:13 logmsgbot: reedy synchronized php-1.20wmf7/includes/specials/SpecialUserlogin.php
• 20:12 logmsgbot: reedy synchronized php-1.20wmf8/includes/specials/SpecialUserlogin.php
• 19:31 RobH: authdns-update for transcode name updates
• 19:28 RobH: powering down old transcode1, was camera gateway sandbox, reclaiming name
• 18:57 maplebed: took ms-be10 out of rotation because it ate itself
• 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf8
• 17:03 RobH: reobooting potassium, shouldnt be in use, had tons of cruft ssh connections from month ago
• 16:26 Reedy: running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf5"
• 16:23 Reedy: running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf4"
• 16:22 logmsgbot: reedy Finished syncing Wikimedia installation... : 1.20wmf8 messages
• 16:04 paravoid: reinstalling srv281; disk full, hasn't run puppet for 300 days, depooled for ages
• 15:54 logmsgbot: reedy Started syncing Wikimedia installation... : 1.20wmf8 messages
• 15:44 logmsgbot: reedy Started syncing Wikimedia installation... : 1.20wmf8 messages
• 15:42 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf8 and rebuild localisation cache
• 15:23 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf8 and rebuild localisation cache
• 15:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rm chwikimedia we dont host it
• 15:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: chwikimedia to 1.20wmf7
• 15:14 Reedy: srv281 has a full / and hasn't had a puppet run in over 434298 minutes
• 15:10 logmsgbot: reedy synchronized php-1.20wmf8/cache/
• 15:06 logmsgbot: reedy synchronized php-1.20wmf8/ 'Initial sync out of 1.20wmf8'
• 13:44 RobH: authdns-update for ms-be eqiad hosts
• 12:46 maplebed: rebooting ms-be10 for xfs errors and a clean boot
• 12:38 maplebed: rebalanced swift rings moving more content to new object servers
• 12:26 logmsgbot: reedy synchronized flaggedrevs.dblist
• 12:26 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
• 12:25 Reedy: Created trwikiquote flagged revs tables
• 12:23 logmsgbot: reedy synchronized wmf-config/
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 23 02:22:45 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 23 02:22:33 UTC 2012

July 22

• 22:56 Ryan_Lane: restarted opendj on sanger. The process OOM'd due to heap size.
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 22 02:23:02 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 22 02:22:49 UTC 2012

July 21

• 17:42 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'deployed ffd469239a80b8bb9813eb029e4cd71a3ea52db9'
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 21 02:23:19 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 21 02:23:06 UTC 2012

July 20

• 18:57 paravoid: powerin on owa1/owa2 again, they're being used as ganglia aggregators for swift
• 18:11 Ryan_Lane: starting gerrit
• 17:59 binasher: db48 is now replicating from db1048 (and db1048 from db48)
• 17:26 RobHalsell: authdns-update run for new servers
• 15:29 paravoid: powering off the rest of mw10[0-9][0-9]
• 14:35 mark: Added server hydrogen to the dns_rec eqiad LVS pool
• 14:22 paravoid: powering off owa1/2/3, unused
• 13:54 paravoid: powering off all of mw1[0-9][0-9][0-9].eqiad.wmnet, unused
• 13:36 mark: Reinstalling hydrogen
• 13:19 paravoid: powercycling hydrogen, down since yesterday
• 12:59 paravoid: powercycling nescio/ns2, unresponsive network & console
• 10:38 mark: Built new varnish 3.0.3~rc1+persistent1-wm1 packages and inserted them into the precise-wikimedia APT repository
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 20 02:23:39 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Fri Jul 20 02:23:27 UTC 2012

July 19

• 22:48 logmsgbot: catrope Finished syncing Wikimedia installation... :
• 22:10 logmsgbot: catrope Started syncing Wikimedia installation... :
• 22:02 binasher: deploying new mobile redirector to pmtpa text squids (currently inactive)
• 21:39 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:36 binasher: deploying new mobile redirector to eqiad text squids
• 21:36 logmsgbot: catrope synchronized php-1.20wmf7/extensions/LastModified 'Remove E3Experiments cruft from LastModified'
• 21:07 binasher: deploying new mobile redirector to esams text squids
• 18:52 preilly: fixing subdomain check for zero vs m
• 18:52 logmsgbot: preilly synchronized wmf-config/mobile.php 'fix subdomain check'
• 18:21 RobH: updating dns with new zonefiles for legally won domain names
• 18:15 Jeff_Green: storage3 dist-upgrade and reboot
• 17:01 AaronSchulz: Running copyFileBackend.php for commons (shards c-f)
• 16:28 cmjohnson1: mw60 powering down to replace DIMM B1 rt3287
• 15:48 RobH: hydrogen repaired per rt 3243
• 15:37 RobH: rebooting hydrogen to set bios redirection
• 14:18 logmsgbot: reedy synchronized wmf-config/
• 13:42 Jeff_Green: dist-upgrade and reboot hosts in payments cluster
• 13:12 Reedy: Pointed /h/w/c/php to php-1.20wmf7
• 13:10 Reedy: Deleted php-1.20wmf6/cache/l10n from mediawiki-installation
• 02:31 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 19 02:31:02 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 19 02:23:27 UTC 2012

July 18

• 22:02 notpeter: ok, changed my mind one more time. removing srv194 from apaches pool for precise testing
• 21:41 notpeter: scratch that. removing srv289 from apaches pool for precise testing, not mw1
• 21:39 notpeter: removing mw1 from apaches pool to do precise test install
• 21:28 notpeter: adding php5 packages to precise-wikimedia repo
• 19:45 cmjohnson1: srv281 powering down for HW checks
• 19:09 cmjohnson1: srv278 powering down for hardware problems -- random reboot system already depooled
• 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 295 remaining wikis to 1.20wmf7
• 17:37 cmjohnson1: mw8 powering down to replace DIMM A3 rt 3273
• 17:25 cmjohnson1: search32 powering down to replace DIMM B1 rt 3076
• 16:55 logmsgbot: reedy synchronized php-1.20wmf7/includes/api/ApiQuerySiteinfo.php
• 12:52 pp-pdf1: upgraded mwlib to 0.13.11, restarted all services
• 07:58 hashar: gallium: added Firefox 14 to Testswarm, disabled Firefox 13.
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 18 02:47:23 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 18 02:24:14 UTC 2012

July 17

• 23:24 Tim: on srv193: removed core dump files, disabled core dumping, restarted apache
• 19:52 logmsgbot: mlitn Finished syncing Wikimedia installation... :
• 19:09 logmsgbot: mlitn Started syncing Wikimedia installation... :
• 15:18 mark: lvs1002 is back up and idling
• 15:15 mark: Fixed serial console redirection after boot to OFF on lvs1002
• 15:06 mark: lvs4 is back up and serving traffic
• 14:51 mark: Reinstalling lvs4 with Ubuntu Precise
• 14:48 mark: Stopped PyBal on lvs4, failing over traffic to lvs3
• 14:47 mark: Reinstalling lvs1002 with Ubuntu Precise
• 14:46 mark: Stopped PyBal on lvs1002, failing over traffic to lvs1005
• 14:44 mark: lvs1001 is back up and serving traffic
• 14:11 mark: Stopped PyBal on lvs1001, failing over traffic to lvs1004
• 13:09 maplebed: changed auth URL for swift to use load balancer rather than round robin DNS
• 13:08 mark: lvs1003 is back up and serving traffic
• 13:07 maplebed: changed order on ganglia swift view graphs to group by metric rather than host
• 13:02 logmsgbot: reedy synchronized wmf-config/
• 12:53 mark: Fixed boot order on lvs1003
• 12:34 mark: Reinstalling lvs1003 with Ubuntu Precise
• 12:28 mark: Stopped PyBal on lvs1003, failing over traffic to lvs1006
• 12:10 mark: amslvs1 is back up and serving traffic
• 11:44 mark: Reinstalling amslvs1 with Ubuntu Precise
• 11:40 mark: Stopped PyBal on amslvs1, failing over traffic to amslvs3
• 11:32 mark: amslvs2 is back up and serving traffic
• 10:47 mark: Reinstalling amslvs2 with Ubuntu Precise
• 10:36 mark: Stopped PyBal on amslvs2, failing over traffic to amslvs4
• 07:30 Tim: testing envvars/apache2.conf change on srv193
• 07:09 Tim: restarted apache on srv193
• 02:58 Tim: graceful restart of all apaches
• 02:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jul 17 02:47:06 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 17 02:23:41 UTC 2012
• 00:44 Tim: deploying new redirects.conf
• 00:28 Tim: testing new redirects.conf on mw1

July 16

• 23:03 pp-pdf1: restarted nserve
• 23:02 pp-pdf1: fixed nserve handling of filenames with whitespace
• 22:46 pp-pdf1: restart all services
• 22:45 pp-pdf2: restart all services
• 22:45 pp-pdf3: restart all services
• 22:45 pp-pdf1: update mwlib to 0.13.9
• 22:45 pp-pdf2: update mwlib to 0.13.9
• 22:45 pp-pdf3: update mwlib to 0.13.9
• 22:44 pp-pdf2: update greenlet to 0.4.0
• 22:44 pp-pdf1: update greenlet to 0.4.0
• 22:44 pp-pdf3: update greenlet to 0.4.0
• 22:44 pp-pdf2: upgrade qserve to 0.2.8
• 22:44 pp-pdf3: upgrade qserve to 0.2.8
• 22:43 pp-pdf1: upgrade qserve to 0.2.8
• 22:20 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Disabling epub generation from Collection extension on all wikis except for simple and test'
• 22:19 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Making epub generation for Collection extension configurable'
• 18:26 AaronSchulz: Running copyFileBackend.php for commons (shards 8-b)
• 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf7
• 17:08 LeslieCarr: reactivating XO transit connections on cr1-sdtpa
• 17:03 LeslieCarr: draining cr2-eqiad to cr1-sdtpa link for moving of fiber
• 15:56 mark: lvs3 is back up, and idling
• 15:56 mutante: fixing pt.planet after missing locale has been added
• 15:48 mutante: installing package upgrades on singer (planet)
• 15:43 cmjohnson1: mw8 shutting down and taking offline to run Dell's dset program
• 15:38 mutante: planet - svn up the config
• 15:29 mark: Reinstalling lvs3 with Ubuntu Precise
• 15:17 mark: lvs5 is back up and serving traffic
• 14:49 mark: Reinstalling lvs5 with Ubuntu Precise
• 14:45 mark: Stopped PyBal on lvs5 to failover traffic to lvs1
• 14:35 maplebed: adjusted swift rings; set new object servers to 20, new container servers to 100
• 14:33 mark: lvs6 is back up and serving traffic
• 14:12 mark: Reinstalling lvs6 with Ubuntu Precise
• 14:02 mark: Stopped PyBal on lvs6 to failover traffic to lvs2
• 13:58 mutante: mw8 - alright, most likely just needs new DIMM per cmjohnson
• 13:50 mutante: mw8 PHP fatal errors, running out of memory
• 12:56 mutante: sync-apache, graceful-all to fix wikipedia.cz redirect
• 05:31 apergos: reboot to upgrade kernel etc on mw8 since it's been flapping anyways
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 16 02:46:14 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 16 02:23:04 UTC 2012

July 15

• 20:19 Aaron|home: Running copyFileBackend.php for commons (shard 7)
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 15 02:46:33 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 15 02:23:27 UTC 2012

July 14

• 18:46 LeslieCarr: powercycling frozen db1029
• 18:01 LeslieCarr: rebooting unresponsive mw1111
• 16:41 logmsgbot: midom synchronized php-1.20wmf6/includes/GlobalFunctions.php 'message key analysis'
• 16:18 logmsgbot: midom synchronized php-1.20wmf6/includes/GlobalFunctions.php 'message key analysis'
• 03:07 Aaron|home: Running copyFileBackend.php for commons (shards 5-6)
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 14 02:47:32 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 14 02:23:14 UTC 2012
• 00:25 logmsgbot: reedy synchronized php-1.20wmf7/includes/api/ApiDelete.php

July 13

• 18:08 logmsgbot: reedy synchronized php-1.20wmf7/includes/GlobalFunctions.php
• 18:08 logmsgbot: reedy synchronized php-1.20wmf6/includes/GlobalFunctions.php
• 17:58 logmsgbot: reedy synchronized wmf-config/
• 17:57 logmsgbot: reedy synchronized php-1.20wmf7/
• 17:56 logmsgbot: reedy synchronized php-1.20wmf6/
• 17:32 logmsgbot: reedy synchronized wmf-config/
• 17:21 AaronSchulz: Running copyFileBackend.php for commons (shards 1-4) (actually started yesterday)
• 15:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable aftv5 on en_labswikimedia due to out of date schema'
• 15:21 logmsgbot: reedy synchronized php-1.20wmf7/includes/GlobalFunctions.php
• 15:20 logmsgbot: reedy synchronized php-1.20wmf6/includes/GlobalFunctions.php
• 15:20 logmsgbot: reedy synchronized php-1.20wmf7/includes/DefaultSettings.php
• 15:19 logmsgbot: reedy synchronized php-1.20wmf6/includes/DefaultSettings.php
• 15:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Setting wgDBerrorLogInUTC = true'
• 15:07 hasharDeadmau5: unknown column 'af_is_featured' on en_labswikimedia@db25
• 14:55 mark: Increased all mobile varnish server weights from 10 to 100 to aid chash
• 14:54 mark: Added cp1041.eqiad.wmnet back into the mobile LVS pool
• 10:10 mutante: mw1016 - was down, reinstalling with precise
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 13 02:45:59 UTC 2012

July 12

• 22:15 Aaron|home: Running copyFileBackend.php for commons (shards 0,c-f)
• 20:58 Aaron|home: Running copyFileBackend.php for commons (shards 0,8-b)
• 14:22 mutante: apache children on srv193 keep segfaulting since yesterday (test.wp)
• 14:00 RobHalsell: shutting down srv206 for chris per rt241
• 11:55 logmsgbot: reedy synchronized wmf-config/
• 10:40 mark: Depooled mobile varnish server cp1041 for vlan/hostname change and reinstallation with precise
• 04:38 binasher: started hot backup of db48 to db1048
• 02:50 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 12 02:50:49 UTC 2012
• 02:29 maplebed: adjusted the swift rings a bit to move some more traffic to the new hosts. set object partitions to weight 10, account and container 60.
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 12 02:26:57 UTC 2012

July 11

• 22:52 AaronSchulz: Running copyFileBackend.php for commons (shards 0,4-7)
• 22:06 AaronSchulz: Running copyFileBackend.php for commons (shards 0,0-3)
• 21:36 binasher: running hotbackup of db48 -> db49 (otrs / external misc)
• 21:31 binasher: rebooting db49 for kernel upgrade
• 21:23 binasher: rebuilding db49 (otrs slave)
• 21:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 20:16 logmsgbot: reedy synchronized php-1.20wmf7/extensions/CentralNotice
• 19:33 cmjohnson1: db10 swapping disk1 out rt3251
• 19:33 RobH: shutting down srv203 for hardware checking per rt3110
• 19:12 hashar: finally stopped using my production ssh key on labs.
• 19:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikiversity to 1.20wmf7
• 19:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.20wmf7
• 19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikisource to 1.20wmf7
• 19:03 cmjohnson1: search35 shutting down to reseat DIMM rt-3260
• 18:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Switch special, private and closed wikis from 1.20wmf6 to 1.20wmf7
• 15:16 logmsgbot: robh synchronized wmf-config/mc.php 'moving srv208 to down'
• 15:14 logmsgbot: robh synchronized wmf-config/mc.php
• 14:30 cmjohnson1: shutting down search32 to reseat DIMM rt-3076
• 11:16 mutante: yay @ svn removal
• 11:13 hashar: /home/wikipedia/conf/httpd cleaned out the svn repository. That directory is now 100% under git tracking yeah!!! Thanks TIm!
• 11:11 mutante: rebooting srv266
• 11:05 mutante: purged wikipedie.cz/Experti_na_prirodu URLs with purgeList.php
• 10:52 mutante: apache-graceful-all
• 10:48 mutante: sync-apache to push fixed wikipedie.cz redirect
• 08:36 hashar: synced /h/w/conf/httpd git and svn repositories
• 08:33 mutante: chmod -R g+w /home/wikipedia/conf/httpd on fenari to fix group write on .git/objects
• 07:59 mutante: srv266 - package/kernel upgrades
• 07:35 mutante: powercycled downed owa3, installed kernel upgrades on owa1-3 (but they are also waiting to be repurposed, RT:2511)
• 04:05 Tim: on hume: running updateCollation.php --dry-run on ptwiki to determine whether there will be any key truncation issues with bug 35632
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 11 02:48:18 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 11 02:25:19 UTC 2012

July 10

• 23:47 AaronSchulz: Doing shards c-f
• 23:47 maplebed: put two new swift front ends into rotation ms-fe3 and ms-fe4
• 23:40 AaronSchulz: Doing shards 8-b
• 23:34 AaronSchulz: Doing shards 4-7
• 23:31 AaronSchulz: copyFileBackend.php run rate for above processes is at /home/aaron/NFStoSwiftCopyRate, currently 50
• 23:26 AaronSchulz: Running copyFileBackend.php for zhwiki public zone shards 0-3 on hume
• 23:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 22:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 22:12 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'moving parsercache from db40 to pc1'
• 22:05 maplebed: put swift backends ms-be9-12 in rotation for containers (weight 30) and objects (weight 5)
• 21:17 maplebed: rebooting new ms-be6-8 to change BIOS setting to boot from disk
• 20:26 logmsgbot: mlitn Finished syncing Wikimedia installation... :
• 20:26 RobH: shutting down sq36 for hardware troubleshooting
• 19:38 LeslieCarr: restarted frozen pdns on ns2
• 19:28 logmsgbot: mlitn Started syncing Wikimedia installation... :
• 19:15 cmjohnson1: sq36-requires a hard shutdown-unresponsive to mgmt. rt-3254
• 19:05 RoanKattouw: chmod g+w /home/wikipedia/common/php-1.20wmf7/cache/l10n/
• 19:03 RoanKattouw: chown l10nupdate:wikidev /home/wikipedia/common/php-1.20wmf7/cache/l10n/
• 18:58 hashar: reworked the git repository in /home/wikipedia/conf/httpd , manually synced changes from svn to the git repo
• 18:12 RobH: srv266 shutdown for chris
• 18:06 cmjohnson1: srv266 being brought down for an extended period of time to run diagnostic tests
• 17:25 cmjohnson1: srv266 shutting down for HW troubleshooting rt-2896
• 16:33 mutante: package upgrades and kernel on niobium
• 16:28 mutante: powercycling niobium
• 15:49 mutante: powercycling mw1008,mw1070,mw1073
• 14:10 logmsgbot: reedy synchronized hastidy
• 13:26 logmsgbot: tstarling synchronized live-1.5/robots.php
• 13:17 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 13:13 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 12:53 Tim: deploying favicon.php test alias
• 12:41 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 12:30 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 12:11 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 12:10 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 12:03 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 12:02 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 11:58 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 10:10 logmsgbot: reedy synchronized images/sul/ 'crushed'
• 10:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 07:49 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 07:46 logmsgbot: tstarling synchronized live-1.5/favicon.php
• 07:41 logmsgbot: tstarling Finished syncing Wikimedia installation... :
• 06:32 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 06:14 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 06:06 Tim: removing php-*/cache/l10n/l10nupdate* and running scap with the new version of scap from I8bcd2817
• 05:01 Tim: on fenari: running git submodule update --init in /var/lib/l10nupdate/mediawiki/
• 03:56 Tim: testing new scheme for LU involving not pushing out LU files to all apaches
• 02:49 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 10 02:49:16 UTC 2012
• 01:53 brion: test.wikipedia.org fails with 'TrustedXFF: hosts file missing.'
• 01:44 Tim: on manganese: ran date -s "`date`" to make sure that isn't the cause of the high CPU usage during clone, it wasn't
• 01:44 Tim: running rebuildLocalisationCache.php for testwiki
• 01:43 brion: test.wikipedia.org broken again, "No localisation cache found for English."
• 00:51 brion: test.wikipedia is broken loading Cite.php

July 9

• 23:53 Tim: deleting and recreating php-1.20wmf7 to test a script
• 20:51 logmsgbot: reedy synchronized php-1.20wmf7/includes/media/FormatMetadata.php
• 20:51 logmsgbot: reedy synchronized php-1.20wmf6/includes/media/FormatMetadata.php
• 20:30 pp-pdf3: upgraded mwlib.epub
• 20:30 pp-pdf1: upgraded mwlib.epub
• 20:30 pp-pdf2: upgraded mwlib.epub
• 20:27 cmjohnson1: db10 swapping disk1 for new disk
• 19:41 logmsgbot: reedy synchronized php-1.20wmf6/includes/WikiPage.php
• 19:37 logmsgbot: reedy synchronized php-1.20wmf7/includes/WikiPage.php
• 18:13 logmsgbot: reedy synchronized php-1.20wmf7/extensions/CodeReview/ 'CR to trunk to fix fatals'
• 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.20wmf7
• 15:45 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf7.php
• 15:43 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
• 15:01 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
• 14:54 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
• 14:47 logmsgbot: reedy synchronized php-1.20wmf7
• 14:17 Reedy: Copying php-1.20wmf7 from /tmp to /h/w/c on fenari
• 13:26 Reedy: Running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf3"
• 13:23 Reedy: Running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf2"
• 13:13 Reedy: Removed wmf2 and wmf3 from bits docroots
• 13:11 mutante: powercycling and upgrading a couple more mw* servers
• 13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 13:01 mutante: labs.wikimedia.org is now a redirect to labsconsole
• 12:49 mutante: authdns-update to add labs.wm entry for redirect to labsconsole
• 08:34 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'adjust $wgCollectionFormats'
• 08:16 hashar: pallium, updating jenkins build script with gerrit 14666 & gerrit 14667
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 9 02:25:57 UTC 2012

July 8

• 12:17 apergos: removed HTCPpurger.log.1 and current log ater restart of purger on ms6, /was full. people reporting thumb issues from europe
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 8 02:26:21 UTC 2012

July 7

• 16:39 logmsgbot: reedy synchronized docroot/bits/DolphinBrowser/js/ 'Remove BOM'
• 16:10 logmsgbot: reedy synchronized php-1.20wmf6/extensions/cldr/ 'remove bom'
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 7 02:27:22 UTC 2012

July 6

• 21:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38227 - Please enable WikiLove on Turkish Wikipedia'
• 21:48 binasher: truncating pagetriage tables on enwiki (per bsitu)
• 21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38227 - Please enable WikiLove on Turkish Wikipedia'
• 21:46 Reedy: Created WikiLove tables on trwiki
• 21:39 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php
• 21:07 maplebed: removed a bit more load from the swift spinning media container servers by adjusting the ring weights
• 20:45 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'rev debug logging'
• 20:22 logmsgbot: aaron synchronized php-1.20wmf6/includes/Revision.php 'es debug logging'
• 20:20 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'es debug logging'
• 19:49 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
• 19:30 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
• 19:24 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
• 17:28 maplebed: adjusted swift ring files to move container listings off spinning disks (ms-be1-4)
• 17:04 logmsgbot: reedy synchronized wmf-config/
• 16:53 Reedy: Running updateCollation.php in foreachwiki on fenari in screen as reedy
• 16:45 Reedy: Running updateCollation.php against enwiki on fenari in screen as reedy
• 16:39 Reedy: Running updateCollation.php against commonswiki on fenari in screen as reedy
• 16:08 logmsgbot: reedy synchronized images/wiki-en.png
• 15:54 logmsgbot: reedy synchronized images/sul/meta.png
• 15:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 15:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 15:21 cmjohnson1: ms-be5 removing disk0 to replace with good disk
• 14:38 RobHalsell: authdns-update to correct typo in mgmt dns entry
• 13:22 mark: Inserted new pybal 1.04 package in the precise-wikimedia APT repository, and upgraded all precise LVS servers
• 11:31 logmsgbot: tstarling Finished syncing Wikimedia installation... :
• 11:25 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 11:20 Tim: running scap to delete docroot files per I48502d90 and Ie3afd137
• 08:58 mutante: dist-upgrading (unused) db10xx servers
• 07:43 Tim: also trusted-xff.phps
• 07:39 Tim: removing some untracked junk from /home/wikipedia/common
• 07:29 mutante: continue to restart and upgrade downed mw10xx servers
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 6 02:26:22 UTC 2012
• 01:35 binasher: updated squid redirector to cover wiki(quotes|books|versity)
• 00:13 Ryan_Lane: restarting gerrit
• 00:09 Ryan_Lane: force running puppet on manganese, it'll restart gerrit

July 5

• 23:54 Ryan_Lane: force running puppet on formey and manganese, since a config change is involved, it's going to restart
• 23:45 Ryan_Lane: restarting gerrit on manganese
• 23:38 Ryan_Lane: upgrading gerrit on formey
• 23:21 Ryan_Lane: updating database for gerrit
• 23:09 Ryan_Lane: upgrading gerrit on manganese
• 23:08 Ryan_Lane: stopping gerrit on manganese and disabling puppet
• 23:07 Ryan_Lane: stopping gerrit on formey and disabling puppet
• 22:47 logmsgbot: preilly synchronized wmf-config/mobile.php 'add subdomain check'
• 22:34 logmsgbot: reedy synchronized wmf-config/
• 22:17 logmsgbot: reedy synchronized wmf-config/ 'Tidying config for randomrootpage'
• 21:55 logmsgbot: reedy synchronized wmf-config/ 'Strike 2'
• 21:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'revert'
• 21:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Random root page enabled everywhere but wikipedias'
• 21:47 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:34 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:18 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Bumping AFTv5 lottery percentage to 20% of en.wiki'
• 21:15 LeslieCarr: powercycling unresponsive mw1116
• 21:09 maplebed: added new ms-be pmtpa hosts to DNS
• 20:03 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue with token'
• 19:14 Jeff_Green: *somebody* purged binary logs on blondel
• 18:58 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue'
• 18:54 mark: Upgraded pybal on all precise LVS servers
• 18:43 mark: Built new pybal_1.03 package and inserted it into the precise-wikimedia APT repository
• 17:41 mutante: powercycling db1048, mw1136, mw1046
• 17:32 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'reenabling db40'
• 17:30 mutante: more powercycling and upgrading: mw1036, mw1134, mw1043 ..
• 17:26 mutante: powercycling db1009,db1010
• 17:22 mutante: powercycling db1027,db1028
• 17:14 mutante: powercycling db1013
• 16:50 mutante: powercycling downed db1015
• 16:44 mutante: powercycling and upgrading more mw10xx servers, 1017,1023,1025 ...
• 16:27 mutante: mw1002, mw1007,mw1009,mw1011 - crashed,powercycling,dist-upgrading+kernel,reboot
• 16:18 mutante: powercycling mw1002
• 15:37 mutante: argon back up with new kernel,mysql,grub,.. looks happy afaict
• 15:33 mutante: argon (limesurvey) fscked, dist-upgrading
• 15:28 mutante: powercycling argon
• 15:25 hashar: updated Jenkins configuration on gallium : Updating f407ebe..4b669b9
• 14:08 cmjohnson1: HUME replacing disk 0
• 14:07 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'disabling parser cache for now'
• 11:54 mark: Inserted new pybal_1.02 package into APT distribution precise-wikimedia
• 11:17 mark: Installed new pybal snapshot build for testing on lvs1005
• 11:13 mutante: add missing Russian locales on singer, run localegen, run ru.planet update
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 5 02:26:45 UTC 2012

July 4

• 16:32 mutante: wikidata.org on now - redirect purged from squids
• 15:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'imports'
• 15:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable LQT on fiwikimedia'
• 15:39 Reedy: Killed php-1.20wmf5 localisation cache from mediawiki-installation group
• 15:37 Reedy: Created LQT tables on fiwikimedia
• 15:25 mutante: wikidata.org works now, besides old redirect may still be cached on cp* boxes (not purged by purgeList.php via multicast?). http://www.wikidata.org/?notcached
• 15:06 mutante: sync-common-file extract2.php, apache-graceful-all
• 15:04 logmsgbot: dzahn synchronized extract2.php
• 13:23 mutante: apache-graceful-all to add wikidata.org virtual host
• 13:21 mutante: svn commiting gerrit 9874, sync-apache
• 13:14 mutante: git pull in /h/w/common/docroot . adding wikidata.org files on fenari. , then "sync-docroot"
• 12:44 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 12:44 mutante: updating/syncing interwiki cache
• 09:32 hashar: swift-container-auditor seems to get down from time. Nagios reporting 0 processes at 8:15am and 9:25am UTC (I guess it get restarted automatically by puppet)
• 07:34 hashar: updating Jenkins copy of integration/jenkins from 0f069c3 to e264d1b. Bring new ant script + update to testswarm fetcher
• 07:09 Tim: on srv193: ran dpkg --set-selections to revert holds on php5 packages and ran apt-get upgrade
• 07:07 Tim: on srv193: fixing broken PHP packages causing puppet failure, nothing in the server admin log about them so I assume they were installed by accident
• 06:21 Tim: deployed Idb6d9a8b and restarting apaches
• 06:11 Tim: deployed Id7008681 and restarting apaches
• 05:45 Tim: reniced apache processes to level 0
• 05:04 Tim: deploying apache nice level change per RT #664
• 03:59 Tim: on mw1: experimenting with renice methods for RT 664
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 4 02:26:39 UTC 2012

July 3

• 23:03 logmsgbot: mlitn synchronized php-1.20wmf6/extensions/ArticleFeedbackv5/
• 22:36 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'weekly update'
• 22:27 hashar: updating testswarm submitter on gallium
• 21:01 logmsgbot: mlitn Finished syncing Wikimedia installation... :
• 20:37 logmsgbot: mlitn Started syncing Wikimedia installation... :
• 20:18 logmsgbot: asher synchronized wmf-config/db.php 'lowering db32 weight'
• 19:34 logmsgbot: asher synchronized wmf-config/db.php 'lowering db36 weight'
• 19:31 logmsgbot: asher synchronized wmf-config/db.php 're-add db36, db32 (low weight), es3 (innodb)'
• 18:11 RobH: virt1006 mgmt serial not set correctly, fixed
• 17:51 RobH: investigating stat1001 power issue
• 17:22 RobH: fluorine offlining to test disks
• 17:22 RobH: pulling helium offline for disk testing with fluorine disks
• 17:10 RobH: db1047 disk0 rebuild in progress
• 17:04 RobH: replacing bad disk in db1047
• 16:45 logmsgbot: reedy synchronized wmf-config/ 'Various config changes'
• 16:30 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.7.xsd
• 16:15 Jeff_Green: silicon gets dist-upgrade & reboot
• 16:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgCheckSerialized is deaded'
• 03:53 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable API action=purge on commonswiki'
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jul 3 02:27:02 UTC 2012

July 2

• 22:33 binasher: rebooted db36 for kernel upgrade
• 22:25 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db36'
• 22:02 brion: fun with routers in tampa, wikis down
• 21:48 maplebed: rebooted emery - it's been unresponsive for 3 days.
• 18:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 18:30 hashar: set up ignore file in httpd configuration directory
• 18:23 logmsgbot: reedy synchronized wmf-config/ 'Enable WikimediaShopLink on enwiki'
• 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 273 wikipedias to 1.20wmf6
• 18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf6
• 15:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '/etc/wikimedia-realm detection https://gerrit.wikimedia.org/r/13888'
• 15:18 logmsgbot: hashar synchronized docroot/bits/static-master '(bug 37245) docroot 'static-master' for beta bits'
• 15:04 mutante: authdns-update to switch jobs.wm redirect to wikimedia-lb to fix SSL cert mismatch (RT-3071)
• 14:55 mark: Reboot of cr1-sdtpa did not fix the RE packet loss issue... therefore unlikely to be leap second related
• 14:41 mark: Rebooting cr1-sdtpa
• 14:37 mark: Shutdown PyBal BGP sessions on cr1-sdtpa
• 14:34 mark: Shutdown BGP session to 2828 on cr1-sdtpa
• 13:36 hashar: db12 suffering some 1400sec (and growing) replag. mysqldump in progress on that host.
• 12:35 mutante: installing upgrades on fenari (linux-firmware linux-libc-dev..)
• 12:27 mutante: rebooting gallium one more time to install kernel
• 12:26 mutante: upgrading kernel on gallium
• 12:23 logmsgbot: hashar synchronized live-1.5/CREDITS
• 11:31 mark: Now we have packet loss within pmtpa/sdtpa... reverting change
• 10:57 mark: Problems on one of two pmtpa-eqiad waves; raised OSPF metric to 60 to failover traffic to the other link
• 10:50 Tim: fixing leap second issue on bastion1 by rebooting it
• 10:47 Tim: fixed leap second issue on bastion-restricted
• 09:56 Tim: fixing leap second issue on virt1,virt2,virt3,virt4,virt5
• 09:52 Tim: fixing leap second issue on aluminium,gallium,manganese
• 09:47 Tim: fixing leap second issue on formey,grosley,hooper,sanger,sockpuppet
• 09:43 Tim: on fenari: fixed leap second issue with the mozilla method
• 09:39 apergos: rebooting gallium, it's pretty unhappy (maybe related to leap second issue)
• 08:14 logmsgbot: hashar: srv190 srv266 srv281 timeouts on sync-file
• 08:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Bug 37457 - fix import sources for viwikibooks'
• 08:11 hashar: Stopped Jenkins on gallium. It is not doing anything anyway. Asked to reboot box RT #3208
• 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jul 2 02:53:51 UTC 2012
• 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 2 02:28:48 UTC 2012
• 01:48 Tim: kill -CONT on populateRevisionSha1.php processes
• 00:47 Tim: on nfs1: trying leap second fix suggested at https://bugzilla.mozilla.org/show_bug.cgi?id=769972#c5
• 00:26 logmsgbot: tstarling synchronized wmf-config/db.php 'reduce db32 read load to zero due to persistent lag'
• 00:12 Tim: switched enwiki back to r/w
• 00:12 logmsgbot: tstarling synchronized wmf-config/db.php
• 00:06 Tim: on hume: stopped all populateRevisionSha1.php processes with kill -STOP
• 00:03 logmsgbot: reedy synchronized wmf-config/db.php 's1/enwiki into readonly'

July 1

• 19:12 logmsgbot: reedy synchronized php-1.20wmf6/extensions/WikimediaMaintenance/ 'Update to master for hashar'
• 17:55 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
• 17:45 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
• 17:43 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
• 17:32 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
• 17:30 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
• 16:53 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
• 16:48 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'logging'
• 12:54 notpeter: also going to reboot all pmtpa search nodes. not in prod, but are still freaking out from leap second bug.
• 05:33 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'logging'
• 04:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jul 1 04:25:25 UTC 2012
• 04:06 Ryan_Lane: virt1000 is back up, rebooting virt0
• 04:01 Ryan_Lane: rebooting virt1000
• 03:16 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 1 03:16:39 UTC 2012
• 01:43 notpeter: that worked. restarting all remaining search nodes.
• 01:39 notpeter: problem with lucene persisting through service restart, but not node restart. restarting en pool nodes.
• 01:20 paravoid: restarting opendj (nfs1/nfs2), load spike, possibly related to leap second
• 00:51 notpeter: search1004 dead. powercycling.
• 00:50 notpeter: based on ganglia evidence, lucene seems to have been affected by leap second bug. restartig each instance, one minute wait in between

June 30

• 16:12 mark: Temporarily added path 6939+ 14907+ to AVOID-PATHs on cr2-knams
• 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 30 02:53:46 UTC 2012
• 02:28 maplebed: corrected LVS pdns_recursor config error causing DNS queries to fail on LVS servers in gerrit r13554 and r13555.
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jun 30 02:27:08 UTC 2012

June 29

• 19:49 hashar: restarting Jenkins to fix an issue with "parameterized builds" plugin. Updated git plugin as well.
• 19:35 RobH: dns update via authdns-update for vanadium ip
• 18:05 Jeff_Green: sync-apache and apache-graceful-all for http://donate.wikimedia.org-->https redirect
• 16:02 RobH: ms-be1001 and ms-be1002 powering down for ssd installation
• 15:59 RobH: authdns-update run
• 15:47 RobH: updating dns
• 15:16 mutante: dist-upgrading srv280,srv270,srv264
• 15:11 Jeff_Green: apache-graceful-all for redirect conf change
• 15:10 Jeff_Green: sync-apache to push out new foundation.conf
• 14:50 mark: Reinstalled chromium with precise
• 13:46 hashar: fixed interwiki on http://wikisource.org/ main page by hacking a script in production and refreshing cache
• 13:46 logmsgbot: hashar synchronized php-1.20wmf6/cache/interwiki.cdb 'Updating interwiki cache for 1.20wmf6'
• 13:32 logmsgbot: hashar synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 12:38 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 12:38 mutante: dumping interwiki and updating interwiki cache (to fix broken interwiki links, like wikisource.org -> wikipedia.org)
• 09:31 hashar: Jenkins: deployed gitsqlhaschanged patch ( d04f779 0f069c3 integration/jenkins.git )
• 07:56 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'send header from CS.php only for non CLI scripts gerrit 13435'
• 07:08 mutante: upgrading apt packages on brewster
• 02:51 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 29 02:51:13 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jun 29 02:26:25 UTC 2012

June 28

• 23:30 logmsgbot: reedy synchronized php-1.20wmf6/includes/resourceloader/ResourceLoader.php
• 23:15 binasher: completed aft offload_large_feedback migration on enwiki
• 23:03 logmsgbot: asher synchronized wmf-config/db.php 'returning db36'
• 21:50 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:39 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:13 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db36'
• 20:38 binasher: ran aftv5 offload_large_feedback migrations on testwiki and en_labswikimedia
• 20:14 RobH: dns update for pc1-pc3
• 19:54 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 19:08 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 18:47 hashar: the internal change to CommonSettings.php caused a lack of stylesheet for less than a minute on most wikis. I did test on test.wikipedia.org and beta project, but there must be a logic error somewhere that mess with the prod projects. Revert changes have been sent out in gerrit and merged in master.
• 18:35 hashar: so the nicely reviewed changes broke the enwiki stylesheets :/ reverted change :-(((
• 18:34 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
• 18:33 hashar: srv190 and srv281 got ssh timeout
• 18:31 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
• 18:30 hashar: did various tests using eval.php. Most important is $realm -> production. $cluster -> pmtpa. Syncing
• 18:25 hashar: updating mediawiki-config to grab a12545d edceb4c & eee97ad
• 18:23 RobHalsell: swapped bad psu out of ms1001-array3, redundant so no downtime
• 15:40 RobHalsell: pulling the following servers, relocating to payments rack: payments1001-1004, boron, beryllium, lithium
• 15:34 RobHalsell: dns updated
• 15:31 RobHalsell: boron appears to be unallocated, pulling IP allocation, rack allocation, moving to payments per 1227
• 14:40 mutante: svn server is rebooting.brb
• 14:38 mutante: dist-upgrading formey (svn/gerrit), rebooting soon
• 14:38 Jeff_Green: manganese rebooted for kernel update
• 14:37 RobH: allocating yttrium to payments rack per rt 1227
• 14:24 Jeff_Green: manganese dist-upgrade
• 14:15 Ryan_Lane: restarting apache on manganese
• 14:15 Ryan_Lane: restarting gerrit
• 05:08 Tim: srv266 was flooding the fatal error log, complaining about a missing file. Killed apache and ran sync-common.
• 05:03 Tim: fixed fatal.log on fenari, socat was writing to a deleted file
• 02:58 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Thu Jun 28 02:58:42 UTC 2012
• 02:30 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jun 28 02:30:03 UTC 2012

June 27

• 23:50 K4-7131: sync'd payments cluster to 592e0a5ba195
• 23:43 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add rule for mediawiki'
• 22:37 K4-7131: sync'd payments cluster to 7e9072c2d571c
• 22:23 binasher: temporarily pulling srv211 from pybal
• 21:56 RobH: mw1102 has no nic0, rather than troubleshoot it for a long time, reinstall! (rt 3058)
• 21:01 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix CSRF'
• 21:00 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'fix CSRF'
• 20:55 RobH: db1003 back online, replaced mgmt cable and mgmt is working now as well
• 20:51 LeslieCarr: rebooting srv266 as it is unresponsive
• 20:44 RobH: db1003 mgmt issue due to bad cable, system booting back up, replacing mgmt cable
• 20:35 RobH: clean mysql shutdown, db1003 now offline
• 20:33 RobH: db1003 mgmt is not responsible, I need to remove power and reboot. confirmed iwth asher this is an s3 slave and can do a short downtime without issues
• 20:31 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'MF fixes and logging'
• 20:29 logmsgbot: maxsem synchronized php-1.20wmf6/extensions/MobileFrontend/ 'MF fixes and logging'
• 19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'disable LastModified and LastModified/E3Experiment'
• 19:54 logmsgbot: reedy synchronized php-1.20wmf6/maintenance/runJobs.php
• 19:53 logmsgbot: reedy synchronized php-1.20wmf5/maintenance/runJobs.php
• 19:32 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/specials/SpecialLanguageStats.php
• 19:19 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/
• 19:06 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/
• 19:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: meta back to wmf6, not cause of translate issues
• 18:53 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed to 1.20wmf6
• 18:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia wikis to 1.20wmf6
• 18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiversity to 1.20wmf6
• 18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.20wmf6
• 18:47 Jeff_Green: added several mobile hostnames to DNS for RT #2996
• 18:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf6
• 18:44 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved metawiki back to 1.20wmf5
• 18:41 K4-713: synchronized payments cluster to fundraising/1.20 de0256084a
• 18:33 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special wikis to php-1.20wmf6
• 18:29 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved en(wikibooks|wikinews|wikiquote|wikisource|wikiversity|wiktionary) to 1.20wmf6
• 18:05 RobH: cp1017 memory replaced
• 17:52 RobH: cp1017 is offline due to memory error. replacement memory on site, pulling system for swap
• 17:46 logmsgbot: preilly synchronized php-1.20wmf6/extensions/ZeroRatedMobileAccess 'update for landing page'
• 17:46 logmsgbot: asher synchronized wmf-config/mc.php 'disabling wgMemCachedPersistent; lowering wgMemCachedTimeout to 2x client default from 30x default'
• 17:45 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
• 17:44 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
• 17:19 LeslieCarr: restarting apache2 on srv258
• 17:11 maplebed: powercycled srv270
• 17:11 mutante: powercycling srv277 (had to, frozen console)
• 17:06 LeslieCarr: rebooting srv287
• 17:05 Ryan_Lane: rebooting srv280
• 17:03 mutante: powercycling srv280
• 17:01 paravoid: rebooting srv264, swapdeath
• 16:52 mark: Rebooting srv279, swapdeath
• 16:52 paravoid: rebooting srv275, swapdeath
• 16:50 paravoid: rebooting srv258, swapdeath
• 16:43 logmsgbot: reedy synchronized php-1.20wmf5/extensions/ArticleFeedbackv5/
• 15:08 Reedy: ExtensionDistributor now works from git on mediawiki.org
• 15:08 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/
• 15:06 logmsgbot: reedy synchronized php-1.20wmf5/extensions/ExtensionDistributor/
• 15:01 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/
• 14:51 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/ 'ED to trunk'
• 14:46 mark: Rebooting lvs1005 (after dist-upgrade)
• 13:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikimediaShopLink
• 13:24 mark: Added IPv6 LVS service IPs to the LVS_import policy on cr2-eqiad, for testing with lvs1005
• 13:22 Tim: installing apache2.2-bin-dbgsym on mw1
• 13:04 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for WikimediaShopLink
• 13:04 mark: Started PyBal 1.02 snapshot build on lvs1005
• 12:39 Reedy: WikimediaShopLink is deployed to testwiki/test2wiki
• 11:59 apergos: kicked morebots
• 13.56.44 (CEST) <logmsgbot> !log reedy synchronized wmf-config/ 'WikimediaShopLink'
• 13.54.28 (CEST) <logmsgbot> !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/
• 13.44.37 (CEST) <logmsgbot> !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/
• 11.36.23 (CEST) <mutante> !log starting swift-container-auditor on ms-be3
• 10.29.51 (CEST) <mutante> !log apt-get upgrade on gallium, installs newer jenkins
• 10.26.41 (CEST) <mutante> !log importing jenkins_1.472_all.deb into lucid-wikimedia using reprepro
• 08.04.26 (CEST) <logmsgbot> !log tstarling synchronized wmf-config/CommonSettings.php
• 08.00.45 (CEST) <logmsgbot> !log tstarling synchronized wmf-config/CommonSettings.php
• 04.48.48 (CEST) <logmsgbot> !log LocalisationUpdate completed (1.20wmf6) at Wed Jun 27 02:48:51 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Wed Jun 27 02:24:59 UTC 2012
• 01:23 Tim: on manganese: restarting gerrit
• 01:02 Tim: on manganese: killing all gitweb.cgi processes

June 26

• 23:49 Tim: on fenari: doing git and 1.19 checkouts for ExtensionDistributor
• 22:17 JeLuF: Slowly starting to import 100,000 images from the Deutsche Fotothek into Commons using importImages.php on fenari as user jeluf.
• 14:07 mutante: shutting down unused cp1037-cp1040 per RT-3189
• 10:48 mark: Moving all API traffic back to API apaches
• 10:41 mark: Restarted gmetad on nickel
• 10:36 apergos: powercycling srv261
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jun 26 02:47:56 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Tue Jun 26 02:25:02 UTC 2012
• 01:05 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix css'
• 01:04 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'fix css'
• 01:03 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'fix css'
• 00:59 LeslieCarr: ignoring cp1037 to cp1040 alarms for now as they are unused
• 00:45 LeslieCarr: rebooting cp1040
• 00:45 LeslieCarr: rebooting cp1039
• 00:43 LeslieCarr: rebooting cp1037

June 25

• 22:49 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Weekly MF deployment'
• 22:40 logmsgbot: maxsem synchronized php-1.20wmf6/extensions/MobileFrontend/ 'Weekly MF deployment'
• 22:36 maplebed: powercycling ms1002 - it's unresponsive to ssh and on the console though it does respond to a ping.
• 21:31 Jeff_Green: manual apache restart on srv265, srv277
• 21:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37345 - Request: Enable Ext:Collection on mk.wiki'
• 21:07 LeslieCarr: rebooting neon
• 21:01 hashar: Triggered several jobs on Jenkins to run tests on change that did not received their blame stick token
• 21:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37507 - Babel configuration for tl.wikipedia'
• 20:58 Jeff_Green: pushing out new redirects.conf adjusted for RT #3138
• 20:52 logmsgbot: reedy synchronized wmf-config/ 'Various site config bugs'
• 20:13 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Math/ 'Updating math to master'
• 20:12 logmsgbot: reedy synchronized php-1.20wmf5/extensions/Math/ 'Updating math to master'
• 20:07 logmsgbot: reedy synchronized php-1.20wmf6/extensions/WikiEditor/
• 20:06 logmsgbot: reedy synchronized php-1.20wmf5/extensions/WikiEditor/
• 19:17 LeslieCarr: rebooting unresponsive gallium
• 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf6
• 17:27 logmsgbot: reedy Finished syncing Wikimedia installation... : Take 2
• 16:43 logmsgbot: reedy Started syncing Wikimedia installation... : Take 2
• 16:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable EducationProgram on enwiki per request'
• 16:30 logmsgbot: reedy synchronized php-1.20wmf6/extensions/EducationProgram/ 'sync education programf iles'
• 16:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Scapping to rebuild message cache for 1.20wmf6
• 16:08 logmsgbot: reedy Started syncing Wikimedia installation... : Scapping to rebuild message cache for 1.20wmf6
• 16:03 logmsgbot: reedy synchronized php-1.20wmf6 'Syncing php-1.20wmf6'
• 15:43 Reedy: Copying php-1.20wmf6 from /tmp to NFS /home on fenari
• 13:52 Reedy: Killed php-1.20wmf4/cache/l10n from mediawiki-installation hosts
• 12:26 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37699) Chage logo on uzwiki'
• 09:38 mutante: so the several redirects for education->outreach requested to work by today look good now. RT-3138
• 09:30 mutante: apache-graceful-all to push out needed redirects for education
• 09:23 mutante: looking good. running sync-apache
• 09:20 mutante: creating dsh group "testwikipedia" with just srv193, creating sync-apache-test to just sync there...testing sync
• 02:49 Tim: configured mediawiki-commits to discard mails from gerrit pending resolution of the "implicit destination" issue
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jun 25 02:24:46 UTC 2012

June 24

• 18:21 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jun 24 02:25:23 UTC 2012

June 23

• 13:28 apergos: powrcycling srv288, swap death etc, some message to mgmt console but only the timestamp so couldn't see the issue, also couldn't get past the login prompt
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 23 02:24:11 UTC 2012

June 22

• 23:11 LeslieCarr: restarted apache on srv278
• 22:23 binasher: stopping mysql on es3, reseeding slave via innodb hotbackup of es1004
• 18:57 logmsgbot: preilly synchronized docroot/bits
• 18:36 LeslieCarr: removing 28790 bounce messages from exim queue on mchenry
• 16:50 Ryan_Lane: added a database account on db9/10 for read-only access to the gerrit database
• 12:06 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37700) update stewardwiki logo & favicon'
• 11:44 mutante: installing upgrades and kernel on pdf1, can reboot? (also needs puppetizing and precise reinstall)
• 10:46 mutante: installing security upgrades and kernel on bast1001 (still needs reboot, but dont break user sessions)
• 10:42 mutante: fenari upgrade - this included replace wikimedia-lvs-realserver 0.04 (using .../wikimedia-lvs-realserver_0.08
• 10:41 mutante: installing security upgrades on fenari
• 10:31 mutante: installing security upgrades on formey (gerrit)
• 08:49 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'gerrit 12569 Load transcode conf on -e /etc/wikimedia-transcoding (wmflabs change)'
• 08:40 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'gerrit 12568 Disable wgNoticeInfrastructure on beta cluster'
• 08:30 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'gerrit 12566 labs use the same wgCentralDBname on all wiki'
• 07:59 apergos: powercycled lvs1001, not pingable, nothing good from mgmt console, etc.
• 04:11 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 22 02:24:15 UTC 2012
• 01:00 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor 'VisualEditor updates'
• 00:37 LeslieCarr: restarting exim4 on mchenry with split_spool_directory = true
• 00:30 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Updated swift user config.'

June 21

• 23:11 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
• 23:07 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/VisualEditor.php
• 22:49 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
• 22:26 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
• 21:28 notpeter: restarting all lucene instances to direct logs to oxygen
• 21:16 binasher: deploying new mobile redirector to esams text squids
• 21:00 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor 'VisualEditor bugfixes'
• 20:34 logmsgbot: reedy synchronized wmf-config/
• 20:29 binasher: deployed new squid mobile redirector, now covers additional projects
• 20:27 logmsgbot: reedy synchronized wmf-config/
• 20:10 logmsgbot: hashar: on gallium, cloning mediawiki/extensions.git to /var/lib/jenkins/jobs/MediaWiki-Extensions-Fetching/workspace
• 19:25 mark: Restarted 3 queue runners as exim -qff &
• 19:15 logmsgbot: catrope Finished syncing Wikimedia installation... : VisualEditor updates
• 18:57 logmsgbot: catrope Started syncing Wikimedia installation... : VisualEditor updates
• 18:50 mark: Started 5 exim queue runners on mchenry with exim -qqff &
• 18:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Point $wgVisualEditorParsoidURL to cadmium'
• 18:36 logmsgbot: catrope synchronized php-1.20wmf5/includes/OutputPage.php 'Core patches for VisualEditor deploy'
• 18:36 logmsgbot: catrope synchronized php-1.20wmf5/resources/Resources.php 'Core patches for VisualEditor deploy'
• 18:36 logmsgbot: catrope synchronized php-1.20wmf5/resources/mediawiki.page/mediawiki.page.watch.ajax.js 'Core patches for VisualEditor deploy'
• 17:51 notpeter: restarting puppet on brewster
• 15:25 notpeter: stopping puppet on brewster
• 14:11 paravoid: powercycling srv272, unreachable due to load spike
• 05:30 binasher: clearing mobile varnish cache - my friend can't expand some article categories on his iphone after rebooting and clearing cache
• 04:41 logmsgbot: catrope synchronized php-1.20wmf4/extensions/LastModified/modules/lastmodified.js
• 04:40 logmsgbot: catrope synchronized php-1.20wmf5/extensions/LastModified/modules/lastmodified.js
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Thu Jun 21 02:25:40 UTC 2012
• 00:10 binasher: stopped puppet on cp1020 until tomorrow - testing new build of the squid mobile redirector on one server until tomorrow

June 20

• 23:29 logmsgbot: kaldari synchronized php-1.20wmf5/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'updating clicktracking for LastModified and E3Experiments exts'
• 21:52 Reedy: pointed /usr/local/apache/common/php at /usr/local/apache/common/php-1.20wmf5 on mediawiki-installation
• 21:49 LeslieCarr: see RT3170 for more details on above change and mchenry pain
• 21:48 LeslieCarr: freezing many bounce messages on mchenry (all older than 2400 minutes)
• 21:04 LeslieCarr: replaced srv268 with srv245 in memcached list
• 21:04 logmsgbot: lcarr synchronized wmf-config/mc.php 'removed broken srv268'
• 21:03 paravoid: powercycling srv268; unreachable due to load spike
• 18:31 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 18:23 LeslieCarr: reloading mr1-pmtpa for sw upgrade (fixing a cpu bug)
• 18:18 logmsgbot: reedy synchronized wikiversions.dat
• 18:17 logmsgbot: reedy synchronized wikiversions.cdb
• 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of pedias to 1.20wmf5
• 18:03 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'test with disable caching on'
• 18:02 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'test with disable caching on'
• 17:59 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'testing with disable caching off'
• 17:58 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'testing with disable caching off'
• 17:42 logmsgbot: preilly synchronized docroot/bits
• 17:11 logmsgbot: preilly synchronized wmf-config/mobile.php 'add Grameenphone Bangladesh'
• 17:08 logmsgbot: preilly synchronized wmf-config/mobile.php 'add telenor'
• 14:00 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
• 13:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on enwiki *gulp*'
• 13:41 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/ 'Push out master EP'
• 13:37 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/ 'Push out master EP'
• 13:19 Reedy: Created EducationProgram database tables on enwiki
• 12:40 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touching InitialiseSettings.php to refresh cache'
• 12:39 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 37740) raise account throttle for an edit marathon'
• 09:43 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37457) viwikibooks can import from fr/it wikibooks'
• 09:40 logmsgbot: hashar synchronized wmf-config/mobile.php '$wgMobileResourceVersion does not exist anymore'
• 09:25 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37327) Configure chr.wikipedia site logo'
• 04:35 Tim: on nickel: there were data sources for both "Apaches 8 CPU" and "Application servers", these were getting the same cluster name from the remote gmonds, and so different threads in gmetad were trying to write to the same summary files. Fixed temporarily, will fix in puppet shortly
• 04:26 Tim: on nickel: ran gmetad with -d3, it spews errors when trying to write to the faulty summary info files
• 04:20 Tim: on nickel: restarting gmetad
• 04:19 Tim: on srv258: started gmond
• 04:12 Tim: experimentally stopping gmond on srv258 to check for effects on oscillating appserver stats
• 03:23 Tim: on fenari, queueing refreshLinks jobs for some 2.8M commons image description pages that use location templates
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Wed Jun 20 02:48:41 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Wed Jun 20 02:26:26 UTC 2012
• 02:24 Tim: started socat for /var/log/mw/fatal.log on fenari
• 01:23 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php

June 19

• 23:37 paravoid: temporarily adding wikimedia.org, wikipedia.org etc. to sodium's /etc/exim4/defer_domains
• 23:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'bug 37611, plan B'
• 22:55 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'bug 37611'
• 21:07 Jeff_Green: deployed a hacked up exim conf on sodium to block a mail ddos, puppet disabled there too
• 20:37 logmsgbot: mlitn synchronized php-1.20wmf5/extensions/ArticleFeedbackv5 'desc'
• 19:45 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
• 19:28 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
• 19:13 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Enable AFTv4 on testwiki'
• 18:06 maplebed: failed out ms-be5 after failed ssd test
• 14:50 RobH: updating dns
• 14:40 RobH: db14 is out of rotation, shutting down to make room for new es servers in rack
• 14:03 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Syncing misc changes
• 11:03 Ryan_Lane: adding IPs for virt6-8
• 10:11 logmsgbot: nikerabbit Finished syncing Wikimedia installation... : Updating TranslationNotifications
• 10:05 hashar: TranslationNotifications extension updated by Nikerabbit!
• 09:56 logmsgbot: nikerabbit Started syncing Wikimedia installation... : Updating TranslationNotifications
• 09:41 hashar: updating TranslationNotifications extension with NikeRabbit
• 09:04 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Merge I8572d5f4 to fix workflowstates in Translate'
• 07:27 apergos: reboot snapshot1, package and kernel updates
• 07:14 apergos: reboot snapshot2, package and kernel updates
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Tue Jun 19 02:47:42 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Tue Jun 19 02:24:46 UTC 2012
• 01:34 logmsgbot: tstarling Finished syncing Wikimedia installation... :
• 00:53 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 00:43 Tim: put DolphinBrowser files in docroot/bits (from preilly's Ie3fefec6) and now running scap

June 18

• 22:31 K4-713: updated production civicrm to r1814
• 22:26 logmsgbot: aaron synchronized php-1.20wmf5/extensions/FlaggedRevs 'deployed e53310f548cf3f3e4f1ddfa10f5efd0eff06eeec'
• 22:17 maplebed: rebooting es1002 to look at the raid setup
• 20:43 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'update to remove bad code'
• 20:43 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'update to remove bad code'
• 19:11 hashar: updating several Jenkins plugins
• 19:06 RobH: updating dns
• 18:52 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/
• 18:51 logmsgbot: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/
• 18:23 logmsgbot: reedy synchronized wikiversions.dat
• 18:15 logmsgbot: reedy synchronized wikiversions.cdb 'sync using sync-file'
• 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf5
• 16:30 mutante: installing package upgrades on sodium
• 16:29 mutante: restarting lighttpd on sodium - redirecting mediawiki-cvs list page
• 16:05 binasher: rebooting es1001
• 15:59 mutante: there have been no archives, so that should be it. there may be another issue in BZ 37690 but should be unchanged by renaming
• 15:58 mutante: copied full config/users/passes from mediawiki-cvs to mediawiki-commits, merged redirects, added old list name to acceptable_aliases in recipient filters
• 15:52 mutante: making the mailing list switch. mediawiki-cvs -> mediawiki-commits
• 15:49 Ryan_Lane: assigned service IPs for labs-ns0/labs-ns1
• 15:25 binasher: rebooting es1002 and es1003
• 15:11 Ryan_Lane: added virt1000 as a secondary ldap server for labsconsole
• 15:08 Ryan_Lane: testing gerrit config with multiple ldap servers
• 14:52 hashar: hume is out of disk space again. Probably the wmf branches taking toooo much space
• 14:52 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37662) change wgUploadNavigationUrl @ dawiki'
• 14:21 mutante: creating new list MediaWiki-commits, not in use yet, but will replace outdated -cvs list soon
• 14:18 apergos: reboot snapshot3, package and kerne updates
• 14:13 apergos: rebooting snapshot4, kernel and other updates
• 09:31 Ryan_Lane: added virt1000 to dns, using titanium misc server
• 08:17 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php ' (bug 37672) Use odf on collection for ml projects '
• 06:03 apergos: powercycling db1047
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Mon Jun 18 02:47:49 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jun 18 02:25:09 UTC 2012

June 17

• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sun Jun 17 02:45:15 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jun 17 02:23:01 UTC 2012

June 16

• 02:49 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sat Jun 16 02:49:55 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 16 02:24:56 UTC 2012
• 00:34 paravoid: esams SSL should be back up
• 00:25 paravoid: SSL ipv6 access logs disabled; force running puppet and rm'ing access.logs on esams
• 00:08 paravoid: esams SSL outage, working on it

June 15

• 20:56 LeslieCarr: attaching asw-c1-pmtpa to asw-d-pmtpa ring
• 20:41 RobHalsell: updating dns for mc1-mc16 mgmt
• 20:07 RobH: mobing asw and msw-d3-sdtpa from single to dual power again, got sidetracked
• 19:59 RobH: mobing asw and msw-d3-sdtpa from single to dual power
• 19:24 RobH: updating dns for ms-be12 mgmt
• 19:18 logmsgbot: aaron synchronized php-1.20wmf5 'deployed 2755f255e45b53a083207d69c3e2d9fca62a3a1c'
• 19:15 paravoid: virt0: modify pdns.conf to listen on the old IP; temporarily disable puppet
• 19:15 paravoid: adding pre-renumbering virt0's IP back on eth1; doing policy routing to work out multihoming
• 18:54 RobH: updating dns for educacao redirect
• 18:53 LeslieCarr: deactivated rpf-filter on cr1-sdtpa and cr2-pmtpa temporarily for virt0
• 18:53 Ryan_Lane: doing a git pull for OpenStackManager on virt0
• 18:33 RobH: morebots, dont leave me again!
• 15:50 RobH: updating dns for new cisco machines
• 14:34 hashar: hume: 5.0G 5.0G 68K 100% /usr/local/apache
• 14:33 hashar: hume is out of disk space
• 14:32 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 34866) Change wgLanguageCode of several wikis to be renamed'
• 14:21 logmsgbot: hashar synchronized phpunit.xml
• 14:09 logmsgbot: hashar synchronized tests
• 13:39 Ryan_Lane: adding labs-ns0 and labs-ns1 dns entries
• 11:46 mark: csw1-esams.wikimedia.org line card 2 in trouble, power cycled it
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Fri Jun 15 02:48:46 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 15 02:26:17 UTC 2012
• 00:00 binasher: rebooting / upgrading kernel on es1003 first

June 14

• 23:58 binasher: stopping mysql on es1003 and disabled notifications. going to convert to innodb via hotbackup of es1004 for testing
• 23:23 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning LastModified on for en.wiki'
• 23:10 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 23:00 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 22:36 logmsgbot: kaldari synchronized php-1.20wmf5/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'sycing E3Experiment js for wmf5'
• 22:34 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 22:07 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:57 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:16 RobH: updating dns for new mgmt ips and move of scs
• 20:37 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 20:33 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 20:29 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update to MoodBar
• 20:07 logmsgbot: bsitu Started syncing Wikimedia installation... : Update to MoodBar
• 19:51 logmsgbot: py synchronized wmf-config/db.php 're-add db43 to s6 pool after kern upgrade'
• 19:28 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update to PageTriage
• 19:10 notpeter: rebooting db43 for kernel upgrading
• 19:08 logmsgbot: asher synchronized wmf-config/db.php 'returning db22'
• 18:58 logmsgbot: bsitu Started syncing Wikimedia installation... : Update to PageTriage
• 18:52 RobH: unracking and decommissioning db21 and db23
• 18:46 RobH: db22 relocated, powering up
• 18:43 RobH: db22 relocating
• 18:41 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db22 for hw move'
• 18:39 pgehres: re-enabled donation queue consumption after all updates
• 18:36 notpeter: pushing new dns zone file (minor change)
• 18:33 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
• 18:33 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
• 18:32 notpeter: new master log and pos for s6 MASTER_LOG_FILE='db47-bin.000230', MASTER_LOG_POS=876357616
• 18:27 logmsgbot: py synchronized wmf-config/db.php 'completed master switch for s6'
• 18:24 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
• 18:23 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
• 18:23 logmsgbot: py synchronized wmf-config/db.php 'switching master for s6 to db50'
• 18:11 Jeff_Green: erzurumi dist-upgrade & reboot [up 633 days]
• 18:02 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'update for ie7'
• 18:01 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'update for ie7'
• 18:00 Jeff_Green: aluminium/db1008 dist-upgrade & reboot
• 17:57 pgehres: disabled queue consumption on aluminum for dist-upgrade
• 17:03 mark: Copied udp-filter package from lucid-wikimedia to precise-wikimedia (but do as I say and rebuild, not as I do...)
• 16:40 binasher: running pagetriage_page schemea changes on enwiki and testwiki via osc (https://gerrit.wikimedia.org/r/#/c/11014/1/sql/PageTriagePagePatch.sql)
• 16:22 Jeff_Green: hume dist-upgrade & reboot
• 16:08 Jeff_Green: loudon dist-upgrade & reboot
• 16:04 mark: Manually disabled GRO on amslvs3/4 eth0
• 16:03 logmsgbot: reedy synchronized wmf-config/ 'Enable EP on test2wiki'
• 16:01 Ryan_Lane: restarting nova-compute on virt5
• 15:58 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
• 15:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisationcache for EP
• 15:49 Jeff_Green: grosley dist-upgrade & reboot
• 15:40 Jeff_Green: silicon dist-upgrade and reboot
• 15:29 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisationcache for EP
• 15:28 mark: Starting dist-upgrade of manutius to Precise
• 15:27 Ryan_Lane: to satisfy mark's pedantry that's lucid-wikimedia
• 15:27 mark: Ryan needs coffee
• 15:26 Ryan_Lane: specifically wikimedia-lucid repo
• 15:26 Ryan_Lane: added adminbot 1.2 to repo
• 15:00 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
• 14:30 mark: Reinstalled stat1 with Ubuntu Precise
• 11:21 pp-pdf1: updated mwlib.rl to 0.12.12
• 11:21 pp-pdf2: updated mwlib.rl to 0.12.12
• 11:21 pp-pdf3: updated mwlib.rl to 0.12.12
• 02:46 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Thu Jun 14 02:46:00 UTC 2012
• 02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Thu Jun 14 02:23:26 UTC 2012
• 00:01 logmsgbot_: tstarling synchronized php-1.20wmf4/includes/DefaultSettings.php

June 13

• 23:39 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Deploying MF fix'
• 23:37 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Deploying MF fix'
• 21:49 logmsgbot_: reedy synchronized wmf-config/ 'More changes from gerrit'
• 21:35 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bringing in numerous changes merged via gerrit'
• 21:32 logmsgbot_: lcarr synchronized wmf-config/mc.php 'replacing broken srv203 with working srv250'
• 21:31 LeslieCarr: replacing srv203 with srv250 in memcache rotation since srv203 is broken
• 21:20 logmsgbot_: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'deployed 82742bccf3b5f2da0d5df05630eb31978afbbce1'
• 19:57 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php
• 19:51 Jeff_Green: payments cluster dist-upgrades & reboots
• 19:43 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Added debug log.'
• 19:37 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php
• 19:29 binasher: drop table enwiki.trackbacks
• 19:28 binasher: converted enwiki.interwiki to innodb
• 19:27 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php 'temporary logging code.'
• 19:26 binasher: drop table enwiki.exlogging
• 18:35 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non wikipedia to 1.20wmf5
• 18:33 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote to 1.20wmf5
• 18:31 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.20wmf5
• 18:30 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity to 1.20wmf5
• 18:22 logmsgbot_: reedy synchronized php-1.20wmf5/extensions/Vector/
• 18:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special.dblist wikis to 1.20wmf5
• 18:14 logmsgbot_: aaron synchronized php-1.20wmf5/extensions/FlaggedRevs 'deployed 537bb248bb93948844f195014227512f169a439b'
• 18:06 Jeff_Green: db1025 dist-upgrade & reboot
• 17:37 logmsgbot_: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'changes for zero needed for carrier testing'
• 17:36 logmsgbot_: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'changes for zero needed for carrier testing'
• 17:35 Ryan_Lane: restarting opendj on virt0 again...
• 17:29 Ryan_Lane: restarting opendj again on virt0
• 17:24 Ryan_Lane: restarting gerrit
• 17:23 LeslieCarr: rebooting stat1 for wipe and reinstall into precise
• 16:59 Ryan_Lane: restarting opendj on virt0
• 16:49 Ryan_Lane: restarting mysql on virt0 with correct bind address
• 16:46 LeslieCarr: changing virt0's ip address and vlan
• 16:38 cmjohnson1: shutting down search32 to replace main board
• 16:24 cmjohnson1: sq48 powercycled
• 16:19 mutante: shut down sq33
• 16:18 cmjohnson1: performing hard reset on sq33
• 16:12 mutante: adding gerrit@wikimedia.org to accepted nonmembers of mediawiki-cvs list
• 15:29 mark: Unstuck torrus
• 14:46 Ryan_Lane: lowering ttl for virt0
• 14:17 Jeff_Green: storage3 dist-upgrade and reboot
• 12:31 mutante: backing up wikitech dir locally on linode instance
• 09:24 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
• 09:08 hashar: finished deploying my wmflabs related change. mediawiki-config is now at commit c0baf3e
• 09:07 logmsgbot_: hashar synchronized wmf-config
• 09:06 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-wmflabs.php
• 09:05 logmsgbot_: hashar synchronized wmf-config/throttle.php
• 09:05 logmsgbot_: hashar synchronized wmf-config/mobile-wmflabs.php
• 09:05 logmsgbot_: hashar synchronized wmf-config/mobile.php
• 08:47 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37006 - fawiki: add Book namespace + aliases'
• 08:45 hashar: reverted '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource' --> used the wrong configuration setting.
• 08:37 mutante: installing samba-common-bin, smbclient package upgrades on tridge
• 08:33 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
• 08:28 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
• 08:24 hashar: deploying several changes made to mediawiki-config gerrit changes 11034 11035 9131 11036 11037 9132 9136 and 9237
• 02:52 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Wed Jun 13 02:52:38 UTC 2012
• 02:27 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Wed Jun 13 02:27:37 UTC 2012

June 12

• 23:58 maplebed: started swift container listing loop to compare purge timing when listings are fresh
• 23:16 logmsgbot_: preilly synchronized php-1.20wmf5/extensions/MobileFrontend/ 'try again'
• 23:15 logmsgbot_: preilly synchronized php-1.20wmf4/extensions/MobileFrontend/ 'try again'
• 22:58 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Updating MobileFrontend'
• 22:53 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess'
• 22:52 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Updating MobileFrontend'
• 22:50 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess'
• 22:49 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Updating MobileFrontend'
• 20:28 RoanKattouw: Correction: the /usr/local/apache filesystem is full on hume, the root fs is not
• 20:27 RoanKattouw: hume has a full disk
• 20:23 RoanKattouw: Fixed ownership of php-1.20wmf{4,5}/cache/l10n , should be l10nupdate:wikidev . The wmf4 copy had wrong ownership causing rebuildLocalisationCache.php to fail for shell users (e.g. from scap)
• 19:40 logmsgbot_: mlitn Finished syncing Wikimedia installation... :
• 19:13 logmsgbot_: mlitn Started syncing Wikimedia installation... :
• 18:09 logmsgbot_: py synchronized wmf-config/db.php 're-adding db25 to pool after kern upgares'
• 18:02 notpeter: halting owa3 for repairs
• 16:44 logmsgbot_: py synchronized wmf-config/db.php 'removing db25 from pools for kern upgares'
• 16:27 logmsgbot_: py synchronized wmf-config/db.php 're-adding dbs 33, 34, 36, 50, 55, 56 to pools after kern upgares'
• 15:55 RobH: virt1001 and virt1002 rebooting, disregard
• 15:48 cmjohnson1_: shutting down search32 to run a diagnostic test
• 15:45 binasher: migrating enwiki.bv2009_edits (?) to innodb
• 15:41 binasher: migrating enwiki.moodbar_feedback to innodb
• 15:39 binasher: migrating enwiki.aft_article_filter_count to innodb
• 15:33 logmsgbot_: py synchronized wmf-config/db.php 'removing dbs 33, 34, 36, 50, 55, 56 from pools for kern upgares'
• 15:31 notpeter: doing another round of DB kernel upgrades
• 15:25 logmsgbot_: py synchronized wmf-config/db.php
• 15:14 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'reenabling mysql pcache'
• 15:00 binasher: rebooting db40
• 14:52 binasher: set innodb_max_dirty_pages_pct = 0 on db40 in prep for shutdown
• 14:48 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'disabling mysql parsercache (db40) in order to perform maintenance'
• 14:45 notpeter: putting kern-upgraded DBs back into pools
• 14:44 binasher: resumed replication on es3, es1002 after cluster23 sync completed
• 13:41 Jeff_Green: added awight to fr-tech@wikimedia.org email alias
• 12:06 mutante: gerrit create-project --name=mediawiki/extensions/UniversalLanguageSelector --parent=mediawiki/extensions
• 11:28 mutante: powercycling downed srv232 (also cause for check_all_memcached crit)
• 11:08 mutante: powercycled mw1042 to check for hardware issues and fscked. appears to be just unused (though down since ~3d like mw1071 per nagios)
• 10:37 mutante: test to show linking from !log via SAL to RT: RT:3100 (before/without template)
• 10:31 hashar: incubatorwiki.translate_messageindex on db39 uses MyISAM engine. See RT #3100
• 10:25 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 , take 2 - Install Translate extension on be.wikimedia.org'
• 10:24 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 , take 2 - Install Translate extension on be.wikimedia.org'
• 10:23 hashar: Compared translate% tables schema on bewikimedia with incubatorwiki. diff prove they are the same so the schema changes made early are successful.
• 10:17 hashar: bewikimedia (db39) : dropped tables translate_tmf , translate_tms and translate_tmt I have incorrectly added
• 09:59 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'revert translate extension on be.wikimedia.org, need DB update'
• 09:58 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 - Install Translate extension on be.wikimedia.org'
• 06:24 apergos: db1047 looks like the aft_article_filter_count is missing a few rows compared to the master (after replication caught up), presumably this is a side effect of the repair, have pinged binasher for help, leaving everything running and hope it's tolerable error for a day
• 02:34 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Tue Jun 12 02:34:49 UTC 2012
• 02:25 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Tue Jun 12 02:25:21 UTC 2012
• 01:35 binasher: passes the dba mantel to notpeter
• 01:22 notpeter: removing one slave from each db shard to upgrade/restart
• 00:24 binasher: shutdown mysql on es3. stopped slaving on es1002, rsyncing cluster23 tables to es3
• 00:09 binasher: pointed es3 to MASTER_LOG_FILE='es1-bin.000788', MASTER_LOG_POS=453509865
• 00:05 binasher: es3:~# rm -rf /usr/local/mysql*

June 11

• 23:54 logmsgbot_: asher synchronized wmf-config/db.php 'fully commenting out es3'
• 23:52 logmsgbot_: asher synchronized wmf-config/db.php 'making es1 the master for blobs cluster 23'
• 23:51 binasher: es1 is the new master, now switching mw conf
• 23:48 binasher: preparing to switch es master to es1
• 23:31 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 27706 - enable RSS extension on uawikimedia'
• 23:12 logmsgbot_: reedy Finished syncing Wikimedia installation... :
• 21:49 Reedy: Applied PageTriage schema updates to testwiki and enwiki
• 20:54 logmsgbot_: reedy Started syncing Wikimedia installation... :
• 20:07 logmsgbot_: reedy synchronized php-1.20wmf5/
• 19:20 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf5
• 18:54 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf5 also
• 18:52 logmsgbot_: reedy synchronized php-1.20wmf5/ 'Now we have some more space...'
• 18:50 logmsgbot_: reedy synchronized php-1.20wmf3/cache/l10n/ 'Kill l10ncache for php-1.20wmf3 as its not needed'
• 18:45 logmsgbot_: reedy synchronized php-1.20wmf2/cache/l10n/ 'Kill l10ncache for php-1.20wmf2 as its not needed'
• 18:44 logmsgbot_: reedy synchronized php-1.20wmf5/ 'Scap is taking an age, just ensure deployment files are in sync'
• 18:42 binasher: resuming coversion of es1004 to innodb, using compact row format after testing dynamic and compressed
• 18:03 logmsgbot_: reedy Started syncing Wikimedia installation... : Consistency
• 17:56 Ryan_Lane: enabling TitleBlacklist on labsconsole
• 17:28 notpeter: moving /usr/local/apache to /a/apche with symbolic link on searchidx1001 as a temp measure until it can be reimaged
• 16:58 logmsgbot_: reedy Started syncing Wikimedia installation... : Running scap to ensure consistency
• 16:53 logmsgbot_: reedy synchronized php-1.20wmf5/cache/l10n/
• 16:28 mutante: installing security upgrades on sodium
• 16:19 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 16:17 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.20wmf5.php
• 16:02 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 15:55 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 15:54 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 15:46 mutante: hume /usr/local/apache is out of disk (just 5GB but more branches now). (LVM vg "tank" lv "tank-apache" ) but no free extents. could take from /archive but unsure about shrinking the xfs.
• 15:35 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 15:30 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf5
• 15:28 logmsgbot_: reedy synchronized php-1.20wmf5/
• 15:20 Reedy: running sync-dir php-1.20wmf5
• 15:13 Reedy: Copying checkout of 1.20wmf5 onto NFS
• 14:53 mutante: running puppet on stat1. installs plotting packages
• 09:56 apergos: shut down mysqld on db1047, reparing tables
• 06:54 pp-pdf2: upgraded mwlib to 0.13.8
• 06:54 pp-pdf3: upgraded mwlib to 0.13.8
• 06:54 pp-pdf1: upgraded mwlib to 0.13.8
• 02:24 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Mon Jun 11 02:24:06 UTC 2012

June 10

• 02:22 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sun Jun 10 02:22:33 UTC 2012

June 9

• 07:45 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sat Jun 9 07:45:17 UTC 2012
• 02:35 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sat Jun 9 02:35:29 UTC 2012
• 02:25 Reedy: Running LU manually
• 02:14 Reedy: Cleared a bit of space on fenari by deleting checkouts from /tmp
• 02:00 logmsgbot_: LocalisationUpdate failed: git pull of core failed
• 01:51 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
• 01:50 logmsgbot_: reedy synchronized wikimedia.dblist
• 01:49 Reedy: fenari has a full /
• 01:49 logmsgbot_: reedy synchronized all.dblist

June 8

• 20:32 Reedy: Updated php to point to php-1.20wmf4 rather than php-1.20wmf3
• 20:09 logmsgbot_: reedy synchronized wikimedia.dblist
• 20:08 logmsgbot_: reedy synchronized all.dblist
• 20:04 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 2 wikimedia wikis to 1.20wmf4
• 19:57 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuilding localisation cache for message updates
• 19:39 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for message updates
• 18:47 logmsgbot_: reedy synchronized php-1.20wmf4/languages/messages/ 'Pushing out updated files upon siebrands request'
• 16:30 cmjohnson1: shutting down search32 to swap DIMM around
• 10:19 notpeter: ganglia down, restarting apache on nickel.
• 09:32 notpeter: stopping indexing on searchidx1001 to re-copy to searchidx2
• 08:45 notpeter: reimaging searchidx2 with correct partitioning
• 06:50 Tim: on cp1001: disabled HTCP plugin in gmond for testing, seems to work so I will disable it properly
• 03:55 Tim: disabled LastModified extension due to overload on cp1005
• 03:51 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
• 03:18 Tim: restarting squid on cp1005, maybe out of FDs or something, cachemgr shows exactly 1000 open connections to 10.2.1.1
• 03:08 Tim: stopped gmond on cp1001 with kill -STOP for memory leak debugging
• 03:02 Tim: on cp1002: killed gmond again, it was leaking memory again, already up to 27GB in the few minutes since I restarted it
• 03:01 Tim: on fenari: copied *.text and *.upload from /home/wikipedia/conf/squid/generated/clusters to /etc/dsh/group
• 02:56 Tim: cp1001: same as on cp1002, restarted gmond
• 02:53 Tim: on cp1002: killed gmond, which was using 100% CPU and 23GB RSS. Restarting squid which had died
• 02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Fri Jun 8 02:23:09 UTC 2012
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Fri Jun 8 02:14:55 UTC 2012

June 7

• 23:32 Tim: deploying varnish configuration change https://gerrit.wikimedia.org/r/#/c/10672/ on cp1041, cp1042, cp1043, cp1044
• 20:46 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/LastModified 'syncing LastModified extension'
• 20:32 logmsgbot_: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on LastModified and E3Experiemnts for en.wiki'
• 19:54 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'syncing js file for E3Experiments'
• 18:58 cmjohnson1_: shutting down search32 for testing
• 16:52 Ryan_Lane: added ldap automount entries for /public/datasets and /public/keys
• 02:19 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Thu Jun 7 02:19:52 UTC 2012
• 02:11 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Thu Jun 7 02:11:49 UTC 2012

June 6

• 22:29 logmsgbot_: aaron synchronized wmf-config/swift.php 'deployed 7dc77e431310580da0dbd368b8b290a293e3ee21'
• 19:55 cmjohnson1: shutting down search32 to reseat DIMM B2
• 19:18 cmjohnson1: shutting down storage3
• 18:05 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaining wikis to 1.20wmf4
• 17:02 Jeff_Green: mailman 'site' password changed per RT 3039
• 11:12 mark: Added AAAA record to mobile
• 10:38 mark: Wikipedia is IPv6-enabled.
• 10:37 mark: Added AAAA records to all non-mobile wiki projects
• 10:19 mark: Added AAAA record to bits.wikimedia.org
• 10:02 Ryan_Lane: repooling ssl3001
• 10:00 mark: Added AAAA record to upload.wikimedia.org
• 09:51 Ryan_Lane: depooling ssl3001
• 08:41 mark: Converted bits.wikimedia.org into a direct geodns record, removed the old bits -> bits-geo CNAME
• 07:53 mark: Converted geoiplookup.wikimedia.org into a separate, IPv4-only geodns record
• 07:37 pp-pdf1: installed tmpreaper cronjob for /home/pp/mathcache directory
• 07:37 pp-pdf2: installed tmpreaper cronjob for /home/pp/mathcache directory
• 07:37 pp-pdf3: installed tmpreaper cronjob for /home/pp/mathcache directory
• 02:35 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Wed Jun 6 02:35:47 UTC 2012
• 02:11 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Wed Jun 6 02:10:58 UTC 2012

June 5

• 22:37 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Undo temporary woff whitelisting'
• 22:34 logmsgbot_: aaron synchronized php-1.20wmf4/includes/filerepo/file/LocalFile.php 'deployed 4791e3d25aebe9643a7cea91f2eb49e6b54593c5'
• 22:33 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Temporarily allow uploading woff files on slwikisource'
• 22:13 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/PageTriage/modules/ext.pageTriage.models/ext.pageTriage.article.js 'updating default PageTriage filters'
• 20:56 logmsgbot_: mmullie Finished syncing Wikimedia installation... :
• 20:16 logmsgbot_: mmullie Started syncing Wikimedia installation... :
• 19:49 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Fix notice in MobileFrontend config'
• 19:45 pp-pdf2: updated mwlib to 0.13.7-1-g827780b
• 19:45 pp-pdf1: updated mwlib to 0.13.7-1-g827780b
• 19:45 pp-pdf3: updated mwlib to 0.13.7-1-g827780b
• 19:45 pp-pdf2: updated mwlib to 0.13.7-1-g827780b
• 18:45 notpeter: starting innobackupex dump from blondel to bellin
• 18:44 notpeter: starting indexing on new searchidx2
• 18:39 mark: Added static routes 2002::/16 and 2001::/32 for 6to4 and teredo on the Tampa routers; these are redistributed in OSPF to eqiad
• 18:29 notpeter: restart indexing on searchidex1001
• 18:20 mark: Replaced static LVS IPv6 routes with correct next-hops on cr1-eqiad and cr2-eqiad
• 18:09 mark: Redistributing static routes in OSPF on cr1-eqiad and cr2-eqiad
• 18:04 paravoid: rebooting capella to make sure things work after a reboot
• 17:53 mark: Redistributed statics in OSPF3 on csw1-esams
• 17:14 logmsgbot_: aaron synchronized php-1.20wmf4/includes/logging/LogEventsList.php 'deployed d9f146ac42f2884e76390d6bc979eb10032adf7f'
• 17:09 mark: Added uRPF exception for 6to4 traffic on all routers
• 17:05 jeremyb: (UTC) 23:42:14 <binasher> !log re-enabled es4 monitoring. its currently our only es server without any tables marked as crashed / needing recovery, myisam recovery has been absent for all systems since the ms servers were migrated off of in nov 2011. (Sum of human knowledge * Renyi entropy = ES)
• 16:52 mark: Pooled ssl1001
• 15:44 logmsgbot_: asher synchronized wmf-config/db.php 'returning es2 to service'
• 15:25 paravoid: rebooting lvs1004 and reinstalling with precise
• 15:17 binasher: rebooting es2 for kernel + mysql upgrade
• 15:16 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es2 for kernel+mysql upgrades'
• 14:56 paravoid: rebooting amslvs3 & amslvs4 to reinstall with precise
• 14:20 paravoid: rebooting lvs1006 to reinstall with precise
• 13:51 cmjohnson1: shutting down bellin to replace main board
• 13:49 notpeter: reimaging db1042
• 13:40 paravoid: rebooting lvs1005 to reinstall with precise
• 12:59 paravoid: rebooting lvs2 to reinstall with precise
• 12:47 Ryan_Lane: changing capella's subnet in DNS
• 12:10 Ryan_Lane: rebuilding capella as precise
• 10:01 logmsgbot_: asher synchronized wmf-config/db.php 'putting es1 in production'
• 09:53 notpeter: cancel that, it's mid-cron. will do later
• 09:52 notpeter: stopping indexing on searchidx1001 to rsync to searchidx2
• 09:35 binasher: rebooting es1 for kernel+mysql upgrade. dont need to pull from db.php because it was never correctly added or queried?
• 09:14 mark: Built PyBal 1.01 for precise, and included it in the precise-wikimedia APT repository
• 08:45 binasher: restarted mysql on es1004 with default innodb file format as barracuda
• 08:31 notpeter: reimaging searchidx2
• 02:36 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Tue Jun 5 02:36:44 UTC 2012
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Tue Jun 5 02:14:17 UTC 2012

June 4

• 23:23 logmsgbot_: asher synchronized wmf-config/db.php 'returning es4'
• 22:49 binasher: started an experiment on es1004 - altering all es tables from myisam to innodb one at a time with file_per_table enabled
• 22:39 binasher: stopping mysql on es4. all tables marked as having repair fails are in cluster22, resyncing just those from es1002
• 21:52 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es4 again'
• 21:43 logmsgbot_: asher synchronized wmf-config/db.php 'returning es4 to service'
• 21:16 Ryan_Lane: restarting nginx on all ssl boxes again
• 21:07 Ryan_Lane: force running puppet on all ssl hosts again
• 21:04 Ryan_Lane: repooling ssl1, ssl1001, ssl3001
• 21:03 Ryan_Lane: restarting nginx on all ssl hosts
• 20:23 binasher: rebooted es4
• 20:18 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es4 for post-crash upgrade'
• 19:55 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/PageTriage/PageTriage.hooks.php 'syncing PageTriage.hooks.php'
• 19:32 Ryan_Lane: force running puppet on ssl servers
• 18:22 Reedy: Nuked php-1.20wmf4 on mw64 then ran sync-common. Seems to have dealt with the permission errors
• 18:11 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf4
• 16:48 binasher: upgraded kernel on db1047 / analytics
• 16:09 Ryan_Lane: restarting ircecho on manganese
• 15:21 paravoid: reinstalling lvs1 with precise
• 15:13 mark: Added new IPv6 LVS prefixes to all routers for uRPF filters; BGP import filters still need adjusting for dual-family sessions
• 15:08 cmjohnson1: physically power cycling lvs1
• 15:02 Ryan_Lane: depooling ssl1001 and ssl3001
• 14:55 Ryan_Lane: disabling puppet on all ssl hosts
• 13:27 mark: Changed upload.esams.wikimedia.org CNAME to upload-lb.esams, effectively disabling the IPv6 selective answer script
• 12:23 mark: Upgrading wikimedia-lvs-realserver to version 0.08 across the cluster (by Puppet)
• 12:18 Ryan_Lane: depooling ssl1
• 11:32 mark: Copied wikimedia-lvs-realserver 0.08 from APT distribution precise-wikimedia to lucid-wikimedia
• 02:38 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon Jun 4 02:38:15 UTC 2012
• 02:14 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Mon Jun 4 02:14:45 UTC 2012

June 3

• 15:45 paravoid: aborting lvs1 install, partition map is not ready; putting it back to production as-is
• 15:31 paravoid: reinstalling lvs1 with precise
• 15:10 RobH: torrus failed to refresh via puppet (failed refresh takes too long) so manually running the refresh/rebuild command as puppet copied the updates to the system
• 14:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37237 - Change Wikisource namespace for Tamil wikisource'
• 14:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 37211 - Set $wgUseCombinedLoginLink = false'
• 14:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37294 - Add English Wikibooks as import source at Vietnamese Wikibooks'
• 14:07 logmsgbot: midom synchronized wmf-config/db.php
• 13:10 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '59753a9 (allow bureaucrats on frwiki to add+remove accountcreator group'
• 13:09 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production) -- was not correctly deployed earlier'
• 13:02 logmsgbot: midom synchronized wmf-config/db.php
• 10:18 hashar: mw64: rsync: write failed on "/apache/common-local/wmf-config/CommonSettings.php": No space left on device (28)
• 10:17 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production)'
• 09:16 notpeter: pushing new zone files. only minor changes
• 02:35 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun Jun 3 02:35:31 UTC 2012
• 02:13 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sun Jun 3 02:13:39 UTC 2012

June 2

• 15:37 hashar: We ran out of beer, see bug 37307
• 15:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily enable $wmgReduceStartupExpiry on testwiki for Berlin tutorial'
• 14:12 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
• 14:10 logmsgbot: hashar synchronized wmf-config/ext-wmflabs.php
• 14:10 hashar: deploying some nasty configuration changes in wmf-config
• 14:10 logmsgbot: hashar synchronized wmf-config/ext-pmtpa.php
• 14:04 logmsgbot: reedy synchronized wmf-config/proofreadpage.php 'Default proofreadpage-showheaders to 1 on enwikisource and svnwikisource'
• 13:46 mutante: rebuilding archives for fd-advisorygroup mailing list
• 13:21 RobHalsell: updated quotas on labstore1 for publicdata-proect
• 08:57 logmsgbot: reedy synchronized wmf-config/ 'https://gerrit.wikimedia.org/r/#/c/9717/'
• 02:36 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat Jun 2 02:36:20 UTC 2012
• 02:14 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sat Jun 2 02:14:25 UTC 2012

June 1

• 20:54 logmsgbot: reedy synchronized php-1.20wmf4/includes/specials/SpecialRecentchanges.php
• 20:53 logmsgbot: reedy synchronized php-1.20wmf4/includes/SpecialPage.php
• 20:28 logmsgbot: reedy synchronized php-1.20wmf4/extensions/ExtensionDistributor/
• 20:27 logmsgbot: reedy synchronized php-1.20wmf3/extensions/ExtensionDistributor/
• 19:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36805 - Enable NewUserMessage extension on mrwiki and mrwikisource'
• 19:34 logmsgbot: reedy synchronized wmf-config/ 'Fix some typos'
• 19:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36965 - Please setup Collection extension on Telugu Wikipedia'
• 19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37027 - Install Collection extension in Hebrew Wiktionary'
• 19:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enabling collection on tawikis'
• 19:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enabling collection on tawikis'
• 17:56 logmsgbot: reedy synchronized wmf-config/
• 16:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Testing
• 16:23 logmsgbot: reedy Started syncing Wikimedia installation... : Testing
• 15:33 Reedy: pointing php to php-1.20wmf3
• 13:22 logmsgbot: maxsem synchronized php-1.20wmf3/extensions/MobileFrontend/ 'Deploying MF fixes'
• 13:21 logmsgbot: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Deploying MF fixes'
• 12:19 Reedy: Purged www.mediawiki.org/xml/export-0.7.xsd
• 12:17 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.7.xsd 'Push out updated version of export-0.7.xsd'
• 06:45 apergos: reboot dataset2, kernel update and security updates
• 02:33 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Fri Jun 1 02:33:36 UTC 2012
• 02:11 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Fri Jun 1 02:11:08 UTC 2012

May 31

• 23:40 logmsgbot: kaldari Finished syncing Wikimedia installation... : scapping for new LastModified and E3Experiments extensions
• 23:14 logmsgbot: kaldari Started syncing Wikimedia installation... : scapping for new LastModified and E3Experiments extensions
• 23:09 logmsgbot: aaron synchronized multiversion/ 'Updating multiversion code to head.'
• 22:25 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 22:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable randomrootpage again'
• 22:04 logmsgbot: aaron synchronized wmf-config/swift.php 'Purge from squid all thumbs in Swift on purge.'
• 21:50 logmsgbot: reedy synchronized php-1.20wmf4/includes/specials/SpecialLog.php
• 21:50 logmsgbot: reedy synchronized php-1.20wmf4/includes/logging/LogEventsList.php
• 20:46 pp-pdf2: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
• 20:46 pp-pdf3: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
• 20:46 pp-pdf1: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
• 20:38 logmsgbot: lcarr synchronized wmf-config/mc.php
• 20:00 LeslieCarr: powering down mw32 for maintenance
• 19:59 LeslieCarr: powering down mw30 for maintenance
• 19:57 LeslieCarr: powering off mw31
• 17:32 LeslieCarr: rebooting mw1135 for kernel upgrade
• 17:24 cmjohnson1: running memtet on mw64
• 17:14 LeslieCarr: rebooting unresponsive mw1143
• 17:13 LeslieCarr: rebooting unresponsive mw1135
• 17:10 LeslieCarr: rebooted mw1091 for kernel upgrade
• 17:09 LeslieCarr: rebooted ms1004 for kernel upgrade
• 17:08 LeslieCarr: rebooted mw1102 because it thinks it has no eth0
• 17:05 LeslieCarr: rebooted mw1091 due to being unresponsive
• 17:01 LeslieCarr: rebooted ms1004 due to it being unresponsive
• 17:01 LeslieCarr: rebooted ms1004
• 14:28 binasher: pulling srv199 from lvs again for further experimentation
• 14:03 binasher: returning srv199 to lvs (dialed back to slow / no longer running php 5.4)
• 13:40 maplebed: upgrading and rebooting eqiad es hosts due to 210 day kernel bug thingy.
• 13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bring in a few shell requests'
• 12:46 logmsgbot: reedy synchronized php-1.20wmf4/languages/Language.php '(bug 36839) Use mb_check_encoding() if available'
• 12:45 logmsgbot: reedy synchronized php-1.20wmf3/languages/Language.php '(bug 36839) Use mb_check_encoding() if available'
• 10:47 binasher: temporarily pulled srv199 from lvs for php testing
• 10:44 Reedy: reedy synchronized php-1.20wmf4/extensions/UploadWizard/ 'Push trunk UW to cluster'
• 10:43 Reedy: reedy synchronized php-1.20wmf3/extensions/UploadWizard/ 'Push trunk UW to cluster'
• 02:32 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Thu May 31 02:32:48 UTC 2012
• 02:17 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings to disable PageTriage on test2'
• 02:10 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Thu May 31 02:10:43 UTC 2012
• 00:26 logmsgbot: awjrichards synchronizing Wikimedia installation... :

May 30

• 23:01 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
• 22:22 logmsgbot: awjrichards synchronizing Wikimedia installation... : Weekly MobileFrontend deployment and picking up ZeroRatedMobileAccess i18n changes
• 21:50 logmsgbot: kaldari synchronized php-1.20wmf4/extensions/CentralNotice 'deploying 98db6a177df977a699576da9688588c77bf81b04'
• 21:30 K4-713: Synchronized payments cluster to DonationInterface 43a457e56d
• 21:16 LeslieCarr: rebooted db1044 (unresponsive server)
• 21:10 LeslieCarr: rebooted db1031 (unresponsive server)
• 21:08 LeslieCarr: rebooted db1029 (unresponsive server)
• 21:07 LeslieCarr: restarted db1026 (unresponsive server)
• 21:03 LeslieCarr: restarted db1012 (unresponsive server)
• 20:57 LeslieCarr: restarted networking on cp1036
• 19:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable randomrootpage on wikibooks and wikisources'
• 19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 27 more of the misc wikis to 1.20wmf4
• 19:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Some more of the misc wikis to 1.20wmf4
• 19:11 LeslieCarr: rebooting neon
• 18:55 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All special wikis to wmf4
• 18:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisource, wikiversity, wiktionary to wmf4
• 18:45 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikibooks, wikinews, wikiquote to wmf4
• 18:29 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Commonswiki to wmf4
• 18:16 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved EN, non-wikipedia, non-special, sites to wmf4
• 18:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Fix capitalisation'
• 18:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable random root page on testwiki'
• 17:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add enabling code for randomrootpage'
• 17:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add setting for random root page'
• 17:47 LeslieCarr: cleared mobile varnish cache
• 17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repository
• 17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repo
• 17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repo
• 17:40 logmsgbot: aaron synchronized php-1.20wmf4/extensions/PageTriage 'Switched to wmf4 extension branch to get 0be1787634613a36439b760d6d5f0639724f8a7b'
• 16:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'subpages for frwikibooks'
• 12:00 mutante: restarting pdns on ns2
• 11:41 mutante: running authdns-update to push analytics1011 to 1022 entries
• 06:05 logmsgbot: hashar synchronized docroot/mediawiki/xml 'bug 37111 deploying export-0.7.xsd'
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Wed May 30 02:37:00 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 30 02:23:47 UTC 2012

May 29

• 21:09 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabled new thumb purge hook on remaining wikis'
• 20:49 LeslieCarr: renaming analytics1001.eqiad.wmnet to analytics1001.wikimedia.org
• 20:25 pp-pdf2: restarted services
• 20:25 pp-pdf3: restarted services
• 20:25 pp-pdf1: restarted services
• 20:25 pp-pdf2: cleaned /tmp and sandbox/cache/
• 20:25 pp-pdf1: cleaned /tmp and sandbox/cache/
• 20:25 pp-pdf3: cleaned /tmp and sandbox/cache/
• 20:15 Thehelpfulone: "Site requests" was renamed to "Site configuration" under the Wikimedia product in Bugzilla, don't know who did it though
• 20:12 LeslieCarr: reloading analytics1001
• 19:50 pp-pdf2: restarted all services
• 19:50 pp-pdf3: restarted all services
• 19:50 pp-pdf1: restarted all services
• 19:50 pp-pdf3: add libtidy.so
• 19:50 pp-pdf1: add libtidy.so
• 19:50 pp-pdf2: add libtidy.so
• 19:49 pp-pdf3: install mwlib.epub
• 19:49 pp-pdf2: install mwlib.epub
• 19:49 pp-pdf1: install mwlib.epub
• 19:49 pp-pdf1: update simplejson to 2.5.2
• 19:49 pp-pdf3: update simplejson to 2.5.2
• 19:49 pp-pdf2: update simplejson to 2.5.2
• 19:49 pp-pdf1: update mwlib.rl to 0.12.11
• 19:49 pp-pdf2: update mwlib.rl to 0.12.11
• 19:49 pp-pdf3: update mwlib.rl to 0.12.11
• 19:48 pp-pdf1: update pip to 1.1
• 19:48 pp-pdf3: update pip to 1.1
• 19:48 pp-pdf2: update pip to 1.1
• 18:59 LeslieCarr: flushed mobile varnish caches after push
• 16:54 maplebed: kicking pdns on dobson to try and make it happy again.
• 16:40 notpeter: decom of all srv lower than 190
• 16:27 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/#/c/9204/ - use protocol-relative url for nostalgiawiki wgSiteNotice'
• 16:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'cleanup wgNoticeBanner_Harvard2011 https://gerrit.wikimedia.org/r/#/c/9205/'
• 16:17 hashar: /usr/local/apache/common-local is 4G where as / is 7G on srv187. Looks like deploying wmf2 + wmf3 + wmf4 will require partitions to be resized.
• 16:10 hashar: srv187 and srv188 are out of disk space
• 16:10 logmsgbot: hashar synchronized search-redirect.php 'https://gerrit.wikimedia.org/r/9206 - cleanup search-redirect.php'
• 14:37 cmjohnson1: removing disk4 on virt1 for replacement
• 12:18 mutante: killing / restarting morebots
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Tue May 29 02:37:44 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 29 02:24:21 UTC 2012

May 28

• 23:15 logmsgbot: reedy synchronized php-1.20wmf4/extensions/UploadWizard/ 'Master UW per Kaldari'
• 18:25 logmsgbot: reedy synchronized php-1.20wmf4/extensions/Translate/tag/PageTranslationHooks.php 'Fix Catchable fatal error'
• 18:08 logmsgbot: reedy synchronized php-1.20wmf4/LocalSettings.php
• 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf4
• 16:13 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild l10n for php-1.20wmf4
• 15:44 logmsgbot: reedy synchronizing Wikimedia installation... : test2wiki to 1.20wmf4 to build localisation cache
• 14:39 logmsgbot: reedy synchronizing Wikimedia installation... : Does running scap on its own with no wikis on that version build l10n for it? I suspect not...
• 14:22 logmsgbot: reedy synchronized php-1.20wmf4/ 'Staging php-1.20wmf4'
• 14:15 Reedy: sync-dir'ing php-1.20wmf4
• 14:12 Reedy: Copying php-1.20wmf4 from /tmp to /h/w/c on Fenari
• 13:12 apergos: doing security updates for a batch of mws in eqiad
• 11:22 apergos: updated kernel etc on mw1133, reboot
• 11:08 apergos: powercycled mw1133
• 09:05 apergos: rebooted snapshot4, 3 for security updates
• 08:43 apergos: rebooted snapshot1002, security updates (will do the same for 1003, 1004 shortly)
• 08:37 apergos: rebooted snapshot1001, security updates
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon May 28 02:24:05 UTC 2012

May 27

• 22:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37134 - s:cs: site settings'
• 14:45 logmsgbot: asher synchronized wmf-config/db.php 'returning db51 to prod as an s4 slave'
• 14:34 maplebed: dns and puppet changes for s4 master rotation done.
• 14:18 binasher: rebooted db51, reslaved
• 14:09 maplebed: new s4 master position post-rotation is master_log_file="db31-bin.000334", master_log_pos=583315125
• 14:02 logmsgbot: ben synchronized wmf-config/db.php 's4 master switch complete; db31 is the new master. turning off read only on s4'
• 13:25 Tim: on db31 set global read_only=1
• 13:17 logmsgbot: tstarling synchronized wmf-config/db.php 's4 read-only and taking out db51'
• 04:27 binasher: kaulen - temporarily disabled swap and set oom_adj score to 15 for apache
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun May 27 02:24:13 UTC 2012

May 26

• 19:54 apergos: restarting apache2 on kaulen
• 19:34 Nemo_bis: bugzilla down again
• 14:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Settting wgBlacklistSettings'
• 11:05 apergos: stopping and restarting apache on kaulen blah blah blah
• 09:12 Reedy: Bugzilla is down, Kaulen looks to be in swap death again
• 06:46 apergos: powercycling kaulen
• 05:53 hashar: kaulen dead :-[
• 05:47 hashar: Bugzilla on Kaulen being super slow again
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat May 26 02:22:37 UTC 2012
• 00:40 K4-713: synchronized payments cluster to DonationInterface 04809f1cf0d

May 25

• 22:57 maplebed: powercycled kaulen on the mgmt interface
• 19:09 maplebed: disabled the outdated /etc/init.d/gmond on spence. use ganglia-monitor instead.
• 18:57 RobH: bugzilla appears back online
• 18:57 RobH: kaulen is rebooted, it may have had a runaway process or a memory leak, not sure yet, but it was locked up from access
• 18:55 RobH: kaulen serial console unresponsive, rebooting
• 17:42 paravoid: rebooting gurvin & yvon with new kernel
• 17:25 paravoid: resetting gurvin, load spiking at 370+, SSH unreachable, 214 days of uptime
• 15:34 mark: Power cycled kaulen
• 15:23 hashar: kaulen (bugzilla) unreacheable :-(
• 13:56 RobH: palladium disk replaced
• 13:52 cmjohnson1: replacing ps2 on mw1017
• 13:50 RobH: palladium has a bad disk, goign to replace it
• 13:37 RobH: updating drac on search18, shouldnt cause system reboot.
• 10:42 apergos: restarted apache on kaulen, was seeing page.cgi segfaults in dmesg and he logs, huge cpu wait spikes (why?)

May 24

• 23:34 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
• 23:32 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Updating technical feedback email address for mobile feedback'
• 23:17 logmsgbot: awjrichards synchronizing Wikimedia installation... : Picking up changes to hide feedback form to prevent spamming of mobile feedback page - f6ed8ba
• 23:12 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Enabling 'technical feedback' link on mobile feedback form to disable feedback form'
• 23:00 binasher: stopped replication on es1002 in order to rsync cluster23 to es1003
• 21:53 logmsgbot: aaron synchronized php-1.20wmf3/extensions/UploadWizard 'deployed 144b58854e38d910210ccd23402225e5b1d2d62d'
• 21:52 RoanKattouw: Restarted morebots

May 23

• 21:46 binasher: shutting down mysql on db12 in able to restart with binlogging disabled
• 21:45 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db60'
• 21:34 ottomata1: upgraded udp-filter to 0.2.4 on oxygen, emery, and locke (with maplebed's help)
• 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last 286 wikis over to 1.20wmf3
• 17:34 maplebed: deployed change to varnish configs for preilly; adding more carriers
• 17:13 logmsgbot: ben synchronized wmf-config/CommonSettings.php 'changing the URL for the mobile feedback page to the Project namespace'
• 17:12 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'RL hack'
• 17:00 logmsgbot: reedy synchronized wmf-config/ 'Tidying unpushed changes'
• 15:35 RobH: ns1 died on update, restarting pdns
• 15:34 RobH: updated dns for analytics mgmt
• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 23 02:45:44 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 23 02:23:33 UTC 2012
• 01:10 K4-713: synchronized payments cluster to DonationInterface 4be175e43f
• 00:16 Ryan_Lane: flushed the varnish cache for mobile again
• 00:13 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'
• 00:11 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'
• 00:06 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'try again'
• 00:00 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'

May 22

• 23:49 Ryan_Lane: flushed the varnish cache for mobile again
• 23:31 Ryan_Lane: flushed the varnish cache for mobile
• 23:21 logmsgbot: preilly synchronizing Wikimedia installation... : MobileFrontend Weekly Deployment
• 21:34 RobH: updating dns for mgmt of new servers in eqiad
• 20:52 logmsgbot: reedy synchronized php-1.20wmf3/extensions/MoodBar/ 'updating to master'
• 20:24 notpeter: powering up db1003
• 20:05 notpeter: starting xtrabackup dump from db1004 to db1020 for new eqiad s4 slave
• 19:55 notpeter: starting xtrabackup dump from db1033 to db1001 for new eqiad s1 slave
• 19:31 notpeter: reimaging db1001 and db1020
• 19:18 RobH: dns update for new servers mgmt ips
• 19:12 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding message files for interwiki extension
• 18:44 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/file/LocalFile.php 'deployed 826f82eaccdf2a017a8ddb27829156f7c474db84'
• 18:44 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
• 18:16 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php 'deployed 178a8597e32122feeb593219452f26864639d9ad'
• 17:56 maplebed: done with deploy to swift to make mediawiki write thumbnails for all wikis
• 17:47 logmsgbot: aaron synchronized wmf-config/swift.php 'Switched all wikis to new Swift thumb copy hook.'
• 17:44 maplebed: starting deploy to make mediawiki write thumbnails to swift for all wikis
• 17:35 logmsgbot: reedy synchronized php-1.20wmf3/extensions/TranslationNotifications/ 'Pushing new version of translationnotification out'
• 17:32 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'pushing interwiki loading code'
• 17:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'pushing interwiki variables out'
• 17:30 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Interwiki/ 'pushing interwiki code to cluster'
• 03:57 hashar: GlusterFS receiving 30Mbytes/sec of input traffic. Killing labs again :-D
• 02:50 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Tue May 22 02:50:09 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 22 02:26:50 UTC 2012
• 01:40 K4-713: updated payments cluster to Donation Interface 67b40c9307b
• 00:17 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/file/LocalFile.php 'deployed dfa7120f1bcd2c172096caf0ca65a06119e592c3'

May 21

• 22:31 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove header fail logging'
• 22:30 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php
• 22:20 logmsgbot: preilly synchronized wmf-config/PrivateSettings.php
• 22:19 logmsgbot: preilly synchronized wmf-config/CommonSettings.php
• 21:52 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Don't do content length checking if it's a head request'
• 21:47 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Don't do content length checking if it's a head request'
• 20:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable PageTriage on enwiki'
• 20:47 logmsgbot: catrope synchronized php-1.20wmf3/extensions/PageTriage 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:47 logmsgbot: catrope synchronized php-1.20wmf3/extensions/ArticleFeedbackv5 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:46 logmsgbot: catrope synchronized php-1.20wmf3/includes/resourceloader 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:46 logmsgbot: catrope synchronized php-1.20wmf2/extensions/PageTriage 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:46 logmsgbot: catrope synchronized php-1.20wmf2/extensions/ArticleFeedbackv5 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:45 logmsgbot: catrope synchronized php-1.20wmf2/includes/resourceloader 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf3
• 18:04 Jeff_Green: dist-upgrade and reboot loudon
• 17:48 andrewbogott_: ran authdns-update on dobson to pick up virt1002-1008 changes
• 17:13 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/LocalRepo.php 'deployed dfa7120f1bcd2c172096caf0ca65a06119e592c3'
• 16:22 mutante: analytics1001 to 1010 installed and up in puppet
• 12:45 mark: Started ircecho on manganese
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Mon May 21 02:46:37 UTC 2012
• 02:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'There's no helping metawiki now!'
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon May 21 02:23:51 UTC 2012

May 20

• 19:29 logmsgbot: aaron synchronized wmf-config/swift.php 'more profiling'
• 17:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Syncing I6b0e91cd/bug 36931: tweaking account creation whitelist for ptwiki event'
• 02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sun May 20 02:43:08 UTC 2012
• 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun May 20 02:21:20 UTC 2012

May 19

• 21:07 cmjohnson1: shutting down storage3 to replace RAID controller card
• 18:37 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling PageTriage extension on enwiki per request from Kaldari, due to bug 36968'
• 02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sat May 19 02:43:56 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat May 19 02:22:21 UTC 2012

May 18

• 23:27 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on commonswiki.'
• 20:18 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Tighten debugging'
• 20:12 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Widen debugging'
• 20:06 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/FileBackendStore.php 'deployed 0624af8f2e9666fbe0820c0caca6d7ea3c6eeb7b'
• 20:02 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 're-enable debugging (fatal disabled)'
• 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Put wikibooks back on 1.20wmf3
• 19:27 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Disable debugging and content length checking for now'
• 19:26 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'better debugging'
• 19:24 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'better debugging'
• 19:21 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'RE-enable header fail stuff with debug logs'
• 19:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add debug log group for headerfail'
• 19:00 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Trying older version'
• 18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Push wikibooks back to 1.20wmf2 due to collection being broken
• 18:31 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection
• 18:28 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection '1.20wmf2 collection for testing'
• 18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.20wmf3
• 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.20wmf2
• 18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable collection on test2wiki'
• 18:02 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection/Collection.session.php
• 18:00 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection/Collection.templates.php 'Testing partial revert'
• 15:45 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Applying various changes made this afternoon: 718fb59..811dbd8'
• 14:25 hashar: setup a Jenkins job to lint PHP files in operations/mediawiki-config.git:/wmf-config/
• 14:00 mutante: authdns-update - pushing fix for reverse lookup in eqiad subnets
• 12:04 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Syncing https://gerrit.wikimedia.org/r/7931 & https://gerrit.wikimedia.org/r/7934 : minor pmtpa/wmflabs switches'
• 10:39 logmsgbot: hashar synchronized wmf-config/wgConf.php 'https://gerrit.wikimedia.org/r/#/c/7933/ change cluster name "beta" to "wmflabs"'
• 08:59 logmsgbot: nikerabbit synchronized php-1.20wmf3/extensions/Translate/specials/SpecialAggregateGroups.php 'Temp fix for bug 36944'
• 03:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable UploadWizard on donatewiki and foundationwiki'
• 02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 18 02:43:38 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Fri May 18 02:22:11 UTC 2012
• 01:42 Tim: on cp1004: set net.ipv4.tcp_tw_recycle=0 and net.ipv4.tcp_tw_reuse=1
• 01:39 binasher: filejournal migration complete
• 00:50 binasher: migrating fliejournal to innodb on all wikis
• 00:22 Tim: on cp1005: set tcp_tw_recycle=0

May 17

• 22:18 binasher: migrated centralauth.wikiset to innodb
• 22:01 binasher: migrating centralauth.spoofuser to innodb via osc (13.5mil rows)
• 22:00 binasher: migrated centralauth.global_group to innodb
• 21:53 maplebed: reverted mobile change from this morning - testing completed.
• 21:42 binasher: es1004 is replicating again
• 21:39 binasher: resumed replication to es1002
• 21:21 logmsgbot: catrope synchronized php-1.20wmf3/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa'
• 21:20 logmsgbot: catrope synchronized php-1.20wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa'
• 21:09 logmsgbot: aaron synchronized wmf-config/swift.php '+commonswiki'
• 20:58 logmsgbot: aaron synchronized wmf-config/swift.php 'revert change to itwiki'
• 20:57 Jeff_Green: several package updates on payments* and silicon
• 20:56 logmsgbot: aaron synchronized wmf-config/swift.php '+itwiki'
• 20:40 logmsgbot: aaron synchronized wmf-config/swift.php
• 20:34 maplebed: deployed change to swift and mediawiki for MW to write thumbnails to swift instead of rewrite.py with aaron
• 20:32 maplebed: deployed parallel thumbnail purging for test, test2, and mediawiki with aaron
• 20:25 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabled thumb copy hook for testwikis and mw.org'
• 20:14 binasher: completed securepoll_votes.vote_ip and all ipv6 schema migration
• 20:10 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
• 20:09 logmsgbot: aaron synchronized wmf-config/swift.php
• 20:03 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabling new purge hook on testwikis again.'
• 20:03 binasher: running securepoll_votes.vote_ip schema migration on s1
• 20:01 binasher: running securepoll_votes.vote_ip schema migration on all s2 dbs
• 19:19 binasher: running securepoll_votes.vote_ip schema migration on all s4 + s3 dbs
• 19:17 binasher: running securepoll_votes.vote_ip schema migration on all s5 dbs
• 19:16 binasher: running securepoll_votes.vote_ip schema migration on all s6 dbs
• 19:02 binasher: running securepoll_votes.vote_ip schema migration on all s7 dbs
• 18:49 binasher: syncing cluster23 tables from es1002 to es1004
• 18:46 binasher: stopped replication on es1002
• 18:44 notpeter: restarting puppet on brewster
• 18:39 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
• 18:13 logmsgbot: aaron synchronized wmf-config/swift.php
• 18:08 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo 'deployed 103efda39dd57bc22898bd0e69932982c1cfd588'
• 18:00 Jeff_Green: shutting down grosley for disk and RAM upgrades
• 17:42 notpeter: temporarily turning off puppet on brewster for preseed hackz
• 17:20 maplebed: flushing the mobile cache post-deploy
• 17:17 maplebed: deploying config change to mobile - more zero IP addresses. gerrit r7867
• 15:31 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 15:30 mutante: sync-common-file interwiki.cdb
• 15:30 mutante: creating fresh interwiki.cdb from dumpInterwiki.php
• 15:30 Jeff_Green: adding DNS records to wikimedia.org for RT #2960
• 14:22 mutante: adding gerrit project analytics/udplog parent analytics
• 13:44 cmjohnson1: shutting down bellin for troubleshooting
• 09:04 hashar: Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown.
• 08:59 hashar: Broken the cluster by having an invalid global set
• 08:58 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
• 08:47 logmsgbot: hashar synchronizing Wikimedia installation... :
• 08:44 hashar: running scap to apply https://gerrit.wikimedia.org/r/7702
• 08:41 hashar: Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination
• 08:15 hashar: WMFLabs seems to have recovered now
• 06:50 hashar: WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore
• 03:41 logmsgbot: asher synchronized wmf-config/db.php 'returning db12 and db46'
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Thu May 17 02:48:02 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Thu May 17 02:22:02 UTC 2012
• 02:18 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable SpecialCite everywhere'
• 01:40 Tim: on cp1004: reverted after TIME_WAIT client connections reached 38k with no sign of a plateau
• 01:37 Tim: on cp1004: trying tcp_tw_reuse=1 instead of tcp_tw_recycle
• 01:00 Tim: reverted after client-side TIME_WAIT connections rose rapidly from 367 to 9000
• 00:59 Tim: experimentally setting net.ipv4.tcp_tw_recycle=0 on cp1004

May 16

• 23:50 logmsgbot: aaron synchronized php-1.20wmf3/includes/upload/UploadBase.php 'deployed 4b0a61227fce37202da2b62b7dc2474bd227873f'
• 22:47 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use $wgSiteStatsAsyncFactor=1.'
• 22:32 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on testwikis.'
• 21:45 maplebed: reverted this morning's mobile push - tests completed
• 21:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36913 - Enable Collection on kkwiki'
• 21:09 logmsgbot: reedy synchronized php-1.20wmf3/cache/interwiki.cdb 'Updating interwiki cache'
• 21:09 logmsgbot: reedy synchronized php-1.20wmf2/cache/interwiki.cdb 'Updating interwiki cache'
• 20:05 binasher: ran ipv6 migrations on globalblocks
• 20:05 binasher: converted centralauth.globalblocks from myisam to innodb
• 19:57 binasher: rebooting db12 for kernel upgrade
• 19:51 binasher: stopping mysql on db12
• 19:50 binasher: recentchanges.rc_ip migration completed
• 19:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last bits of tidying up
• 19:44 binasher: rebooted db46
• 19:38 binasher: shutting down mysql on db46, preparing to reboot for kernel upgrade
• 19:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 19:28 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db59'
• 19:25 binasher: running recentchanges.rc_ip (ipv6) schema migration on enwiki master (5.2mil rows) via os������c - batten down the hatches!
• 19:17 binasher: running recentchanges.rc_ip (ipv6) schema migration on s2 dbs via os������c
• 19:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 19:14 Reedy: manually ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 "sudo -u mwdeploy rsync -a 10.0.5.8::common/*.dblist /usr/local/apache/common-local" because sync-dblist is woefully out of date..
• 19:13 notpeter: restarting ganglia on nickel
• 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 12 more misc/wikimedia wikis to 1.20wmf3
• 18:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All closed wikis to 1.20wmf3
• 18:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All special wikis to 1.20wmf3
• 18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikimedia wikis to 1.20wmf3
• 18:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisource to 1.20wmf3
• 18:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote to 1.20wmf3
• 18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiversity to 1.20wmf3
• 18:45 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktibooks to 1.20wmf3
• 18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf3
• 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikinews to 1.20wmf3
• 18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedia en projects to 1.20wmf3
• 18:27 binasher: running recentchanges.rc_ip (ipv6) schema migration on s3 dbs via os������c (s4 already completed during prior testing)
• 18:25 mutante: synced wikiversions.* files from NFS to spence local to prevent death of check_job_queue monitoring
• 18:21 binasher: running recentchanges.rc_ip (ipv6) schema migration on s5 dbs via os������c
• 18:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf3, again
• 18:17 logmsgbot: aaron synchronized php-1.20wmf3/includes/ImagePage.php 'deployed 86e2372772e618c5d1238ae480d9f632789bbe50'
• 18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf2
• 18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf3
• 18:10 binasher: running recentchanges.rc_ip (ipv6) schema migration on all s6 dbs via os������c
• 18:03 binasher: running recentchanges.rc_ip (ipv6) schema migration on all s7 dbs via os������c
• 17:43 binasher: ipblocks migration completed for all wikis
• 17:38 binasher: running ipblocks schema migration on all s2 dbs via os������c
• 17:35 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend 'Picking up fix for fatal in api in MobileFrontend at 9936e7a'
• 17:34 logmsgbot: awjrichards synchronized php-1.20wmf3/extensions/MobileFrontend 'Picking up fix for fatal in api in MobileFrontend at 9936e7a'
• 17:16 maplebed: deploying change to swift to make which containers write thumbs configurable
• 17:11 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'zero and mobile changes'
• 17:10 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'zero and mobile changes'
• 17:08 RobH: aluminum back online
• 17:00 binasher: running ipblocks schema migration on all s3 (819) dbs via os������c
• 16:59 binasher: running ipblocks schema migration on all s4 dbs via os������c
• 16:58 binasher: running ipblocks schema migration on s5/dewiki via osc
• 16:57 RobH: aluminum shut down for hard disk additions
• 16:56 binasher: running ipblocks schema migration on all s6 dbs via osc
• 16:51 RobH: udpating dns for osm web servers
• 16:50 binasher: running ipblocks schema migration on all s7 dbs via osc
• 16:49 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
• 16:41 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php 'deployed 634c3be2bba6a46e28aa997d7ab388ebf90b36a6'
• 16:31 maplebed: clearing the mobile varnish cache
• 16:29 maplebed: deploying gerrit change 7798 to the mobile varnish servers
• 07:41 logmsgbot: raindrift synchronized php-1.20wmf3/extensions/PageTriage/api/ApiPageTriageTemplate.php 'fixing exception bug that makes lots of logspam'
• 07:41 logmsgbot: raindrift synchronized php-1.20wmf2/extensions/PageTriage/api/ApiPageTriageTemplate.php 'fixing exception bug that makes lots of logspam'
• 06:20 Ryan_Lane: restarted lucene on search1015
• 05:58 Tim: setting net.ipv4.tcp_tw_recycle=1 on cp1005 seems to have fixed it, doing it on cp1004 as well now
• 05:52 Tim: on cp1005 setting tcp_tw_recycle=1
• 05:29 Tim: experimentally started squid on cp1004
• 04:05 hashar: updating a few plugins on Jenkins (host: gallium )
• 03:34 Ryan_Lane: stopped the squid process on cp1004 and stopped puppet to avoid it being restarted. it's having issues and I can't debug it right now.
• 03:22 Ryan_Lane: repooling squid frontend on cp1004
• 03:14 Ryan_Lane: depooling cp1004 and stopping the squid backend service to let some connections close
• 02:43 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 16 02:43:51 UTC 2012
• 01:10 logmsgbot: reedy synchronized php-1.20wmf3/extensions/RandomRootPage 'dark deploy randomrootpage extension (I'll enable it later)'

May 15

• 23:53 logmsgbot: aaron synchronized php-1.20wmf3/includes/Block.php 'deployed 7694faf68f975ea9c4888d575b33dabb84e90083'
• 23:42 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
• 23:27 ssmollett: upgraded ganglia-monitor and gmetad from 3.1.2-2.1 to 3.3.5-2
• 23:26 logmsgbot: awjrichards synchronizing Wikimedia installation... :
• 23:24 K4-713: upgraded minfraud version on the payments account
• 23:22 K4-713: updated and synchronized the payments cluster to DonationInterface d997e7ea1c
• 23:06 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
• 22:50 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Disable CentralAuth logging to file'
• 22:50 logmsgbot: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend to 0880467
• 22:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable CentralAuth logging to file'
• 22:13 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
• 21:09 logmsgbot: raindrift synchronizing Wikimedia installation... : PageTriage update
• 21:05 logmsgbot: raindrift synchronizing Wikimedia installation... : PageTriage update
• 19:40 logmsgbot: aaron synchronized wmf-config/swift.php 'disabled new hook for now.'
• 19:35 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Add wfDebugLog call'
• 19:31 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Add wfDebugLog call'
• 19:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php '$wgDebugLogGroups[updateTranstagOnNullRevisions] = udp://10.0.5.8:8420/updateTranstagOnNullRevisions'
• 19:26 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Only conditionally disable updateTranstagOnNullRevisions hook. Debugging to come'
• 17:45 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
• 17:41 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
• 17:35 logmsgbot: aaron synchronized wmf-config/swift.php 'Use new thumb purge hook for testwikis'
• 16:54 RobHalsell: updated apache config for wiki-pedia.org, seems the bot doesnt spam that anymore =[
• 16:36 mutante: srv app servers max. uptime with older kernel down to ~120 days after another bunch of upgrades
• 16:34 RobHalsell: updating dns for wiki-pedia.org
• 12:20 hashar: deployment-prep replaced most occurrences of /mnt/upload to /mnt/upload6
• 10:37 apergos: on db39 dropped triggers pt_osc_elwiki_recentchanges ins, del, upd, they were preventing all elwiki edits except bot edits with the complaint Table 'elwiki._recentchanges_new' doesn't exist ... binasher, doublecheck me please?
• 09:24 mutante: srv278 - still has issues as in reopnened RT #24 - upgrading kernel anyways
• 03:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'update wgUploadNavigationUrl on all cs wikis'
• 02:35 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 15 02:35:53 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Tue May 15 02:23:47 UTC 2012
• 01:09 logmsgbot: asher synchronized wmf-config/db.php 'returning db31 as an s4 slave'
• 01:05 logmsgbot: aaron synchronized php-1.20wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed f20e752630575f8384083f0ad0401e250c8babf5'
• 01:00 binasher: shutting down mysql on db31, then rebooting
• 00:59 logmsgbot: asher synchronized wmf-config/db.php 'pulling db31 from s4 for kernel upgrade'
• 00:58 binasher: new s4 master position - MASTER_LOG_FILE='db51-bin.000114', MASTER_LOG_POS=1772578
• 00:57 logmsgbot: asher synchronized wmf-config/db.php 'new s4 master'
• 00:55 logmsgbot: asher synchronized wmf-config/db.php 'setting s4 to read-only, switching master to db51'
• 00:54 binasher: preparing to rotate s4 master from db31 to db51
• 00:48 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bring in numerous shell requests from gerrit'
• 00:48 binasher: rebooting db51 for kernel upgrade, prior to promoting to s4 master
• 00:47 logmsgbot: asher synchronized wmf-config/db.php 'pulling db51 from s4 for kernel upgrade'
• 00:01 binasher: just completed an online schema change for commonswiki.recentchanges in prod. woo!

May 14

• 22:02 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
• 21:08 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Live hack out updateTranstagOnNullRevisions'
• 20:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf3
• 19:38 logmsgbot: reedy synchronizing Wikimedia installation... : Running scap to make sure everything is ok...
• 19:36 logmsgbot: reedy synchronized php-1.20wmf3/cache/l10n/ 'Resync localisation cache'
• 19:26 logmsgbot: reedy synchronized live-1.5/ 'Push live-1.5 new symlinks'
• 19:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf3
• 19:16 logmsgbot: reedy synchronized php-1.20wmf3/cache/trusted-xff.cdb
• 19:14 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf3.php
• 19:13 logmsgbot: reedy synchronized php-1.20wmf3/extensions/ 'Push extensions out properly'
• 19:11 binasher: resyncing cluster22 from es1002 to es1004
• 19:02 logmsgbot: reedy synchronized php-1.20wmf3/LocalSettings.php 'Use newer version'
• 19:01 logmsgbot: reedy synchronized php-1.20wmf2/LocalSettings.php 'Use newer version'
• 18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf3
• 18:57 logmsgbot: reedy synchronized php-1.20wmf3/cache/l10n/ 'Syncing localisation cache files'
• 18:49 Ryan_Lane: added OATHAuth to components list for MediaWiki Extensions product in bugzilla
• 18:43 Ryan_Lane: switching sessions back to memcached for labsconsole
• 18:42 Ryan_Lane: adding OATHAuth to labsconsole
• 18:40 Ryan_Lane: completed upgrade to 1.20wmf2 on labsconsole
• 18:30 Ryan_Lane: upgrading labsconsole to 1.20wmf2
• 18:26 logmsgbot: reedy synchronized php-1.20wmf3 'Initial pushing of php-1.20wmf3 files to apaches'
• 18:12 Reedy: Killing old php-1.20wmf1 directories from apaches to save full disks
• 13:48 mutante: copying outdated wikiversions.dat/.cdb files from /home to /usr/local on spence, which fixes check_job_queue (thanks jeremyb)
• 13:07 mutante: opening a bz bug for check_job_queue issue related to CommonSettings.php BZ:36835
• 07:43 mutante: still upgrading/rebooting a couple srv (API) application servers with long uptime
• 06:22 apergos: restarted lucene search on search1016 it had stopped doing anything useful (see ganglia graphs, also nothinig wtitten to logs)
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Mon May 14 02:22:09 UTC 2012

May 13

• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sun May 13 02:24:51 UTC 2012

May 12

• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sat May 12 02:22:18 UTC 2012

May 11

• 22:10 logmsgbot: preilly synchronized wmf-config 'add wikimedia to zero image disable list fix header'
• 21:52 logmsgbot: preilly synchronized wmf-config 'add wikimedia to zero image disable list'
• 19:49 Reedy: ran apache-graceful-all
• 19:42 RobH: apache restarted by puppet run on srv286
• 19:31 RobH: shutting down srv286 and srv286 for power rebalancing
• 19:23 RobH: srv260 and srv261 back in business
• 19:10 RobH: srv261 & srv261 shutting down for power rebalancing within the rack
• 18:33 notpeter: shutting down search 13-20 for hd upgrades
• 18:05 maplebed: swift: deleting the unsharded version of all sharded containers
• 18:03 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UploadWizard/ 'Deploy 4b5df1a1151ac80e309d396102e5e2a8d0c27ccb'
• 17:46 maplebed: deleted wikipedia-de-local-thumb container from swift. the sharded version is currently being used.
• 15:33 mutante: adding DNS entries for analytics hosts in new vlan 1121 (10.64.21.0/24), hosts starting at .101 to match names analytics1001 = .101 and ++
• 15:03 mutante: mw62 -unless somebody was on that right now it died. mgmt also just Create Instance Error
• 14:06 mutante: kernel upgrading / rebooting srv servers where uptime > 200 d order by uptime desc limit 1
• 13:12 mutante: installing package upgrades on pdf1-3 (and installed requested indic fonts via new puppet role class)
• 11:39 mutante: starting ms-be swift-container-auditors every once in a while
• 11:35 mutante: stat1 - installed new kernel, but waiting to reboot. schedule with aotto
• 11:24 mutante: upgrading packages/kernel on hooper, rebooting (Blog,Etherpad,Racktables)
• 09:21 mutante: ekrem was close running out of disk again. logrotated apache logs, changed config to: size 512M,rotate 3
• 08:58 mutante: package upgrades on ekrem (IRC server, WAP, Apple dict...)
• 08:51 mutante: rebooting marmontel (blog)
• 08:48 mutante: upgrading apache/mysql/kernel on marmontel (blog)
• 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 11 02:20:39 UTC 2012
• 02:00 RoanKattouw: Started Apache back up on srv200, done debugging
• 01:58 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UserDailyContribs/UserDailyContribs.hooks.php 'Deploy 3c45831ffe1817f3dc18f06644db46b1b74173e7'
• 01:17 RoanKattouw: Stopping Apache on srv200 so I can use it as my guinea pig for segfault debugging
• 00:56 logmsgbot: tstarling synchronized php-1.20wmf2/includes/User.php 'header log'
• 00:49 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:48 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:40 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:30 Tim: restarted socat on fenari so that fatal.log is reopened
• 00:29 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'removed logging hack tweaks.'
• 00:29 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'logging hack tweaks.'
• 00:27 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'removed some temp logging'
• 00:27 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:16 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:00 binasher: pulling cp1044 from lvs for testing

May 10

• 23:38 logmsgbot: reedy synchronized php-1.20wmf2/extensions/LiquidThreads/classes/Hooks.php 'Updating to master'
• 22:38 logmsgbot: catrope synchronized php-1.20wmf1/.git 'Make Special:Version show the correct commit now that I have fixed the weird repo state'
• 22:37 logmsgbot: catrope synchronized php-1.20wmf2/.git 'Make Special:Version show the correct commit now that I have fixed the weird repo state'
• 22:36 RoanKattouw: Cleaned up weird git repo states on fenari in php-1.20wmf1 and php-1.20wmf2
• 22:04 maplebed: swift: deleting the unsharded wikipedia-de thumb container contents (the sharded version is currently serving traffic)
• 19:51 notpeter: rebooting db29 for do a test install of precise
• 19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36420 - Wikipedia namespace alias for sr.wp'
• 19:02 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
• 18:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36694 - Set wgSitename on srwikisource'
• 18:43 LeslieCarr: restarting mobile varnish
• 18:33 LeslieCarr: reloaded and purged cache of mobile varnish
• 18:03 notpeter: starting innobackupex from db10 to blondel
• 17:39 notpeter: pushing out new zone files. only minor changes
• 16:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'showupdatemarker on enwiki tooooo'
• 03:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable show update markers on dewiki'
• 02:14 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Thu May 10 02:14:03 UTC 2012
• 01:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/7133/'
• 00:11 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UploadWizard 'Deploy b45437b6e09018dacfc78c8e4fa822a917858b2d / 62631485ba36f973c0d4a850ef494a8f84c4c86b'
• 00:11 logmsgbot: catrope synchronized php-1.20wmf1/extensions/UploadWizard 'Deploy b45437b6e09018dacfc78c8e4fa822a917858b2d / 62631485ba36f973c0d4a850ef494a8f84c4c86b'
• 00:06 logmsgbot: preilly synchronized wmf-config 'remove MF passwords'
• 00:01 logmsgbot: preilly synchronized wmf-config 'remove MF passwords'

May 9

• 23:33 notpeter: taking down search20 to do precise test-install
• 23:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Translate on outreachwiki'
• 23:25 Reedy: Created Translate tables on outreachwiki
• 22:49 Reedy: ExtensionDistributor fixed
• 22:32 Reedy: Debugging ExtensionDistributor being broken. Likely to show more debug output on mw.org if you attempt to use it (though, it wouldn't give you what you wanted anyway)
• 22:15 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 21:53 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed b9ac85cbf304a65d900cda00fafe53bf82d7a227'
• 21:53 logmsgbot: aaron synchronized php-1.20wmf2/includes/SiteStats.php 'deployed b9ac85cbf304a65d900cda00fafe53bf82d7a227'
• 20:52 LeslieCarr: done
• 20:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bump memory limit to 128MB'
• 19:39 Ryan_Lane: updating OpenStackManager on virt0 to master again
• 19:16 Ryan_Lane: updating OpenStackManager on virt0 to master
• 18:54 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed fa1a8d5119e1174f7458eb9516287f4867c46484'
• 18:50 RobH: dns update for db61 and db62
• 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 295 other wikipedias over to 1.20wmf2
• 18:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki to 1.20wmf2
• 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: ruwiki to 1.20wmf2
• 18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwiki to 1.20wmf2
• 18:11 notpeter: turning db30 back on
• 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.20wmf2
• 17:51 cmjohnson1: to shutting down storage3
• 16:58 LeslieCarr: restarted mobile varnish instances
• 16:58 LeslieCarr: flushed mobile varnish cache
• 16:54 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Make sure Swift backend will have journaling too.'
• 16:31 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Removed backend config conditional now that everything was switched over.'
• 14:06 mutante: started container-auditor on ms-be1
• 09:24 mutante: started container-auditor on ms-be3 and 4
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed May 9 02:37:02 UTC 2012
• 02:19 Reedy: Running cleanupUploadStash.php over all wikis
• 02:13 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 9 02:13:10 UTC 2012
• 01:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36506 - Site logo for Tsonga Wikipedia -- ts.wikipedia.org'
• 01:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36522 - Upload link should lead to UploadWizard instead of commons:Special:Upload'
• 01:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36663 - Please allow bureaucrats to add and remove autoreviewer status on pt.wiki'
• 01:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgShowUpdatedMarker enabled on anything that isn't enwiki or dewiki'
• 01:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36533 - Set sitename to Telugu Wiktionary'
• 01:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36595 - Please enable Extention:NewUserMessage on ml.wikipedia'
• 01:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36595 - Please enable Extention:NewUserMessage on ml.wikipedia'
• 01:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36571 - Please lock wikimania2011 wiki'
• 01:21 logmsgbot: reedy synchronized closed.dblist 'Closing wikimania2011wiki'
• 00:11 maplebed: started process to delete objects that don't exist in the container listings on all swift backends

May 8

• 23:44 K4-713: synchronized payments cluster to r115155, DonationInterface ccfbb304
• 23:34 LeslieCarr: purged varnish mobile cache
• 23:25 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobilefrontend resource version again'
• 23:25 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ 'd43f5f19ff3599f16200d247b6838cfb04ef1473'
• 23:25 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ 'd43f5f19ff3599f16200d247b6838cfb04ef1473'
• 23:22 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobilefrontend resource version'
• 23:11 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
• 23:11 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ '2b1e8573fdbcab0feb3a2481167b68fb96abf663'
• 23:10 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ '2b1e8573fdbcab0feb3a2481167b68fb96abf663'
• 22:53 RoanKattouw: Actually fixed it now with chmod -R g+w /h/w/conf/httpd
• 22:47 RoanKattouw: Fixed permissions in /h/w/conf/httpd by running find -group wikidev -not -perm 020 -exec chmod g+w \{\} \;
• 22:38 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/stylesheets/sections.css 'Live hack to live test broken interface on ICS devices on very large articles'
• 22:34 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile url transformation on testwiki'
• 22:13 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'bumping MobileFrontend resource version number'
• 22:13 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ 'd828a8196d8bc877afdbd1559e8e6d639b51cef7'
• 22:12 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ 'd828a8196d8bc877afdbd1559e8e6d639b51cef7'
• 21:53 binasher: rebooting db1018 one more time
• 21:47 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed 43aa35016b03935b27d439afe9a6b3f1aad1aa8b'
• 21:45 Ryan_Lane: adding adminbot to the repo
• 21:32 binasher: rebooting eqiad core db slaves for kernel upgrade
• 21:29 logmsgbot: aaron synchronized wmf-config/swift.php 'Added new thumbnail purge/import hooks handlers that use the swift backend class; unused atm.'
• 21:23 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Added swift backend config; unused atm.'
• 21:15 logmsgbot: asher synchronized wmf-config/db.php 'returning db45 to service'
• 21:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
• 21:13 maplebed: delpoyed container sharding for thumbnails to swift for 'dewiki', 'fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki' (in addition to existing sharding for commons and enwiki)
• 21:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move wikimania2013wiki to php-1.20wmf2
• 21:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
• 21:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
• 21:10 binasher: shutting down mysql across all eqiad core db slaves
• 20:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'logo for wikimania2013wiki'
• 20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'remove w'
• 20:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on wikimania2013wiki'
• 20:56 logmsgbot: aaron synchronized wmf-config/swift.php 'Switching purge hook to use new sharding scheme.'
• 20:54 Reedy: Created translate related tables for wikimania2013wiki
• 20:31 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AntiBot/
• 20:30 logmsgbot: reedy synchronized php-1.20wmf2/extensions/AntiBot/
• 20:14 maplebed: creating sharded containers for swift for 'dewiki','fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki'
• 19:54 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Moved remaining wikis over to new backend config'
• 19:34 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/
• 19:12 LeslieCarr: flushed mobile varnish cache
• 19:11 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/
• 19:10 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/
• 18:37 LeslieCarr: reenabled services on fpc5 of cr1-eqiad
• 18:16 cmjohnson1: updating md1000 controller card firmware on storage3
• 18:14 LeslieCarr: turned off fpc5 on cr1-eqiad to swap
• 18:05 LeslieCarr: powering on fpc 5 on cr1-eqiad
• 18:03 LeslieCarr: powering off fpc5 on cr1-eqiad in order for RobH to physically reseat the card
• 17:48 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable TranslationNotifications on meta, incubator and wikimania2012'
• 17:44 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable TranslationNotifications on mediawikiwiki'
• 17:42 LeslieCarr: switching all masterships over to cr2-eqiad in preparation to reseat cr1 linecard
• 17:25 LeslieCarr: flushed the mobile cache
• 17:24 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding localisation cache files before TranslationNotification deploy
• 17:18 logmsgbot: reedy synchronized php-1.20wmf2/extensions/MobileFrontend/ 'Pushing out head'
• 17:16 logmsgbot: reedy synchronized php-1.20wmf1/extensions/MobileFrontend/ 'Pushing out head'
• 17:14 RobH: asw-c1-eqiad connected to both cr1 and cr2
• 15:16 cmjohnson1: shutting down storage3 to replace raid card
• 12:40 pp-pdf1: updated mwlib to 0.13.7
• 12:39 pp-pdf2: updated mwlib to 0.13.7
• 12:36 pp-pdf3: updated mwlib to 0.13.7
• 11:59 mutante: merging CSS fix for broken mobile site table layout
• 02:18 RoanKattouw: Removed and recloned /var/lib/l10nupdate/mediawiki/extensions , it was in a weird state because magic extension submodules work now but my hacky workaround for them not working was still in place
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 01:12 logmsgbot: tstarling synchronized php-1.20wmf2/includes/api/ApiMain.php
• 01:10 logmsgbot: tstarling synchronized php-1.20wmf1/includes/api/ApiMain.php
• 00:44 binasher: rebooted db1034
• 00:42 logmsgbot: tstarling synchronized php-1.20wmf2/includes/Exception.php
• 00:42 logmsgbot: tstarling synchronized php-1.20wmf2/includes/DefaultSettings.php
• 00:37 logmsgbot: tstarling synchronized php-1.20wmf1/includes/Exception.php
• 00:36 logmsgbot: tstarling synchronized php-1.20wmf1/includes/DefaultSettings.php
• 00:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Switched enwiki to new backend config.'

May 7

• 23:52 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobilefrontend resource version #'
• 23:44 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/
• 23:43 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PageTriage/
• 23:35 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/includes/PageTriageUtil.php
• 23:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgShowExceptionDetails to true for testwiki and test2wiki'
• 23:25 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07, take 3
• 23:15 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PageTriage/includes/PageTriageUtil.php
• 23:15 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/includes/PageTriageUtil.php
• 23:00 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'wgShowExceptionDetails = false'
• 22:57 Ryan_Lane: restarting glusterd processes on virt1-5
• 22:56 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobile resource version'
• 22:54 Ryan_Lane: upgrading glusterfs on virt1-5
• 22:49 Ryan_Lane: upgrading glusterfs on labstore1-4
• 22:48 binasher: running an osc against plwiktionary.recentchanges on master
• 22:40 paravoid: deleting 14k tmp files from spence's /home/nagios
• 22:35 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07
• 22:34 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07
• 22:24 RoanKattouw: chmod 775 /usr/local/apache/common-local/php-1.20wmf2/extensions/PageTriage with dsh as root
• 22:19 logmsgbot: raindrift synchronized php-1.20wmf1/resources/startup.js 'touch'
• 22:18 binasher: rebooting nfs2 to new kernel
• 22:16 logmsgbot: raindrift synchronized wmf-config/InitialiseSettings.php 'enabling PageTriage on enwp'
• 22:14 logmsgbot: raindrift synchronized php-1.20wmf2/extensions/PageTriage 'Syncing PageTriage to enwp, a la carte'
• 22:14 logmsgbot: raindrift synchronized php-1.20wmf1/extensions/PageTriage 'Syncing PageTriage to enwp, a la carte'
• 21:59 mutante: was still upgrading/rebooting amssq* and knsq* hosts on the side (slow,b/c upload squids). expect temp. nagios squid reports tomorrow as well. out for now.
• 21:44 binasher: moved default resolution for upload from eqiad to pmtpa
• 21:29 cmjohnson1: shutting down storage3 for troubleshooting
• 20:37 binasher: attempting a live online schema change for zuwikitionary.recentchanges on the prod master
• 20:22 LeslieCarr: (above) restarted nagios-wm on spence
• 20:20 LeslieCarr: restarted irc bot
• 20:15 binasher: rebooting db45
• 20:11 binasher: rebooting db1019
• 18:46 logmsgbot: reedy synchronized php-1.20wmf1/extensions/Collection/Collection.session.php 'head'
• 18:45 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection/Collection.session.php 'head'
• 18:25 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php
• 18:24 logmsgbot: reedy synchronized php-1.20wmf1/extensions/GlobalBlocking/GlobalBlocking.class.php
• 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf2
• 16:16 cmjohnson1: shutting down storage3 to reseat RAID card
• 15:58 cmjohnson1: Going to power cycling storage3 several times to troubleshoot hardware issue
• 15:15 RobH: updating firmware on storgae3
• 14:20 Jeff_Green: stopped cron jobs on storage3 because of RAID failure
• 12:49 mutante: pushing out virtual host for wikimania2013 wiki. sync / apache-graceful/all
• 11:18 mutante: continuing with upgrades/reboots in amssq* on the side during the day
• 11:09 mutante: squids - sq* done. all latest kernel and 0 pending upgrades.
• 09:27 mutante: rebooting bits varnish sq68-70 one by one..
• 08:00 mutante: upgrading/rebooting the last couple sq* servers
• 07:20 binasher: power cycled db45 (crashed dewiki slave)
• 07:05 logmsgbot: asher synchronized wmf-config/db.php 'db45 is down'
• 02:25 Tim: on locke: introduced 1/100 sampling for banner impressions, changed filename to bannerImpressions-sampled100.log
• 02:12 Tim: on locke: moved fundraising logs back where they were
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 01:38 Tim: on locke: compressing bannerImpressions.log
• 01:35 Tim: on locke: moved bannerImpressions.log to archive and restarted udp2log
• 01:26 Tim: on locke: moved fundraising logs from /a/squid/fundraising/logs to /a/squid so that they will be processed by logrotate

May 6

• 07:03 apergos: manually rotates udplogs on locke, copying destined_for_storage3 off to hume:/archive/emergencyfromlocke/ (jeff, this note's for you in particular)
• 06:36 apergos: bringing up storage3 with neither /a nor /archive mounted, saw "The disk drive for /archive is not ready yet or not present" etc on boot, waited a long time, finally skipped them
• 06:12 apergos: and powercycling the box instead. grrrr
• 06:05 apergos: rebooting storage3: we have messages like May 6 05:45:12 storage3 kernel: [465081.410025] Filesystem "dm-0": xfs_log_force: error 5 returned. in the log, and the raid is unaccessible, megacli doesn't run either
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

May 5

• 09:37 mutante: squids - upgrading in the sq5x range (upload)
• 08:53 apergos: disabling modcompress temporarily for lightty on dataset2 (live hack), let's see what that does as far as it dying. could be issue similar to http://redmine.lighttpd.net/issues/2391
• 06:45 mutante: squids - upgrading sq44,48 (upload)
• 05:23 mutante: squids - finishing a couple reboots in the sq7x range
• 03:04 binasher: rebooting db1006 as well
• 03:04 binasher: rebooting db1038, kernel uptime scheduler chaos
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 00:21 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php

May 4

• 23:46 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php
• 23:45 logmsgbot: reedy synchronized php-1.20wmf1/extensions/GlobalBlocking/GlobalBlocking.class.php
• 22:35 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/backend/FSFileBackend.php 'deployed a807624'
• 22:34 LeslieCarr: clearing varnish cache and reloading varnish on mobile
• 21:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 21:13 logmsgbot: reedy ran sync-common-all
• 20:18 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Fix typo (cswikquote vs cswikiquote)'
• 20:06 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 writable'
• 20:05 binasher: performing mysql replication steps for s2 master switch to db52
• 20:04 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 read-only, db52 (still ro) as master, db13 removed'
• 19:49 logmsgbot: asher synchronized wmf-config/db.php 'setting db52 weight to 0 in prep for making new s2 master'
• 19:32 binasher: powering off db24
• 18:08 LeslieCarr: reloaded mobile varnish caches and purged them
• 18:02 Ryan_Lane: gerrit upgrade is done
• 17:55 Ryan_Lane: starting gerrit
• 17:32 Ryan_Lane: installing gerrit package on manganese
• 17:28 Ryan_Lane: adding gerrit 2.3 package to the repo
• 17:25 Ryan_Lane: shutting down gerrit so that everything can be backed up
• 16:45 apergos: lighty on dataset2 is running under gdb in screen session as root, if it dies please leave that alone (or look at it if you want to investigate)
• 16:26 notpeter: turning off db30 (former s2 db, still on hardy, will ask asher what to do with it) to test noise in DC
• 15:50 mutante: rebooting sq67 (bits)
• 15:42 mutante: going through sq7x servers (text), full upgrades
• 15:32 notpeter: removing srv281 from rending pool until we figure out what's going on with it
• 15:23 notpeter: putting srv224 back into pybal pool
• 15:09 notpeter: removing srv224 from pybal pool for repartitioning
• 14:56 notpeter: putting srv223 back into pybal pool
• 14:50 mutante: going through sq6x (text), full upgrades
• 14:08 notpeter: removing srv223 from pybal pool for repartitioning
• 14:02 notpeter: putting srv222 back into pybal pool
• 13:50 notpeter: removing srv222 from pybal pool for repartitioning
• 13:43 notpeter: putting srv221 back into pybal pool
• 13:30 notpeter: removing srv221 from pybal pool for repartitioning
• 13:16 mutante: going through sq80 to sq86 (upload), full upgrade & reboot
• 12:56 mutante: maximum uptime in the sq* group down to 171 days, so we have like a month now for the rest. stopping upgrades for the moment being.
• 12:54 notpeter: starting script to move /usr/local/apache to /a partition on all remaing non-imagescaler apaches
• 12:47 mutante: (just) new kernels & reboot - sq45,sq49 (upload)
• 12:30 mark: Sending ALL non-european upload traffic to eqiad
• 12:23 mutante: (just) new kernels & reboot - sq63 to sq66 (209 days up)
• 12:06 mutante: dist-upgrade & kernel & reboot - sq42,sq43 - rebooting upload squids one by one
• 11:48 mutante: powercycling srv266 one more time, but now creating RT for it, once already showed CPU issue before it was reinstalled recently
• 11:13 apergos: restarted lighty on dataset2 ... about ... half an hour ago. stupid case sensitivity
• 10:02 apergos: tossed knsq1 through 7 from squid_knams dsh nodegroups file, prolly lots more cleanup where that came from
• 09:34 mutante: dist-upgrade/kernel/reboot: sq37, sq41. rebooting upload squid sq41
• 08:49 mutante: dist-upgrade & new kernel & reboot: sq33, sq36
• 07:47 mutante: preemptive rebooting of sq* servers identified as having > 200 days of uptime
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 4 02:22:42 UTC 2012
• 02:13 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Fri May 4 02:13:58 UTC 2012
• 00:20 logmsgbot: raindrift synchronizing Wikimedia installation... :
• 00:18 logmsgbot: raindrift synchronizing Wikimedia installation... : Syncing the PageTriage extension, but only enabling on testwiki
• 00:08 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Adding 'fr' to language codes for mobile feedback'
• 00:06 maplebed: moved ms1-3 from the production cluster to the test cluster

May 3

• 23:29 LeslieCarr: restarting networking on sq55
• 23:29 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 23:27 LeslieCarr: restarting networking on sq54
• 23:24 LeslieCarr: restarting networking on sq53
• 23:21 LeslieCarr: restarting networking on sq52
• 23:16 LeslieCarr: restarting networking on sq51
• 21:30 notpeter: removing srv220 from pybal pool for repartitioning
• 21:29 LeslieCarr: switching asw-a4-sdtpa from single uplink to lag
• 21:19 notpeter: putting srv219 back into pybal pool
• 21:14 logmsgbot: asher synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23'
• 21:09 logmsgbot: asher synchronized wmf-config/db.php 'reverting cluster23 change'
• 21:05 logmsgbot: asher synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23'
• 21:02 binasher: about to move ES writes to cluster23
• 20:47 notpeter: removing srv219 from pybal pool for repartitioning
• 20:37 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection/Collection.templates.php
• 20:37 logmsgbot: reedy synchronized php-1.20wmf1/extensions/Collection/Collection.templates.php
• 19:50 binasher: restarted profiling collector post parser.php livehack and stats.db removal
• 19:45 notpeter: starting script to move /usr/local/apache to /a partition on all non-imagescaler, non-jobrunner apaches
• 19:42 logmsgbot: aaron synchronized php-1.20wmf2/includes/parser/Parser.php 'live-hack out template profiling...again.'
• 19:40 logmsgbot: aaron synchronized php-1.20wmf1/includes/parser/Parser.php 'live-hack out template profiling...again.'
• 19:31 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert $wgDefaultUserOptions[enotifwatchlistpages] = 1'
• 19:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 36316 - Set Add pages I edit to my watchlist to true by default for new users'
• 19:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php '$wgDefaultUserOptions[enotifwatchlistpages] = 1'
• 19:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable show update markers for some more of the larger wikis'
• 19:00 paravoid: powercycling all of sq51-sq62, hanged due to 209 days uptime
• 18:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36092 - Activation of flood flag on vec.wikipedia.org'
• 18:43 paravoid: powercycling sq59; inaccessible via either SSH or serial due to load
• 18:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36183 - Fix namespace alias on Hindi Wikipedia'
• 18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36171 - Imports from Wikibooks'
• 18:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
• 18:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
• 18:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
• 18:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36386 - cswikiquote user group changes'
• 18:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36480 - Create namespace Comments: in Greek Wikinews'
• 17:44 RobH: db1029 ssd test items removed, can go back to normal service via asher
• 17:43 notpeter: returning mw58 to pool
• 17:34 RobH: shutting down db1029 for ssd card testing removal per rt 2766
• 17:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36320 - Set $wgShowUpdatedMarker back to true on ptwiki'
• 17:18 notpeter: removing mw58 from pool for more testin'
• 17:16 LeslieCarr: reloaded and purged varnish cache for mobile in eqiad
• 17:03 notpeter: mwm59 out of apache pool. using it for some testing
• 16:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36359 - Add namespace 102 to $wgContentNamespaces on ptwiki Bug 36360 - Add namespace 102 to $wgNamespacesToBeSearchedDefault on ptwiki'
• 16:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 36460 - Enable chunked uploads as opt-in user preference'
• 16:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 31406 - Set $wgUseMathJax = true on Wikimedia wikis'
• 15:12 notpeter: chris is taking down search1-12 to replace with new search nodes
• 15:05 mutante: powercycling srv266
• 13:49 mark: Built new wikimedia-base 1.00 package, stripped of most stuff now handled by Puppet, and inserted it into the lucid-wikimedia and precise-wikimedia APT repositories
• 10:33 mutante: starting container-auditor on ms-be3
• 08:42 logmsgbot: ariel synchronized php-1.20wmf2/LocalSettings.php 'job runners don't have /home mounted'
• 08:16 Nemo_bis: siebrand: job queue stuck, on en.wiki jumped from o to 37k in the last ~36h
• 04:52 jeremyb: fixed complaints of beta simplewiki appearing in #cvn-simplewikis on freenode on the labs side. details
• 04:00 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:47 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:38 Tim: fixed scap, was failing on the remote side due to mwversionsinuse exiting with status 1 due to /home/wikipedia/common not existing on apaches
• 02:21 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:21 Tim: aborted scap and re-ran with fanout=5 instead of 30, since nfs1 CPU was maxed out
• 02:14 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:04 logmsgbot: aaron synchronized multiversion/activeMWVersions 'deployed r115116'
• 02:00 logmsgbot: LocalisationUpdate failed (php-1.20wmf2) at Thu May 3 02:00:13 UTC 2012
• 02:00 logmsgbot: LocalisationUpdate failed (php-1.20wmf1) at Thu May 3 02:00:12 UTC 2012

May 2

• 23:56 logmsgbot: aaron synchronized multiversion/ 'deployed svn HEAD'
• 23:41 maplebed: started swift old-object-deleter on ms-be3
• 23:28 maplebed: update - roan takes the blame
• 23:28 logmsgbot: raindrift synchronized wmf-config/InitialiseSettings.php 'Aborting todays PageTriage deployment'
• 23:22 maplebed: swift is recovered; ~20 minutes of impaired service. cause unknown, but the swiftcleaner looks likely.
• 23:18 RoanKattouw_away: Scap tried to push two new source trees to php-1.20wmf1-* and php-1.20wmf2-* , causing full disks. Cleaning up now
• 23:13 LeslieCarr: restarting nagios bot
• 22:59 logmsgbot: raindrift synchronizing Wikimedia installation... :
• 22:49 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'contact us change'
• 22:48 logmsgbot: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/ 'contact us change'
• 21:43 logmsgbot: asher synchronized wmf-config/db.php 's2: pulling db30, raising weights on new hosts'
• 21:02 ^demon: finished database maintenance on db9.reviewdb
• 20:24 hashar: hashar: updated TestSwarm to distribute tests to Firefox 12 users.
• 20:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Re-pushing for srv219 and srv220
• 20:07 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 20:04 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special, wikimedia, wikiquote, wikiversity, and wiktionary wikis to 1.20wmf2
• 19:59 logmsgbot: asher synchronized wmf-config/db.php 'adding dbs 52,53,57 to s2 at lower weights'
• 19:55 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 19:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.20wmf2
• 19:40 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf2
• 19:36 preilly: fix for PHP Warning: in_array() expects parameter 2 to be array, string given in /usr/local/apache/common-local/php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php on line 156
• 19:36 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/skins/SkinMobile.php 'fix php notice for in_array'
• 19:35 logmsgbot: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php 'fix php notice for in_array'
• 19:34 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikibooks to 1.20wmf2
• 19:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwikibooks to 1.20wmf2
• 19:21 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved sourceswiki to 1.20wmf2
• 19:20 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 19:11 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikisource sites to 1.20wmf2
• 19:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikinews sites to 1.20wmf2
• 19:00 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'only remove images for DIGI'
• 18:51 logmsgbot: asher synchronized wmf-config/db.php 'added ES cluster23 to templateOverridesByCluster but not activating'
• 18:48 binasher: creating a blobs_cluster23 ES shard table for all active projects
• 18:31 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikisource to 1.20wmf2
• 18:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf2
• 18:24 RobH: updating dns
• 18:20 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf2
• 18:09 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'only remove images for DIGI'
• 17:57 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 17:56 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 17:46 K4-713: updated production civicrm to r1726
• 17:36 logmsgbot: aaron synchronized php-1.20wmf2/includes/specials/SpecialContributions.php 'Deployed 799998c3a160ef6dd3b926b7d6fec223682b788c'
• 17:30 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 17:28 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 17:14 logmsgbot: catrope synchronized php-1.20wmf2/skins/vector/ 'Deploying 7260cc5fe4071e03241378ba1a48bc0b6f188948'
• 16:51 RoanKattouw: Changing docroot/bits/skins-1.19 and other 1.19 symlinks to point to the 1.20wmf1 tree instead. This is needed because we're still getting requests for magnify-clip.png at the 1.19 URL from cached HTML
• 16:16 notpeter: starting innobackupex from db1040 to db1022 for new s6 snapshot slave
• 15:31 notpeter: no nagios bot, kicking nagios on spence
• 15:04 RobH: shutting down mw64 for hw test per rt 1890
• 15:03 RobH: bellin crashed, unresponsive to ssh or serial console
• 14:43 mark: Built varnish for precise as 3.0.2-2wm5 and imported it into APT repository precise-wikimedia
• 11:52 mark: Started distribution upgrade of server stafford from Lucid to Precise
• 10:41 mutante: refreshLinks.php - started it once again in a screen on hume, just for s1. last cron failed with "mwscript command not found"?? well now it is there again and running
• 10:09 mark___: Started distribution upgrade of server sockpuppet from Lucid to Precise
• 09:20 mutante: upgrading bugzilla to 4.0.6
• 08:43 mutante: kaulen: installing various upgrades (apache,mysql,cron,php-wikidiff2,...)
• 08:40 logmsgbot: hashar synchronized php-1.20wmf2/includes/GitInfo.php 'Fix Special:Version for 1.20wmf2 (commit ae12df0 , bug 36361 )'
• 08:20 hashar: cherry-picked ae12df0 commit to 1.20wmf2 since there are mobilefrontend commits pending.
• 02:35 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 2 02:35:51 UTC 2012
• 02:32 K4-713: updated production civicrm to r1723
• 02:13 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed May 2 02:13:30 UTC 2012
• 01:01 notpeter: starting innobackupex from db57 to db53 for new s2 slave for the one zillionth time

May 1

• 22:28 logmsgbot_: asher synchronized wmf-config/db.php 'returning db45'
• 22:23 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db45, last coredb on prior fb mysql build'
• 22:17 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable doublepage on test2wiki'
• 22:11 binasher: upgraded percona-toolkit on coredbs to 2.1.1 - now with the potential to run online schema changes on tables without single column unique keys!!
• 21:39 binasher: created an ops db on all core mysql shards
• 21:00 notpeter: reinstalling db53. this time with correct raid!
• 20:40 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Fixing mailto links on mobilefrontend feedback form to properly populate subject lines'
• 19:32 LeslieCarr: reverting vrrp mastership of row a to cr2-eqiad
• 19:29 LeslieCarr: switching vrrp mastership of row a to cr1-eqiad
• 18:32 logmsgbot_: awjrichards synchronized wmf-config/InitialiseSettings.php 'Make testwiki use mobile domain for URLs'
• 18:28 LeslieCarr: making routing change, higher risk
• 17:51 Ryan_Lane: make that virt0
• 17:51 Ryan_Lane: switching the session cache back to filesystem on virt1, since it isn't working properly with memcache
• 17:29 maplebed: kicking nagios to check a change to fix the mobile LVS alert
• 17:25 logmsgbot_: nikerabbit synchronized php-1.20wmf2/extensions/TranslationNotifications/ 'Deploying TranslationNotifications code'
• 17:08 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf2
• 16:27 notpeter: starting innobackupex from db1034 to db53 for new s2 slave
• 16:27 notpeter: starting innobackupex from db57 to db52 for new s2 slave
• 16:03 notpeter: rebuilding db52 and db53 as s2 slaves
• 15:47 logmsgbot_: asher synchronized wmf-config/db.php 's1: raising db59,60 weights, pulling db52/53 for reuse'
• 09:23 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'hewiki account creation high throttle limits'
• 04:04 Tim: on all apaches, running "chmod -R a+rX /usr/local/apache/common-local/" to clean up after killed rsyncs which left files unreadable
• 02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf2) at Tue May 1 02:23:29 UTC 2012
• 02:21 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileFeedback.php
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue May 1 02:14:06 UTC 2012
• 02:06 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileOptions.php
• 01:51 Ryan_Lane: bringing up all labs instances with a 60 second lag
• 01:40 Ryan_Lane: rebooting virt0
• 01:35 Ryan_Lane: rebooting virt3
• 01:33 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/HtmlFormatter.php
• 01:26 Ryan_Lane: rebooting virt5
• 01:18 Ryan_Lane: rebooting virt4
• 01:03 Ryan_Lane: rebooting virt2
• 00:51 LeslieCarr: restarted swift-container-auditor on ms-be5
• 00:38 logmsgbot_: tstarling synchronizing Wikimedia installation... :
• 00:26 Tim: removed large syslogs from mw60 and ran sync-common
• 00:18 Tim: on mw60 there was an actual directory at /usr/local/apache/common/php where a symlink should have been. fixed

April 30

• 23:58 logmsgbot_: aaron synchronized php
• 23:44 RoanKattouw: Started Apache back up on mw60
• 23:39 RoanKattouw: Running scap-1 on the Apaches with dsh
• 23:38 RoanKattouw: Moved /home/catrope/php-1.19 to /home/wikipedia/lazy-backups/php-1.19
• 23:38 Reedy: mediawiki.org to 1.20wmf2
• 23:37 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mw.org to 1.20wmf2
• 23:35 RoanKattouw: Strike that, instead moving /home/w/common/php-1.19 to /home/catrope/php-1.19
• 23:34 RoanKattouw: Removing /home/w/common/php-1.19 , NFS might freak out a bit
• 23:31 RoanKattouw: Removed php-1.19 from mw60 , synced it, and restarted Apache
• 23:28 RoanKattouw: Synced docroot and purged varnish for static-1.20wmf2, bits seems to be working for 1.20wmf2 now
• 23:27 RoanKattouw: mw60 has full disk, stopping Apache for now
• 22:50 Ryan_Lane: rebooting virt5
• 22:42 Ryan_Lane: rebooting virt3
• 22:35 Ryan_Lane: rebooting virt4
• 22:28 Ryan_Lane: rebooting virt1
• 22:23 Ryan_Lane: bringing down all instances (yay gluster)
• 21:12 pgehres: re-enabled Jenkins jobs on Aluminium after db1008 reboot
• 21:11 pgehres: CiviCRM back to normal after db1008 reboot
• 21:07 Jeff_Green: db1008 gets kernel update and reboot
• 21:00 pgehres: put CiviCRM on Aluminium in maintenance mode for db1008 reboot
• 20:59 logmsgbot_: reedy synchronized php-1.20wmf2/resources/startup.js 'touch'
• 20:57 pgehres: disabled all Jenkins jobs on Aluminium in prep for db1008 reboot
• 20:50 Jeff_Green: db1025 and storage3 get new kernels and reboot
• 20:28 notpeter: restarting, once again, innobackupex from db1034 to db57 for new s2 slave after fenari crash killed my screen
• 20:24 Reedy: Running ddsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 '/usr/bin/scap-1' in the hope it syncs all the files that would be nice to be on the app servers
• 20:18 logmsgbot_: reedy synchronized php-1.20wmf2/cache/ 'Synching whole cache directory'
• 19:59 notpeter: restarting nagios to get rid of some old checks
• 19:57 Jeff_Green: payments cluster gets kernel updates and reboots
• 19:55 logmsgbot_: reedy synchronizing Wikimedia installation... : Rebuiild l10n for 1.20wmf2
• 19:49 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.20wmf2.php 'Syncing file'
• 19:49 logmsgbot_: reedy synchronized php-1.20wmf2/LocalSettings.php 'Pushing LocalSettings.php'
• 19:48 paravoid: upgraded & rebooted ssl3001, ssl3002, ssl3003
• 19:45 logmsgbot_: reedy synchronizing Wikimedia installation... : Pushing out new symlinks etc, moving test2wiki to 1.20wmf2
• 19:30 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 live hack revisions'
• 19:28 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf1 live hack revisions'
• 19:26 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 for deployment'
• 19:18 Reedy: Syncing php-1.20wmf2 files from NFS to apaches. Likely to upset NFS (or the uplink for the switch nfs is on) for a little while...
• 19:14 paravoid: rebooting ssl1004
• 19:06 paravoid: rebooting ssl1003
• 19:00 paravoid: rebooting ssl1002
• 18:59 notpeter: starting innobackupex from db1034 to db57 for new s2 slave
• 18:50 paravoid: rebooting ssl1001
• 18:42 Jeff_Green: grosley gets new kernel + reboot
• 18:35 Jeff_Green: aluminium gets kernel update, yayyyyyyy!
• 18:34 paravoid: pooled back ssl1; depooling ssl3 and rebooting
• 18:29 binasher: rebooting mw45 for kernel upgrade
• 18:27 Jeff_Green: power cycling aluminium which faceplanted
• 18:22 binasher: rebooting mw45
• 18:21 notpeter: rebuilding db57 again, this time with more correct raid level!
• 18:19 logmsgbot_: asher synchronized wmf-config/db.php 'adding db59,60 to s1 with low weights'
• 18:16 paravoid: depooled & rebooting ssl1
• 18:09 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Sanity run after script changes.
• 18:00 logmsgbot_: aaron synchronized multiversion
• 17:58 logmsgbot_: reedy synchronized php-1.20wmf1/includes/MagicWord.php 'https://gerrit.wikimedia.org/r/6135'
• 17:44 logmsgbot_: aaron synchronized wikiversions.cdb
• 17:43 AaronSchulz: updating multiversion code
• 08:34 mutante: reinstalling srv266
• 08:08 mutante: upgraded mw1,mw2,mw35
• 07:59 mutante: reinstalling srv206
• 07:50 mutante: upgrading mw36
• 07:37 apergos: powercycling srv266, had this message on mgmt console: Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted
• 07:22 mutante: installing upgrades on srv212
• 07:19 apergos: reinstalled srv284, seems to be up now
• 07:17 mutante: powercycled mw8
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Mon Apr 30 02:13:59 UTC 2012

April 29

• 20:13 apergos: srv206 won't run puppet, see syslog, clearing out the yaml file didn't help, since it's not urgent I'm leaving it for tomorrow
• 19:51 Ryan_Lane: depooling ssl3004
• 19:51 Ryan_Lane: removed the ipv6 addresses from maerlant and added them to ssl3001, then restarted nginx
• 19:50 Ryan_Lane: repooling ssl3001
• 19:46 apergos: powercycled mw60, same reason as the rest
• 19:12 apergos: power cycled mw48 and mw52 (hung just like the others)
• 18:05 apergos: sll3002 and 3003 were rebooted and are the entire ssl esams pool right now
• 18:02 apergos: ok the ssl300x situation: ssl3001 is now disabled in the pybal conf file on fenari; it is picking up the ipv6and4labs tmplate and I don't know if that's right, anyways nginx doesn't want to bind to one of those addresses. ssl3004 isn't reachable or pingable even via mgmt but at leasy lvs sees it's gone
• 16:34 apergos: powercycling the ssl300x.esams hosts. 212 days of uptime... (and 3001 had gone out to lunch)
• 12:34 mutante: and finally mw1, so just leaving mw1102 and mw60 for having other issues for a while (->Nagios)
• 12:22 mutante: check_all_memcached recovered, but still same treatment for mw10 and 11 (8 and 15h ago)
• 12:15 mutante: powercycling mw32,mw33,mw44,mw46 one by one, they were all frozen and went down between like 17 and 24 hours ago approx.
• 12:07 mutante: powercycling mw30
• 02:56 paravoid: rebooting ssl2 (has 214 days uptime)
• 02:47 paravoid: powercycled ssl3
• 02:13 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sun Apr 29 02:13:58 UTC 2012

April 28

• 22:53 Reedy: Job queue logs on gdash seem to have stopped on the 26th...
• 22:29 logmsgbot_: reedy synchronized php-1.20wmf1/includes/EditPage.php 'https://gerrit.wikimedia.org/r/6088'
• 21:52 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
• 21:51 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
• 21:12 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
• 21:10 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
• 21:09 logmsgbot_: reedy synchronized common/php-1.20wmf1/extensions/cldr/LanguageNames.body.php 'more debugging'
• 20:51 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php 'Add debugging'
• 20:49 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Add debuglog group for language code not being a string'
• 19:04 logmsgbot_: reedy synchronized php-1.20wmf1/includes/ExternalEdit.php 'https://gerrit.wikimedia.org/r/6077'
• 19:03 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api/ApiParse.php 'https://gerrit.wikimedia.org/r/6076'
• 02:24 Ryan_Lane: rebooting all mediawiki boxes that have uptimes affected by the bug are being rebooted at 8 minute intervals
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 28 02:14:14 UTC 2012
• 01:33 paravoid: powecycled mw29
• 01:21 paravoid: powercycled mw38
• 00:17 notpeter: db12 is sooooo sloooooow, starting innobackupex from db1017 to db60 for new s1 slave

April 27

• 22:15 paravoid: upgraded ssl4 to nginx 0.7.65-5wmf1 and added it back to the pool
• 21:45 paravoid: rebooting ssl4 after upgrading (incl. a kernel update)
• 20:00 notpeter: starting innobackupex from db1040 to db1022 for new eqiad s6 snapshot slave, again
• 19:59 notpeter: starting innobackupex from db12 to db60 for new s1 slave, again
• 19:58 notpeter: starting innobackupex from db1017 to db59 for new s1 slave, again
• 19:49 paravoid: de-pooling ssl4
• 19:30 mutante: test - added new gerrit interwiki prefix for SAL/wikitech - gerrit:6002
• 19:14 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Fix rights for afttest and afttest-hide groups'
• 18:25 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Cleanup enotif related settings'
• 18:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnotifWatchlist to true for all wikis. Leaving wgShowUpdatedMarker set to false for all the big wikis'
• 16:50 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Simplify enotif code'
• 16:45 notpeter: starting innobackupex from db1040 to db1022 for new eqiad s6 snapshot slave
• 16:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgEnotifWatchlist defaulting to true. Big wikis explicitly set to false'
• 12:25 mutante: fixing integration.mw testswarm and applying fixed erb template by hashar
• 04:35 Tim: added an account for myself on observium
• 04:22 logmsgbot_: tstarling synchronized wmf-config/mc.php 'increased wgMemCachedTimeout from 500ms to 3000ms for bug 35900'
• 02:13 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 27 02:13:51 UTC 2012
• 00:12 Ryan_Lane: upgrading gluster on all instances
• 00:09 Ryan_Lane: upgrading gluster on labstore1-4

April 26

• 23:46 logmsgbot_: asher synchronized wmf-config/db.php 'raising db58 weight'
• 23:09 Reedy: Recreated resources directory symlinks in bits docroot
• 21:21 LeslieCarr: started deletion script on ms-be4
• 19:20 notpeter: restarting puppet on db59
• 19:18 Ryan_Lane: made LiquidThreads disabled by default on labsconsole, now users must add the special string to a page to enable it there.
• 19:18 Ryan_Lane: enabled NewUserMessage on labsconsole
• 19:06 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Add group permissions settings for AFTv5'
• 18:33 logmsgbot_: catrope synchronizing Wikimedia installation... : Deploy AFTv5 updates
• 17:17 LeslieCarr: reloaded varnish on mobile caches
• 14:19 notpeter: cleaned log space on search1017 and search1018 and started lucene
• 14:04 notpeter: stopping lucene on search1017 and 1018 to take that out of the equation
• 13:57 mutante: installing some (security) upgrades on fenari (apt,cron,samba,...)
• 13:54 notpeter: restartin lucene on search1017 and search1018
• 13:27 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayamon tewiki bug 33480'
• 13:23 logmsgbot_: nikerabbit synchronized php-1.20wmf1/extensions/Narayam/ 'Updating Narayam'
• 13:03 notpeter: (re)starting innobackupex from db1017 to db59 for new s1 slave
• 12:56 mark: Created precise-wikimedia APT distribution
• 08:27 mark: Power cycled mw40
• 06:57 binasher: restart pybal on amlvs1 with bgp disabled
• 06:57 binasher: restarted pybal on amlvs2 with bgp enabled
• 06:47 binasher: restarting pybal on amslvs2
• 06:26 binasher: shifting all traffic out of esams
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Thu Apr 26 02:14:03 UTC 2012
• 01:42 Ryan_Lane: starting mysql on db46
• 01:40 Tim: on professor: restarted udpprofile collector
• 01:37 Ryan_Lane: powercycling db46
• 01:33 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db46, host down'
• 00:44 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php

April 25

• 22:14 LeslieCarr: restarted swift-container-auditor on ms-be3
• 21:55 RobH: pushing dns update for scs-c1-eqiad and ps1-c#-eqiad
• 21:22 LeslieCarr: reloading varnish on mobile caches cp1041 cp1042 cp1043 cp1044
• 21:21 LeslieCarr: clearing mobile varnish cache
• 19:38 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'Attempted fatal fix'
• 19:33 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/Math/ 'Deploying 4c9e7dbe761c798ce15d7e2acef829a1582c058b'
• 19:14 notpeter: starting innobackupex from db12 to db59 for new s1 slave, per mr. feldman's directions
• 18:56 notpeter: starting innobackupex from db1017 to db60 for new s1 slave
• 18:49 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/FeaturedFeeds/SpecialFeedItem.php 'Deployed 4fb14a7b2ca9be715b820a9847d999f21c7d2cfc'
• 18:36 logmsgbot_: aaron synchronized php-1.20wmf1/img_auth.php 'Deployed f7e49bd71bd8356751242c5ce1cbae076a27cf7a'
• 18:10 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moving all remaining wikis to php-1.20wmf1
• 17:07 LeslieCarr: reloaded mobile varnish configs
• 17:06 LeslieCarr: purging mobile cache
• 16:40 LeslieCarr: starting delete script on ms-be3
• 16:14 RobH: done moving mgmt connections and serial connections in s8-eqiad for now
• 16:05 RobH: reshuffling cables in eqiad for serial and mgmt connections in a8, this may affect all eqiad mgmt and serial connections for the next 5 minutes
• 15:29 hashar: hashar: gallium: MySQL had issues most probably because of the mysql configuration snippets. https://gerrit.wikimedia.org/r/5796 might solve that.
• 14:03 mutante: gallium - don't start puppet unless the erb template fix for mysql has been merged
• 13:52 mutante: gallium stopped puppet, moved log_slow_queries config, re-setting up mysql again
• 13:41 mutante: gallium/testswarm - back up after mysql upgrade and issue starting the service
• 13:36 mutante: gallium - dpkg-reconfigure mysql-server-5.1, mysql does not start right
• 13:27 mutante: running apt-get upgrade on gallium
• 12:29 mark: Sending US, Brazil, Indian traffic to upload.eqiad
• 11:39 mutante: running authdns-update to add analytics100x and labsdb100x mgmt names
• 05:35 paravoid: powercycled lvs6, was dead and not responding to serial
• 03:43 logmsgbot_: asher synchronized wmf-config/db.php 'adding db58 to s7 as a new slave with a low weight'
• 03:24 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db58'
• 03:23 logmsgbot_: asher synchronized wmf-config/db.php 'adding db58 to s7 as a new slave with a low weight'
• 02:28 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Wed Apr 25 02:28:47 UTC 2012
• 02:14 logmsgbot_: LocalisationUpdate completed (1.19) at Wed Apr 25 02:14:46 UTC 2012
• 00:02 binasher: profiling collector was pegged at 100% cpu and graphs were turned to swiss cheese due to a bad stats call in 1.20, now fixed

April 24

• 23:59 binasher: powering off db16
• 23:55 binasher: streaming hot backup of db1041 to db58 (building a new s7 slave)
• 23:48 logmsgbot_: aaron synchronized php-1.19/includes/Setup.php 'Hacked out session request stats.'
• 23:46 logmsgbot_: aaron synchronized php-1.20wmf1/includes/Setup.php 'Deployed 42fcd43299246ecd1b265fcfcdd01a60319cf378'
• 23:19 AaronSchulz: Running 'mwscriptwikiset maintenance/populateRevisionSha1.php all.dblist' on hume
• 22:43 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Enabled file change journal on wikis using the new backend config.'
• 22:20 AaronSchulz: Tables added
• 22:18 binasher: rebooting db16 with updated kernel. it's probably still hopeless (dimm errors)
• 22:18 AaronSchulz: Creating the filejournal table on all wikis
• 21:59 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched commonswiki to the new backend config format.'
• 21:48 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db16, memory errors'
• 20:13 apergos: re-enabled replication via cron on ms7, it should catch up within an hour or so
• 20:10 binasher: reimaged db58 with fixed raid setup, imaging db59
• 19:51 notpeter: starting innobackupex from db1034 to db57 for new s2 slave
• 19:50 Ryan_Lane: repooling ssl3001
• 19:28 Ryan_Lane: depooling ssl3001
• 18:18 LeslieCarr: deploying to frontend
• 17:48 notpeter: deploying new squid conf to cp1001 frontend. is just a udp2log port change.
• 17:19 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Using newer backend for shared repos for testwiki, test2wiki, and mediawikiwiki.'
• 16:55 logmsgbot_: nikerabbit synchronized wmf-config/CommonSettings.php 'Translate extension configuration changes'
• 11:54 apergos: after much cursing and kicking zfs, a manual snapshot replication is running in screen as root on ms7 to ms8, expect it to take at least a day
• 11:44 mark: Sending all non-european upload traffic back to pmtpa to prepare for eqiad varnish storage rework
• 08:56 mutante: updated blog theme per guillaume (April commits)
• 08:05 apergos: temporarily disabled automatic zfs replication from ms7 -> ms8, cleared out space on ms8, catching up by hand
• 04:00 Ryan_Lane: powercycling ssl1
• 02:47 logmsgbot_: aaron synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Fixed notice spam.'
• 02:45 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/MobileFrontend/MobileFrontend.body.php 'Fixed notice spam.'
• 02:37 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Restructed filerepo a config a bit; nothing changed yet.'
• 02:28 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue Apr 24 02:28:47 UTC 2012
• 02:15 logmsgbot_: LocalisationUpdate completed (1.19) at Tue Apr 24 02:15:00 UTC 2012
• 00:15 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/stylesheets/common.css '0be2dc1288361c51f91533f1f77e78d9279b86e0'
• 00:13 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/common.css 'r115019'

April 23

• 23:35 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bumpging MobileFrontend resource version'
• 23:07 logmsgbot_: awjrichards synchronizing Wikimedia installation... :
• 23:02 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Add code for new URL scheme based on version_compare() logic'
• 22:51 logmsgbot_: awjrichards synchronizing Wikimedia installation... : MobileFrontend updates per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#23_April.2C_2012
• 22:33 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobile frontend resource version'
• 21:49 logmsgbot_: catrope synchronized php-1.20wmf1/resources/mediawiki/mediawiki.js 'Deploy 6e55a770b26b17b8fc9b5b4fe943dcc2867df4f3'
• 21:27 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'Deploy 93d470b'
• 20:41 mutante: neon - upgraded libssl, started icinga after adding monitor group
• 20:32 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/FileRepo.php 'Disabled write checks in the cleanDir() function.'
• 20:31 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/FileRepo.php 'Disabled write checks in the quickImport/quickPurge functions.'
• 19:43 logmsgbot_: catrope synchronized php-1.20wmf1/includes/specials/SpecialListgrouprights.php 'Deploy 047543b6805a268c8d689a7a1ce12ec545ef79a9'
• 18:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 18:43 logmsgbot_: reedy synchronized flaggedrevs.dblist 'Seems I never added ukwiki to the dblist... Oh well'
• 18:32 logmsgbot_: aaron synchronized wikiversions.dat
• 18:32 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf1
• 18:28 logmsgbot_: aaron synchronized php-1.20wmf1/includes/specials/SpecialContributions.php 'Deployed 72969cf8c9a403430c8c93fc20ab3118328c4d9c'
• 17:06 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use the newer backend config.'
• 14:33 notpeter: stopping puppet on cp1041 as well
• 14:17 notpeter: temp stopping puppet on cp1042-1044
• 13:09 mutante: powercycling frozen mw25, looks like mw21 above but no console output to paste here
• 13:07 mutante: fix puppet run on spence by removing searchidx1 resources from db9 (was in weird state being in site but also decommissioned)
• 11:23 mutante: mw21 powercycling mw21 - it died with this http://etherpad.wikimedia.org/mw21
• 10:55 mutante: force-reload ircecho on manganese to make gerrit-wm rejoin #mediawiki
• 10:48 hashar: banned CIA bots from #mediawiki IRC channel. It started spamming us with notifications from KDE and mandriva projects. See http://permalink.gmane.org/gmane.science.linguistics.wikipedia.technical/60905
• 10:30 mutante: searchidx1 was in site.pp and decom.pp at the same time. breaks puppet runs on spence. cannot override local resource. removing from site
• 10:27 mutante: killed a couple morebots processes on wikitech and it came back by itself :p

April 21

• 02:29 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 21 02:29:40 UTC 2012
• 02:15 logmsgbot_: LocalisationUpdate completed (1.19) at Sat Apr 21 02:15:20 UTC 2012

April 20

• 22:03 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched test2wiki to use the new LocalRepo config style.'
• 22:01 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched testwiki to use the new LocalRepo config style.'
• 21:52 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Added NFS backends for local/shared repos; they are not used yet.'
• 21:12 LeslieCarr: starting swift delete script on ms-be2
• 20:02 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/file/LocalFile.php 'deployed c77fbd394cda701758ad4523113f567bff7ede66'
• 19:45 apergos: powercycled mw4, it was unresponsive to pings and via mgmt
• 18:48 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449'
• 18:48 logmsgbot_: catrope synchronized php-1.19/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449'
• 18:07 notpeter: restarting nginx on ssl1002 and ssl1004 as they are not back up
• 18:01 logmsgbot_: awjrichards synchronizing Wikimedia installation... :
• 17:31 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Remoev wgArticleFeedbackv5OversightEmails override that was messing things up'
• 17:15 notpeter: stopping puppet on locke and emery. just to be safe...
• 17:11 RoanKattouw: Fixed ownership of /h/w/common/php-1.20wmf1/cache/l10n , should be owned by l10nupdate but was owned by reedy
• 17:01 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36124 - Deploy ProofreadPage extension on test2'
• 17:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Giving test2wiki moar namespaces'
• 16:11 mutante: add missing memcached servicegroup to nagios, restarted
• 15:10 mutante: apache error log on stafford has ruby exceptions re: phusion_passenger
• 15:01 mark: Converted OSPF directly connected redistributed routes from type 2 to type 1
• 14:51 mutante: starting swift-container-auditor on ms-be1
• 14:30 mark: Disabled down-pref of Tampa AS2828 routes
• 13:14 logmsgbot_: demon synchronized php-1.20wmf1/maintenance/backupTextPass.inc 'Pushing out Idb58ce27 for Ariel/Chris for dumps'
• 13:10 mark: Sending India upload traffic to upload-lb.eqiad
• 12:40 mark: Disabled iptables firewalls on internal prod swift cluster servers as it's dropping packets
• 12:22 mutante: restarted pdns on ns2
• 11:19 mark: Sending US upload traffic to eqiad as well
• 10:27 mark: Sending Brazil upload traffic to eqiad
• 08:39 hashar: Gave up running l10nupdate script it has some file permissions issues. Opened bug 36119 and bug 36120
• 08:36 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 08:36:53 UTC 2012
• 08:27 logmsgbot_: LocalisationUpdate completed (1.19) at Fri Apr 20 08:27:36 UTC 2012
• 08:13 hashar: rerunning l10nupdate for bug 34938
• 08:02 hashar: running l10nupdate for bug 34938
• 06:27 pgehres: re-eanabled PayPal on donatewiki and wmfwiki and resumed queue consumer on Aluminium
• 05:32 LeslieCarr: flushing mobile varnish cache
• 04:56 pgehres: disabled paypal on donatewiki and disabled queue consumer for duration of PayPal outage
• 02:33 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 02:33:02 UTC 2012
• 02:23 logmsgbot_: LocalisationUpdate completed (1.19) at Fri Apr 20 02:23:57 UTC 2012
• 01:47 logmsgbot_: awjrichards synchronizing Wikimedia installation... : r114983 on wikis still running 1.19

April 19

• 23:33 binasher: powercycled es1004
• 21:08 Jeff_Green: changed nagios contactgroup fundraising from tfinc/awrichards --> jgreen
• 21:03 RoanKattouw: Scap is broken in some weird way, it just stops running after the scap1-skins step. Doesn't run scap-1 (which does the actual sync), doesn't log "sync done", doesn't update graphite
• 21:01 logmsgbot_: catrope synchronizing Wikimedia installation... : Running scap again, AFTv5 is acting up
• 19:34 logmsgbot_: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
• 19:29 RoanKattouw: Running scap to deploy AFTv5 updates, and running AFTv5 schema changes on enwiki at the same time
• 18:50 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Set wmgArticleFeedbackv5OversightEmails for enwiki'
• 18:25 notpeter: nothing obvious in logs on db1005, starting mysql
• 18:15 notpeter: rebooting db1005. it's dead, jim.
• 17:52 RoanKattouw: Running schema changes for AFTv5 on testwiki
• 17:51 Jeff_Green: discovered nfs1 had ~1K redundant iptables rules, removed extras and reloaded
• 17:42 Jeff_Green: discovered sanger had ~7K redundant iptables rules, removed extras and reloaded
• 13:56 mutante: adding refreshLinks cron jobs to hume per RT-2355 (via puppet). if there should be any performance issues, schedule can be changed like <cluster>@<hour> in mediawiki.pp (and/or remove mediawiki::refreshlinks from hume and clear out the jobs of user mwdeploy)
• 08:35 mutante: emery - "udp2log_age" says some squid logfiles have not been written to in 6 hours, but from the filenames looks like this isnt a reason to worry, right
• 07:49 mutante: stat1 - this also needs udp2log stuff fixed. currently Could not find class misc::udp2log::udp-filter
• 07:47 mutante: gilman - what's up with it? closes SSH, does not like mgmt pass, was running jenkins but broken
• 07:43 mutante: owa[1-3] They dont have real puppet freshness issues, it's rather firewalling and the snmp traps
• 02:30 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Thu Apr 19 02:30:33 UTC 2012
• 02:21 logmsgbot_: LocalisationUpdate completed (1.19) at Thu Apr 19 02:21:31 UTC 2012

April 18

• 22:55 LeslieCarr: updating exim4.conf on mchenry to not allow old ranges
• 21:03 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 20:47 logmsgbot_: catrope synchronized php-1.20wmf1/resources/startup.js 'touch'
• 20:46 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/SyntaxHighlight_GeSHi/ 'Deploying GeSHi fix https://gerrit.wikimedia.org/r/#change,4949'
• 20:04 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: specieswiki and foundationwiki to 1.20wmf1
• 19:56 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/LiquidThreads/classes/Hooks.php 'Avoid fatals on invalid title in API'
• 19:51 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All *wiki wikis to 1.20wmf1
• 19:25 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote and wikiversity projects to 1.20wmf1
• 19:22 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikibooks to 1.20wmf1
• 19:18 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikinewses to 1.20wmf1
• 19:07 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisources to 1.20wmf1
• 19:07 logmsgbot_: catrope synchronized wmf-config/mc.php 'Swap out 10.0.2.251 (down) with 10.0.11.24 (spare). This is the last spare, there are now NO SPARES LEFT in mc.php'
• 19:00 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf1
• 18:57 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/LiquidThreads/classes/Dispatch.php 'Added type hint for better fatals'
• 18:44 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikiversity to 1.20wmf1
• 18:43 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikiquote to 1.20wmf1
• 18:41 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikibooks to 1.20wmf1
• 18:40 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf1
• 18:39 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwiktionary to 1.20wmf1
• 18:32 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikisource to 1.20wmf1
• 17:20 logmsgbot_: catrope synchronized docroot/bits/ 'Remove static-1.00 again'
• 16:57 logmsgbot_: catrope synchronized docroot/bits 'Add docroot/bits/static-1.00 for testing'
• 16:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wmfUseRevSha1Columns to true for enwiki'
• 13:30 mutante: applied a patch to etherpad that allows admins to delete pads
• 12:53 mutante: restarting/fixing etherpad issue
• 11:08 mark: Sending European bits traffic back to esams
• 02:30 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Wed Apr 18 02:30:50 UTC 2012
• 02:21 logmsgbot_: LocalisationUpdate completed (1.19) at Wed Apr 18 02:21:49 UTC 2012
• 02:13 logmsgbot_: catrope synchronized php-1.20wmf1/README 'Dummy sync to capture which hosts time out on sync-file'
• 00:52 K4-713: updated production civi to r1631
• 00:41 Ryan_Lane: adding interface for per-project sudo on OpenStackManager

April 17

• 23:36 K4-713: updated production civi to r1628
• 23:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Fixes for cswiktionary changes per Danny B'
• 22:49 RoanKattouw: That was bug 34885 of course
• 22:43 logmsgbot_: catrope synchronized php-1.19/extensions/WikiEditor/ 'Deploy fix for bug 348885'
• 22:38 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Deploy fix for bug 348885'
• 22:05 K4-713: updated prod civi to r1625
• 21:51 logmsgbot_: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero needed for carrier testing'
• 21:42 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'use $wmgUseMathJax'
• 21:41 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'use $wmgUseMathJax'
• 21:38 K4-713: queue consumer re-enabled
• 21:35 K4-713: updated prod civi to r1623
• 21:32 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php
• 21:29 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/templates/ApplicationTemplate.php 'ec7c5cc'
• 21:28 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114947'
• 21:24 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgUseMathJax on mediawikiwiki'
• 20:33 logmsgbot_: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.flagging.php
• 20:26 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/VisualEditor/ 'Deploy VisualEditor beta warning'
• 19:52 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bump mobile resource version'
• 19:52 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/
• 19:51 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/
• 19:50 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobile frontend resource version'
• 19:01 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php
• 18:55 logmsgbot_: reedy synchronized php-1.19/includes/api/ApiQueryBlocks.php 'r114941'
• 18:53 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
• 18:47 binasher: returning sq68
• 18:36 binasher: pulling sq68 from pybal for a bit
• 18:29 RoanKattouw: Did a graceful restart of all job runners using dsh about 15 mins ago
• 18:29 RoanKattouw: Restarted morebots
• 07:44 apergos: morebots test
• 07:44 apergos: restarted varnish service manually a bit a go on sq67 and sq70, the cron job didn't seem to have gone off. restarted morebots too while I was at it
• 03:37 Jeff_Green: dist-upgrade arsenic
• 03:29 LeslieCarr: restarting varnish on arsenic again
• 03:12 maplebed: started a script to delete old objects on ms-be1 for swift truncated object cleaning
• 02:53 Jeff_Green: dist-upgrade on strontium
• 02:43 LeslieCarr: restarted varnish on arsenic
• 02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue Apr 17 02:26:40 UTC 2012
• 02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Tue Apr 17 02:17:24 UTC 2012
• 01:44 LeslieCarr: restarting varnish on niobium
• 00:52 LeslieCarr: reloading amslvs4
• 00:27 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo 'deployed 552ff0f482f3e65e9795fe304dd810e9ae1b03fb'

April 16

• 23:31 logmsgbot_: catrope synchronizing Wikimedia installation... : Now with a touch of the specific WikiEditor.i18n.php file
• 23:11 logmsgbot_: catrope synchronizing Wikimedia installation... : Hopefully fixing the WikiEditor messages this time, now with MessagesEn.php touch
• 23:07 logmsgbot_: catrope synchronizing Wikimedia installation... : Hopefully fixing the WikiEditor messages this time
• 22:58 logmsgbot_: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend to r114934
• 22:49 logmsgbot_: catrope synchronizing Wikimedia installation... : Need to run scap for this WikiEditor change, contains i18n changes
• 22:39 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Deploy WikiEditor revert'
• 20:53 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Actually deploy the recent WikiEditor fixes'
• 18:58 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Commons Wiki to 1.20wmf1
• 18:47 logmsgbot_: reedy synchronized php-1.20wmf1/resources/mediawiki/mediawiki.js
• 18:46 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/WikiEditor
• 18:37 mutante: manually added iptables nat rules on nfs2
• 18:13 notpeter: upgrade of udp2log on nfs1/2 complete. should be operating normally now.
• 17:41 mutante: LDAP on nfs2 warnings - opendj was _just_ started there when puppet was fixed with an unrelated issue
• 17:38 mutante: restarting opendj on nfs2 because it refused connections
• 17:08 logmsgbot_: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ 'zero and mobile changes'
• 16:07 notpeter: upgrading and restarting udp2log on nfs1/2
• 15:04 mutante: puppet fresh on nfs[12] after removing nonexistent misc::mediawiki-logger class
• 14:46 mark: Shutdown db24 for memory testing by Chris
• 13:27 mark: Sending European bits traffic back to pmtpa
• 12:24 mark: Sending European bits traffic back to esams
• 12:06 mark: Testing sess_leak_fix2 patch with a snapshot varnish build on cp3001
• 11:56 Reedy: Ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -- "cd /usr/local/apache/common && sudo -u mwdeploy ln -s php php-1.18" to create symlink for php-1.18
• 11:51 Reedy: Killing php-1.18 again
• 11:48 mutante: sq34 - System halted! Error: Internal Storage Slot, powered down, -> RT
• 11:45 logmsgbot_: reedy synchronized php-1.18/ 'Symlink php-1.18 back to php (our current main running version) as lots of requests on bits are for 1.18 resources'
• 11:44 mutante: sq34 was broken and died when connecting to mgmt, powercycling
• 11:37 mutante: nfs1 - Could not find class misc::mediawiki-logger for nfs1
• 10:57 Krinkle: bits.wikimedia.org back up, mark fixed it.
• 10:33 Krinkle: bits.wikimedia.org serving Error 503 Service Unavailable on all load.php requests for mediawiki.org and nl.wikipedia.org, maybe more
• 09:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnableJavaScriptTest to true for test2wiki'
• 02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Mon Apr 16 02:26:58 UTC 2012
• 02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Mon Apr 16 02:17:57 UTC 2012

April 15

• 17:35 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api '/me whistles'
• 17:20 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api
• 02:25 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sun Apr 15 02:25:58 UTC 2012
• 02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Sun Apr 15 02:17:19 UTC 2012

April 14

• 18:14 mark: Shifting european bits traffic back from esams to pmtpa, session leak is still there
• 17:08 mark: Shifting european bits traffic back from pmtpa to esams
• 15:31 mark: Reverted varnish to 3.0.2-2wm4 on cp3001; the race condition patch did not fix the problem
• 14:56 mark: Sending European bits traffic to pmtpa for testing
• 13:52 mark: Backported varnish bug #897 patch to varnish 3.0.2, testing a snapshot build on cp3001
• 11:37 mark: Raised session_max to 300000 (runtime) on cp3001/cp3002
• 05:58 K4-713: re-enabled the queue consumer on aluminium
• 02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 14 02:26:55 UTC 2012
• 02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Sat Apr 14 02:17:34 UTC 2012
• 02:16 K4-713: updated prod civi to r1616
• 01:36 K4-713: turned off queue consumption on prod civicrm
• 01:36 K4-713: updated production civicrm to r1614

April 13

• 20:53 mark: Rebooting cp3002
• 20:37 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114889'
• 17:54 Jeff_Green: created new repo operations/debs/wikimedia-search-qa to stay within package naming conventions
• 17:31 notpeter: upgrading udplog on locke to 1.8-2 and restarting, etc
• 17:27 Jeff_Green: created new operations/debs/search-qa repo for packaging search qa scripts
• 17:17 notpeter: restarting udp2log on emery
• 12:53 notpeter: restopping puppet on locke/emery
• 12:09 mark: Deploying varnish 3.0.2-2wm4 and enabling persistent storage on all even numbered eqiad upload varnish hosts
• 11:46 mark: Imported varnish 3.0.2-2wm4 into the Wikimedia APT repository
• 02:48 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:39 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Fri Apr 13 02:39:01 UTC 2012
• 02:20 logmsgbot: LocalisationUpdate completed (1.19) at Fri Apr 13 02:20:35 UTC 2012
• 01:40 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fix robots file'
• 01:18 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ 'zero and mobile changes'
• 01:06 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix html formatter'
• 00:56 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 00:08 Ryan_Lane: rebooting ssl1004

April 12

• 23:39 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 23:08 logmsgbot: preilly synchronizing Wikimedia installation... : zero rated mobile access changes and mobile frontend updates
• 21:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34923 - namespace required for PORTAL'
• 19:46 notpeter: stopping puppet on locke and emery
• 18:41 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 updates
• 18:22 Reedy: Ran namespaceDupes against bewiki
• 18:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34024 - Install ArticleFeedback on es.wikinews'
• 18:15 logmsgbot: reedy synchronized php-1.19/resources/startup.js 'touch'
• 18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34024 - Install ArticleFeedback on es.wikinews'
• 18:11 Reedy: Created AFT tables on eswikinews
• 17:54 RoanKattouw: Running schema updates for ArticleFeedbackv5 on enwiki
• 17:46 RoanKattouw: Deploying ArticleFeedbackv5 updates to testwiki and rebuilding localization cache
• 16:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Allow bnwiki crats to grant/remove import'
• 16:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35258 - Allow bureaucrats to remove sysop rights on fr.wikipedia'
• 16:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix imports for wm2012'
• 16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35917 - allow transwiki imports on wikimania2012'
• 16:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35666 - Renaming Namespace Wikisource:Author in gu.wikisource'
• 16:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35694 - Add enotif on page changes in watchlist (guwiki and source)'
• 16:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35818 - Change of Armenian Wikipedia namespace'
• 16:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35905 - Change namespaces configuration - pl.wikipedia'
• 16:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35261 - Add block permissions in rollback on Lusophone Wikipedia'
• 16:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35823 - Wikijunior and cookbook namespaces for the Vietnamese Wikibooks'
• 16:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35659 - Set logo for sl.wikiversity'
• 16:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35853 - Set a non-empty default value for wmgArticleFeedbackBlacklistCategories on WMF wikis'
• 15:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35878 - Enable e-mail notifications for watchlist (EnotifWatchlist) on tawiki'
• 15:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35852 - Add a category to $wgArticleFeedbackBlacklistCategories for Portuguese Wikipedia to remove AFT from disambiguation pages'
• 15:10 mutante: gallium - after files have been deleted/moved, puppet back to normal operation (and new clone directory in Apache)
• 13:23 mutante: killed puppets on gallium
• 12:33 mark: repooled ssl1002
• 12:27 mutante: powercycling frozen ssl1002
• 12:22 mark: Manually depooled down ssl1002 in pybal
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Thu Apr 12 02:24:29 UTC 2012
• 02:15 logmsgbot: LocalisationUpdate completed (1.19) at Thu Apr 12 02:15:54 UTC 2012

April 11

• 22:37 maplebed: deployed more log filters to emery: gerrit/r4758
• 21:35 LeslieCarr: restarted nrpe on db10
• 21:33 LeslieCarr: db1004 puppet is fubar
• 21:33 LeslieCarr: restarted puppet on db30
• 21:33 LeslieCarr: restarted puppet on mw1110
• 19:41 notpeter: reimaging bellin and blondel
• 19:28 logmsgbot: reedy synchronized live-1.5/ 'fix resources symlinks'
• 19:23 logmsgbot: reedy synchronized live-1.5/ 'fix resources symlinks'
• 16:54 notpeter: enabling notifications for eqiad lucene vips
• 16:31 mark: Sending Canadian upload traffic to the eqiad varnish upload cluster
• 15:59 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 4 to eqiad. for realz this time!'
• 15:45 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 1 and prefix pool to eqiad. for realz this time!'
• 15:31 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 2 to eqiad. for realz this time!'
• 15:15 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 3 to eqiad. for realz this time!'
• 14:40 notpeter: restarting indexer on searchidx2
• 13:48 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AbuseFilter/special/SpecialAbuseLog.php
• 13:35 mutante: applied patch-RT-2804.diff to bugzilla per RT:2804 re: XMLRPC content-type verification
• 12:07 mutante: moved another list: museum-l -> glam (http://lists.wikimedia.org/pipermail/glam/2012-April/000000.html)
• 11:58 mark: Setup cp1036 with the persistent storage backend
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed Apr 11 02:26:28 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Wed Apr 11 02:17:55 UTC 2012
• 00:11 LeslieCarr: nagios down

April 10

• 23:50 RoanKattouw: Removed srv187-189 from /etc/dsh/group/job-runners , their jobrunner class has been commented out in puppet since October
• 23:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'bug 35869 - Add strategywiki as an import source on testwiki'
• 22:53 RoanKattouw: Trying a graceful restart of the job runner on mw1 by sending SIGHUP to the jobs-loop.sh process
• 22:53 logmsgbot: catrope synchronized php-1.19/extensions/WikimediaMaintenance/jobs-loop.sh 'r114834'
• 22:24 logmsgbot: reedy synchronized php-1.20wmf1/extensions/CentralAuth/ 'g4102'
• 22:23 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AntiSpoof/ 'g4103'
• 21:20 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling mobile URL template for mediawiki.org (using "mediawikiwiki" this time)'
• 21:18 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling mobile URL template for mediawiki.org'
• 21:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf1
• 21:04 logmsgbot: reedy synchronized php-1.20wmf1/extensions/MobileFrontend/javascripts 'minified JS'
• 20:55 logmsgbot: reedy synchronized docroot/ 'Fix symlinks'
• 20:45 logmsgbot: reedy synchronized docroot/
• 20:35 logmsgbot: reedy synchronized docroot/
• 20:31 logmsgbot: reedy synchronized live-1.5/
• 20:24 logmsgbot: reedy synchronized php-1.20wmf1/ 'Resyncing for apaches with no space'
• 20:23 logmsgbot: reedy synchronized live-1.5 'Fix symlinks'
• 20:18 Reedy: Deleting php-1.18 from all apaches due to lack of space
• 20:14 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PrefSwitch/ 'PrefSwitch is needed by SimpleSurvey'
• 19:35 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding localisation cache for test2/1.20wmf1
• 19:24 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf1.php 'Sync ExtensionMessages'
• 19:23 logmsgbot: reedy synchronized php-1.20wmf1/extensions/ 'Would you like some extensions to go with that, sir?'
• 19:21 LeslieCarr: restarting gmond on db1004 after removing it's 5gig log
• 19:07 logmsgbot: reedy synchronized php-1.20wmf1/LocalSettings.php 'Push LocalSettings out'
• 19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf1
• 19:00 logmsgbot: reedy synchronized php-1.20wmf1/ 'Pushing files for 1.20wmf1'
• 18:03 logmsgbot: aaron synchronized wmf-config/swift.php 'Catch e bogus empty file names from listings'
• 14:17 robh: search in eqiad is being reinstalled, no need to be alarmed (thats a pun!)
• 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgLanguageConverterCacheType for git deployment later'
• 11:50 mutante: pxe boot / reinstall cp1029 - cp1036
• 11:24 mark: Imported varnish 3.0.2-2wm3 into the Wikimedia APT repository
• 09:30 apergos: restarted slaving on es1003, it will be a bit before it catches up. patience, young nagios
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Tue Apr 10 02:16:58 UTC 2012
• 01:33 Tim: on sodium: enabling mod_auth on lists.wikimedia.org by running puppet

April 9

• 23:14 mutante: migrated foundation-l to wikimedia-l (users/passwords/archive urls/settings stay, old mail address & siteinfo redirect)
• 22:32 logmsgbot: asher synchronized wmf-config/db.php 'returning db12 as enwiki recentchange/watchlist db'
• 21:39 LeslieCarr: restarted mysql on es1004 and cleared out its disk space
• 17:49 LeslieCarr: moving es monitoring to nrpe and variables, may cause false pages if i did it wrong :)
• 17:36 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 35426 - WebFonts on mr.wikisource.org'
• 14:54 RobH: i killed eqiad search nodes, woooo
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Apr 9 02:17:22 UTC 2012

April 8

• 08:45 Nemo_bis: Servers have been very slow, almost unresponsive, and network had a drop of ~0.3 Gb/s, at ~8.35-40.
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sun Apr 8 02:16:58 UTC 2012

April 7

• 17:55 logmsgbot: reedy synchronized wmf-config/codereview.php 'Remove deferred paths'
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sat Apr 7 02:16:54 UTC 2012

April 6

• 22:23 LeslieCarr: deploying new squid config to all squids
• 22:14 LeslieCarr: added neon into tiertwo of squid allowed hosts
• 22:13 LeslieCarr: deploying new squid config to amssq35
• 21:55 LeslieCarr: restarted puppet on spence
• 21:35 LeslieCarr: moved jenkins_1.458_all.deb to /srv/wikimedia/incoming/ on brewster
• 21:32 LeslieCarr: restarted squid on brewster
• 18:27 Ryan_Lane: updating OpenStackManager to r114758 on virt0
• 17:33 mark: Sending Japanese upload traffic to varnish in eqiad
• 17:15 mark: Power cycled down host lvs5
• 16:43 mutante: changed master and started slave on es1004
• 15:55 mutante: used gerrit create-project to create operations/debs/wikistats.git
• 14:13 mutante: manganese (gerrit) now sends SSL CA certificate on https, (curl -vvv says verify ok), should resolve RT:2777 and BZ:35709
• 11:51 mutante: es1004 - rsync was finished, deleted all binlogs from old host, mysqld_safe& , but did not "change master.." and "start slave" (see mail)
• 11:39 notpeter: restarting lsearchd on search3... again...
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Apr 6 02:17:37 UTC 2012
• 01:21 Ryan_Lane: updating OpenStackManager to r114757 on virt0
• 00:18 Ryan_Lane: updating OpenStackManager to r114754 on virt0

April 5

• 23:49 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Change guwikisource logo to point to the unscaled file instead'
• 21:46 notpeter: halting db15 for it to await decom
• 21:39 binasher: started enwiki.revision sha1 migration on db12
• 21:32 notpeter: restarting lsearchd on search18
• 21:22 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12, moving enwiki watchlist,recentchange,etc to db53'
• 21:19 logmsgbot: asher synchronized wmf-config/db.php 'returning db53'
• 21:17 logmsgbot: py synchronized wmf-config/lucene.php 'pushing all search traffic back to pmtpa'
• 18:34 Ryan_Lane: updating OpenStackManager to r114746 on virt0
• 18:19 Ryan_Lane: updating OpenStackManager to r114744 on virt0
• 16:49 RobH: brewster puppet running again, cisco installs wont work again until i finish puppetizing the files later today
• 15:41 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool4 to eqiad. this is the smaller wikis shard'
• 15:40 notpeter: pointing search pool4 to eqiad (this is the "smaller languages" shard)
• 15:14 Rob_H: puppet daemon being halted on brewster, i need to make local test changes to dhcp
• 14:52 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search prefix pool live in eqiad'
• 14:51 notpeter: pushing search prefix pool live in eqiad
• 14:51 mutante: gallium - disabled incompatible GitTool plugin on jenkins and restarted it
• 14:34 mutante: importing jenkins_1.458_all.deb to wikipedia apt repo and upgrading it on gallium
• 14:08 apergos: started rsync in screen session as root on es1003 copying snapshot from es1001 to /a/
• 14:04 andrewbogott: created labs account for cneubauer
• 14:02 logmsgbot: py synchronized wmf-config/lucene.php 'pointing enwiki search and enwiki.prefix at eqiad'
• 14:00 notpeter: pointing enwiki and enwiki.prefix at eqiad search cluster
• 13:48 mutante: gallium - upgraded all pear packages
• 13:45 mutante: gallium - upgraded phpunit and php_codesniffer via pear (have been installed via pear before, distro outdated)
• 13:43 mutante: gallium - upgrading pear
• 13:33 mutante: installing package upgrades on gallium. apache,apt,postgres,php5-*,ruby,...various libs
• 13:24 logmsgbot: py synchronized wmf-config/lucene.php 'pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad'
• 13:21 notpeter: pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad
• 12:27 notpeter: search1 and search4 seem to be dead. restarting lsearchd
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Thu Apr 5 02:16:52 UTC 2012
• 00:33 Ryan_Lane: updating OpenStackManager to r114730 on virt0
• 00:24 Ryan_Lane: updating OpenStackManager to r114729 on virt0
• 00:19 Ryan_Lane: updating OpenStackManager to r114728 on virt0
• 00:12 Ryan_Lane: updating OpenStackManager to r114726 on virt0
• 00:00 Ryan_Lane: updating OpenStackManager to r114724 on virt0

April 4

• 22:16 maplebed: deployed (3rd time's the charm!) udp-filter changes to emery for diederik
• 22:14 logmsgbot: py synchronized wmf-config/lucene.php 'pointing all search back to pmtpa'
• 22:13 notpeter: flipping all search back to pmtpa (until tomorrow...)
• 22:00 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback 'r114717'
• 21:24 cmjohnson1: replacing power cable to psu1 (bottom) es1
• 21:22 cmjohnson1: replacing power cable to psu1 (top) es1
• 21:14 logmsgbot: py synchronized wmf-config/lucene.php 'pointing de, fr, and ja search at lvs pool in eqiad for live testing'
• 21:12 notpeter: moving de, fr, and ja search to eqiad
• 21:04 cmjohnson1: replacing power cable on labstore2 array psu2 (right side)
• 21:00 cmjohnson1: replacing power cable on labstore1 array psu1 (left side)
• 20:57 cmjohnson1: removing power from bottom power supply labstore 2
• 20:54 cmjohnson1: removing power from top power supply on labstore2
• 19:44 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wmgArticleFeedbackv5AbuseFiltering on enwiki'
• 19:40 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable wmgArticleFeedbackv5AbuseFiltering on enwiki'
• 19:14 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114716'
• 19:12 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wmgArticleFeedbackv5AbuseFiltering on enwiki'
• 19:04 RobH: dns update for zhen mgmt
• 18:54 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying AFTv5 update
• 18:52 logmsgbot: py synchronized wmf-config/lucene.php 'pointing ru, nl, pl, pt, zh, and sv search at lvs pool in eqiad for live testing'
• 18:51 notpeter: moving ru, nl, pl, pt, zh, and sv search to eqiad
• 18:27 mutante: nuked /a contents on es1004, started rsync from es1001
• 18:16 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add code for wmgArticleFeedbackv5AbuseFiltering'
• 18:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgArticleFeedbackv5AbuseFiltering, enabled on testwiki only'
• 17:55 RoanKattouw: Running AFTv5 schema changes on enwiki
• 17:47 RobH: i didnt crash the site, weeee
• 17:46 RobH: gracefully restarting apaches
• 17:46 RobH: pushing out redirects change to apaches for wikipedia.org/com.il redirect to he.wikipedia.org
• 17:41 binasher: started enwiki.revision sha1 migration on db53
• 17:38 logmsgbot: asher synchronized wmf-config/db.php 'returning db52, pulling db53'
• 17:32 RobH: update done, all nameservers still online
• 17:31 RobH: dns update for wikipedia.org/com.il being resolved
• 17:08 RoanKattouw: Applying AFTv5 schema change on testwik
• 15:30 logmsgbot: py synchronized wmf-config/lucene.php 'pointing eswiki search at lvs pool in eqiad for live testing'
• 15:28 notpeter: pointing eswiki search at eqiad
• 12:51 mutante: db1007 - add mysql startup via 'update-rc.d mysql defaults'
• 12:42 apergos: started mysqld on db1007 via /etc/init.d/mysql (this doesn't seem to point to a special fb build, and can't seem to find one on this host, what's up with that?)
• 12:31 apergos: rebooted bd1007, it was dead in the water (also no helpful messages on console, bah)
• 11:16 mutante: enabled Renameuser extension on wikitech, renamed tchay per RT request, disabled extension again (it was installed but disabled)
• 02:19 logmsgbot: LocalisationUpdate completed (1.19) at Wed Apr 4 02:19:03 UTC 2012
• 01:50 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileOp.php 'deployed r114697'
• 01:39 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'

April 3

• 23:17 LeslieCarr: updating bgp policies on cr1.sdtpa
• 22:44 LeslieCarr: reinstalling neon
• 22:04 maplebed: rolled back changes to emery in udp-filter due to the new binary crashing.
• 21:50 maplebed: ran /etc/init.d/udp2log reload on emery to enact the puppetted changes
• 21:41 maplebed: deploying new udp-filter and teahouse filters to emery for diederik
• 20:13 notpeter: restarting lsearchd on search7. was taosted
• 18:37 logmsgbot: root synchronized wmf-config/mc.php
• 18:37 RobH: syncing new mc.php, forgot to check for all three of the servers i took down, opps.
• 18:28 RobH: shutting down mw28, mw49, & mw58 for rack relocation due to power overload in d2-pmtpa, relocation to d1-sdtpa per rt 2692
• 17:59 K4-713: Synchronized payments cluster to r114642
• 17:52 logmsgbot: reedy synchronized php-1.19/extensions/MobileFrontend/
• 17:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix commas in guwikisource namespaces'
• 17:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix commas in guwikisource namespaces'
• 16:38 RobH: bringing down srv237 for phase balancing
• 16:37 RobH: srv230 back in rotation
• 16:26 RobH: shutting down srv230 for power phase move per rt 2759
• 16:10 RobH: updating brewster to use new dhcp files for cisco, no more local hackin.
• 15:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35482 - Add Patroller & Autopatroller groups on ml.wikisource'
• 15:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35482 - Add Patroller & Autopatroller groups on ml.wikisource'
• 15:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35545 - Grant the abusefilter-log-detail right to patrollers on Commons'
• 15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35545 - Grant the abusefilter-log-detail right to patrollers on Commons'
• 15:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35624 - Subject namespace for the Vietnamese Wikibooks'
• 15:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35603 - Enable Transwiki import on KN:WP'
• 15:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35581 - Closure of nz.wikimedia.org'
• 15:15 logmsgbot: reedy synchronized closed.dblist 'Bug 35581 - Closure of nz.wikimedia.org'
• 13:35 Tim: manually reloaded rsyslogd on all apaches
• 06:16 Tim: deploying limited/split apache syslog (https://gerrit.wikimedia.org/r/#change,4149)
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Tue Apr 3 02:16:32 UTC 2012
• 00:37 logmsgbot: aaron synchronized php-1.19/includes/Block.php 'deployed r114672'

April 2

• 23:54 Tim: restarting all apaches with apache-restart-all-hard
• 23:51 logmsgbot: tstarling synchronized php-1.19/extensions/ConfirmEdit/FancyCaptcha.class.php
• 23:37 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 23:36 maplebed: cleared the varnish cache for preilly
• 23:34 Tim: on all apaches: running logrotate -f and deleting the resulting backup syslog files, to free up disk space
• 23:32 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114673'
• 23:21 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version number'
• 23:05 logmsgbot: awjrichards synchronizing Wikimedia installation... : Deploying MobileFrontend changes at r114671 per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#2_April.2C_2012
• 21:43 maplebed: reverted changes to emery's logging due to a broken package in the deploy.
• 21:30 LeslieCarr: turned down ms7's secondary ethernet port to prevent the flapping (stupid sun boxes)
• 19:51 maplebed: deploying new udp-filter to emery rt-2501 gerrit/r4120
• 19:51 notpeter: running authdns-update on dobson
• 18:30 RobH: brewster puppet daemon stopped, doing local hacks
• 18:17 RobH: removed old bin files on db1004 and prolly borked it by removing the wrong files
• 17:54 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php '35436 - Enable Narayam at Hindi Wikipedia'
• 17:47 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add disable images option for default on zero domain'
• 17:45 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 35328 - Enable WebFonts for fr.wikisource.org'
• 17:40 logmsgbot: nikerabbit synchronized php-1.19/languages/Names.php 'I18ndeploy r114656'
• 17:15 preilly: carrier testing push for DIGI
• 17:15 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
• 16:46 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Mon Apr 2 02:16:47 UTC 2012

April 1

• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Apr 1 02:17:22 UTC 2012

March 31

• 10:22 mutante: srv222,225 were also upgraded but stopping there for now in favor of reinstalls
• 09:58 mutante: nuked /usr/shared/doc on a couple srv's, hey at least 700MB or something, and yes we really should reinstall with a decent partitioning scheme as M ark said
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 31 02:18:10 UTC 2012

March 30

• 19:37 hashar: configured jenkins on gallium to use smtp.pmtpa.wmnet as outgoing SMTP server
• 19:28 RobH: puppet daemon restarted on brewster
• 18:13 RobH: killing puppet daemon on brewster, i need to hack at local configuration for cisco server stuff
• 12:56 mutante: db1047 - added system startup for /etc/init.d/mysql
• 12:47 mutante: powercycling db1047
• 12:28 mutante: deleted old kernel sources on upgraded srvs for that little extra space during peaks, suggesting to nuke /usr/share/doc if there should be more disk space warnings
• 10:41 mutante: same for srv223
• 09:18 mutante: srv224,srv219,srv220, upgrade apache, dist-upgrading w/ kernel, disabling ureadahead, rebooting one by one
• 08:06 mutante: storage3 - gmond unable to find the metric information for any mysql_* .."module has not been loaded", starting mysql, running puppet ...
• 07:57 mutante: powercycling storage3
• 07:03 Tim: running bug 35578 cleanup script in screen on fenari
• 06:41 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
• 06:40 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
• 06:39 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
• 06:15 Tim: killed vi on fenari owned by awjrichards, locking CommonSettings.php for two days
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 30 02:17:56 UTC 2012
• 01:13 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove more crap'
• 01:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove some dupe code'
• 01:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove wmgUsabilityPrefSwitch'
• 00:59 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove wmgUsabilityPrefSwitch'
• 00:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove unused wmgUseUsabilityInitiativeAlpha'

March 29

• 23:49 logmsgbot: aaron synchronized php-1.19/includes/revisiondelete/RevisionDeleteUser.php 'deployed r114619'
• 21:20 LeslieCarr: rebooting db47
• 20:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Swap wgUseCommaCount to wgArticleCountMethod'
• 20:07 notpeter: restarting lsearchd on search2 to del the logfile to end all logfiles
• 20:05 RoanKattouw: Stopping and starting Gerrit on manganese to apply Chad's change of the -1 text in the DB
• 20:02 notpeter: restarting lsearchd on search7 to del the logfile to end all logfiles
• 18:11 logmsgbot: catrope synchronized php-1.19/extensions/ClickTracking/ClickTracking.hooks.php
• 17:59 RobH: search1021 coming back up, done with tests
• 17:53 RobH: search1021 coming down for ssd fit test
• 17:07 notpeter: disabling notifications for search lvs nagios checks for 24 hours to test fix
• 15:42 notpeter: finished clearning up all pmtpa search hosts. hey look! they all have lots of space now!
• 15:15 notpeter: restarting lsearchd on search3
• 15:02 RobH: brewster puppet re-enabled
• 15:02 RobH: virt1001 pxe boots via dhcp and fails tftp download, i have to hold off on further troubleshooting until i have a network admin
• 14:47 RobH: did virt1001 wrong, reupdating dns
• 14:39 RobH: all nameservers still online after udpate
• 14:37 RobH: updating dns for virt1001 testing
• 14:29 RobH: stopping puppet runs on brewster so my hacking at the dhcpd.conf file won't get overwritten until I have it working right
• 14:01 Jeff_Green: restarted varnish on on cp3002 because it was thrashing futiley
• 13:45 notpeter: rebooting (mostly) down cp3001
• 13:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add participation namespace to metawiki per request'
• 13:11 notpeter: trimming logs and such on search1-20
• 09:59 mutante: srv221, disabling ureadahead, installing package upgrades and new kernel, rebooting
• 09:40 mutante: kill and start lsearchd on search7
• 09:36 mutante: restarted defunct lsearchd on search6
• 09:10 mutante: gallium - added demon,hashar,reedy to group jenkins as it's a problem using puppet when users and groups already exist
• 06:25 mutante: powercycling sq40
• 06:21 mutante: installed more package upgrades on sodium
• 05:58 mutante: installed security upgrades on brewster, cadmium, capella (apache,mysql,ruby,apt..)
• 05:49 mutante: db42 - mysql did not autostart after boot, added using update-rc.d
• 05:42 mutante: db42 - reboot worked despite the grub warning about unreliable blocklists
• 05:37 mutante: rebooting db42 to finish upgrades
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 29 02:17:53 UTC 2012

March 28

• 23:27 Tim: running apt-get upgrade on mw22,mw66,srv193,srv250,srv253,srv236
• 23:25 Tim: cleaned up stuck apt-get process on srv236
• 23:22 Tim: cleaned up stuck apt-get processes on mw22,mw66,srv193,srv250,srv253
• 21:44 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobile frontend resrouce version'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/application.min.js 'r114576'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/banner.min.js 'r114576'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114576'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114576'
• 21:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/application.js 'r114576'
• 21:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/banner.js 'r114576'
• 20:43 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 28 20:43:20 UTC 2012
• 20:29 notpeter: restarted search1020. nothing conspicuous in logs
• 19:56 RoanKattouw: Running a patched version of l10nupdate that rebuilds the localization cache
• 18:49 logmsgbot: catrope synchronizing Wikimedia installation... : Bugfixes for ArticleFeedbackv5, ArticleFeedback and ClickTracking
• 16:47 cmjohnson1: msw1-d1-pmtpa replacement complete
• 16:34 cmjohnson1: replacing msw-d1-pmtpa per rt2639
• 15:36 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech'
• 15:34 Reedy: srv221 is full
• 15:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech'
• 14:39 RobH: restarted morebots in screen on wikitech, no longer as catrope, as roan has root on that box
• 14:36 RobH: got virt1001 to pxe, but dhcp doesnt know how to handle, need subnet details.
• 14:34 notpeter: lucene hosed on search9 and search15. restarting, then will look after cause
• 13:14 Jeff_Green: restarting puppet/puppetmaster on stafford to experiment with report settings
• 02:10 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 28 02:10:34 UTC 2012

March 27

• 23:12 logmsgbot: tstarling synchronized php-1.19/cache/trusted-xff.cdb
• 20:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33789 - Enable botadmin usergroup on ml.wikipedia'
• 19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix lezwiki namespace'
• 19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove ruwiki arbcom talk from namespaceprotection'
• 19:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
• 18:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
• 18:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
• 18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
• 18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
• 18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
• 18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35290 - Create Slovenian Wikiversity'
• 18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35290 - Create Slovenian Wikiversity'
• 17:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
• 16:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
• 16:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
• 16:48 logmsgbot: reedy ran sync-common-all
• 16:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'prep work for new wikis'
• 16:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34527 - Create a Arbcom namespace on Russian Wikipedia'
• 16:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34527 - Create a Arbcom namespace on Russian Wikipedia'
• 15:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35161 - Incubator configuration updates'
• 15:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32825 - Favicon for siwiki'
• 14:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35516 - Add Skin: namespace to MW.org'
• 08:15 apergos: test you silly morebot

• 07:59:56 hashar: archived old server admin logs since the old page was too long for my connection to download :-/
• 06:59:02 apergos: !log powercycled emery, it was unresponsive via the mgmt console and not pingable
• 02:17:52 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 27 02:17:52 UTC 2012
• 00:56:51 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114507'
• 00:55:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/LeaveFeedbackTemplate.php 'r114508'
• 00:42:50 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bmping resource version for MobileFrontend'
• 00:41:58 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r114509'
• 00:37:30 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version #'
• 00:36:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/MobileFrontendTemplate.php 'r114507'
• 00:36:09 logmsgbot: awjrichards[00:36:36] synchronized php/extensions/MobileFrontend/templates/LeaveFeedbackTemplate.php 'r114508'
• 00:35:50 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r114508'
• 00:08:55 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114506'

March 26

• 23:18:17 logmsgbot: awjrichards synchronizing Wikimedia installation... : Syncing MobileFrontend to r114504 changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#26_March.2C_2012
• 22:44:53 RobH: !log also rolling firmware to ps1-d[1|2|3]-pmtpa
• 22:28:10 RobH: !log pushing firmware updates to servertechs in sequence: ps1-[a2|a3|a4|a5|b2|b3|b4|b5|c1|c2|c3|d1|d2|d3]-sdtpa, disregard any errors from rebooting alerts
• 19:55:09 notpeter: !log stopping puppet on search6 and search15 for 24 hours to test new log rotation script
• 19:19:35 RobH: !log cp1019 memory replaced per rt 2651
• 19:07:14 apergos: rebooting ms1001 (new kernel)
• 17:53:34 RobH: cp1019 coming down for memory replacement per rt 2651
• 17:51:39 RobH: fluorine disk upgrade done, os install pending, details on rt 2350
• 17:43:48 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r114492'
• 17:36:51 RobH: fluorine coming down for new disks
• 17:14 notpeter: backingup plwiki.nspart1 index on search7, deleting working copy, and restarting lsearchd. (note: this will probably cause some downtime on some languages while the proc restarts...)
• 15:18 RobH: db59 has errors, but as it was a fusion io testbed server, it is more than likely tweaked for such, it is not in any rotation
• 14:54 RobH: db59 shutting down for io card removal per rt 2589
• 13:37 mutante: while on it, installing a whole bunch of package updates on db42
• 13:25 mutante: db42 was out of disk , caused by ~5G citations.csv in /tmp, gzipped the file
• 09:59 mutante: ..and on ms-be-3. running puppet on db59
• 09:43 mutante: another corrupted .yaml file on ssl2
• 09:33 mutante: brewster - delete puppet lock file, restart lighttpd, puppet ...
• 09:05 mutante: brewster was out of disk - deleted lighttpd access.log.1, gzipped access.log
• 08:24 mutante: on several mw* boxes puppet did not run because .yaml files on the puppetmaster became corrupted. need to delete the $hostname files in /var/lib/puppet/yaml/node on stafford and re-run. puppet bug similar to http://projects.puppetlabs.com/issues/7836
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 26 02:18:03 UTC 2012

March 25

• 22:26 RobH: row b servertech firmware in eqiad all updated, should clear alarms as they come back online
• 22:18 RobH: firmware updates on servertechs in row b eqiad, disregard alarms
• 20:14 RobH: to fellow ops, you can disregard those observium errors, as I caused them
• 20:13 RobH: firmware updated on all power strips in row a eqiad.
• 16:22 RobH: ps1-a1-sdtpa firmware update complete
• 16:15 RobH: updating firmware on ps1-a1-sdtpa
• 16:14 RobH: ps1-b1-sdtpa firmware updated successfully
• 16:14 RobH: ps1-a1-eqiad firmware updated successfully
• 16:09 RobH: updating firmware on ps1-s1-eqiad and ps1-b1-sdtpa
• 16:07 RobH: updated firmware successfully on ps1-a8-eqiad, if it has observium alarms now then there are bigger issues.
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 25 02:17:21 UTC 2012
• 00:59 LeslieCarr: admin down asw-a-eqiad xe-1/1/2 and cr2-eqiad xe-5/0/0 due to framing errors causing packet loss and lacp sporadic timeouts. source of the issue

March 24

• 19:46 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
• 19:46 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
• 17:35 mark: Migration from br1-knams to cr2-knams completed.
• 17:09 mark: Migrated second knams-esams dark fiber link from br1-knams to cr2-knams
• 16:36 mark: Corrected MTU setting on cr2-knams's AMS-IX interface
• 16:20 Reedy: Some european users reporting oruting issues
• 16:01 mark: Cleared OSPF session between csw1-esams and csw2-esams which magically made some internal routes reappear
• 15:40 mark: Brought up AMS-IX ipv4 BGP sessions
• 15:30 mark: Brought up AMS-IX ipv6 BGP sessions
• 15:25 mark: Moved AMS-IX connection to cr2-knams:xe-1/1/0
• 15:22 mark: Shutdown all AMS-IX BGP sessions
• 15:06 mark: Disabled BFD on OSPF3 between cr2-knams and csw1-esams
• 14:49 mark: Moved AS6908 and AS1257 PIs to cr2-knams
• 14:18 mark: Brought up AS13030 and AS1299 BGP sessions on cr2-knams
• 13:57 mark: Shutdown AS1299 BGP session on br1-knams
• 13:14 mark: Established full iBGP mesh with added router cr2-knams. cr2-knams now has full Internet connectivity.
• 12:48 mark: Moved fiber from br1-knams:e1/2 to cr2-knams:xe-0/0/0
• 12:44 mark: Disabled br1-knams:e1/2 (DF leg 1 to esams)
• 12:43 mark: Rack mounted and powered up cr2-knams
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 24 02:17:02 UTC 2012

March 23

• 23:49 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ 'r114466'
• 23:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia'
• 23:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia'
• 23:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce'
• 23:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce'
• 23:07 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce'
• 23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce empty arrays'
• 22:24 RobH: scs-a1-eqiad back online
• 21:58 RobH: scs-a8-eqiad coming down for re-grounding
• 19:51 RobH: all power strips in eqiad are now properly grounded
• 18:12 maplebed: removed ms1 and most of ms2 from the production swift rings. no effect expected.
• 18:04 logmsgbot: asher synchronized wmf-config/db.php 'returning db32, pulling db52 for migration'
• 16:44 RobH: cp1019 in middle of firmware update, please dont touch
• 16:44 RobH: cp1017 memory error seems ot have cleared post firmware update, will keep an eye on it for the rest of the day
• 16:09 RobH: raid rebuilding on magnesium, however swift stuff is kind of black box mystery right now to me, need Ben to review magnesium later for that
• 15:53 RobH: magnesium coming back online
• 15:44 RobH: shutting down magnesium for disk swap
• 15:37 RobH: firmware updating on cp1017, no one touch it please
• 15:30 RobH: db1020 can go back into whatever rotation Asher wants it in
• 15:29 RobH: db20 memory error on raid controller resolved with firmware updarte
• 06:39 logmsgbot: tstarling synchronized php-1.19/includes/filerepo/file/LocalFile.php 'r114442'
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 23 02:18:35 UTC 2012
• 01:55 mutante: deleting puppet report files older than 60hours on stafford to free disk space

March 22

• 23:30 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 23:18 RobH: db1020 firmware still updating, will check on it later tonight. offline until then
• 22:19 notpeter: all 3 dns servers are responding to digs after reload
• 22:10 notpeter: pushing a new zone file to add 2 more search-related vips for eqiad
• 20:52 notpeter: stopping puppet on brewster temporarily
• 20:25 notpeter: rebuilding search1015 and 1016 for disk shuffles
• 20:01 RobH: magnesium goign down and up again, troubleshooting the disks
• 19:47 apergos: rebooting ms1002, had stuck rsyncs, and kswapds at 100% cpu, weirdness like "ls /export/upload/wikipedia/am/0/00" hanging.
• 18:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
• 15:45 RobH: search 1015 and search1016 back up with added disks
• 15:08 RobH: shutting down search1015 & search1016 for hdd additions
• 14:45 RobH: db1020 still offline, requires firmware update on raid controller per rt 2621, will perform later today
• 14:33 logmsgbot: reedy synchronizing Wikimedia installation... :
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 22 02:17:47 UTC 2012
• 01:14 K4-713: Re-enabled the donations queue consumer in Jenkins
• 00:28 binasher: started enwiki.revision alter on db32
• 00:26 binasher: disabled lvm snapshots and puppet on db32 for revision sha1 alter
• 00:24 logmsgbot: asher synchronized wmf-config/db.php 'pullin db32 for revision alter'

March 21

• 22:27 ^demon|away: wmf-deployed extensions now r/o in SVN
• 21:52 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php
• 21:27 Ryan_Lane: bringing up all instances on virt3
• 21:08 cmjohnson1: swapped 2 DIMMS in virt3 (b2 and b5)
• 21:01 Ryan_Lane: shutting down virt3 to replace dimms
• 20:47 ^demon: /trunk/phase3 is now r/o in SVN
• 20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable prefswitch'
• 20:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgArticleFeedbackv5OversightEmails on enwiki'
• 18:59 maplebed: rebooted ms-be3 after it crashed.
• 18:51 binasher: brought db24 back up after hang, and reslaving, but leaving out of db.php. just replicating until a replacement s2 snapshot host is built
• 18:51 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 update
• 18:46 logmsgbot: asher synchronized wmf-config/db.php 'returning db36'
• 18:35 logmsgbot: asher synchronized wmf-config/db.php 'pulling db24, failing hw'
• 18:03 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php
• 18:01 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily disable ShortUrl on testwiki because we think it might conflict with ArticleFeedbackv5'
• 17:59 K4-713: updated and synchronized payments cluster to r114382
• 17:47 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 12:25 notpeter: disabling notifications for search-pool1
• 08:58 mutante: rebooting ms-be4
• 08:37 mutante: stopped/started lsearchd on search9
• 08:05 mutante: ms-be4 down but cant powercycle it yet..Unable to establish LAN session / ipmitool /ipmi_mgmt
• 07:58 mutante: restarted lsearchd on search3 and 9
• 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/CoreParserFunctions.php
• 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/Parser.php
• 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/StripState.php
• 05:22 logmsgbot: tstarling synchronized php-1.19/tests/parser/parserTests.txt
• 03:51 mutante: added "lez" to langlist and running authdns-update, for lez.wikipedia per RT-2665
• 03:29 mutante: magnesium - shutting down, has existing RT-2669 to replace disk
• 03:18 mutante: magnesium - "..drive on port B of the Srial ATA controller is operating outsde of normal specifications.. Strike F1 key to continue"..
• 03:16 mutante: powercycling magnesium - down and just "init: tty4 main" on mgmt, frozen
• 03:10 mutante: running puppet on aluminium
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 21 02:18:10 UTC 2012
• 01:06 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114342'
• 00:25 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo'
• 00:03 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo'

March 20

• 23:19 Ryan_Lane: fixing the zero redirect
• 22:46 logmsgbot: reedy synchronized wikipedia.dblist 'test'
• 22:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/api/ApiQueryExtracts.php 'r114319'
• 22:09 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping resrouce version # for MobileFrontend'
• 21:54 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#20_March.2C_2012
• 21:46 binasher: stopped eqiad bits servers from udplogging to emery, packet loss is back to zero
• 20:59 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
• 20:17 binasher: killed enwiki.revision sha1 migrator (upgrade-1.19wmf1-2.php). after db36 completes, will run the rest by hand
• 19:52 Ryan_Lane: pushing change for zero.wikipedia.org to redirect to the english message
• 19:41 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page'
• 19:16 cmjohnson1: pulling disk 5 on virt1 for reseating
• 18:34 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page'
• 18:02 pgehres: flipped Template:CC-status on wmfwiki since credit cards are still disabled on payments.wikimedia.org
• 17:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35193 - Enable sub page feature in Telugu Wikisource'
• 17:49 notpeter: restarting lsearchd on search10
• 17:30 logmsgbot: aaron synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed r114285'
• 17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that then'
• 17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Test something for sewikimedia'
• 16:42 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Bug 35355 - Reset of permissions for Hindi Wikipedia'
• 16:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove hiwiki botadmin from whGRoupsRemoveFromSelf'
• 15:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
• 15:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
• 15:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
• 15:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35161 - Incubator configuration updates'
• 15:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31209 - Enable the WikiLove extension for incubator'
• 14:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove more group dupes'
• 14:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35355 - Reset of permissions for Hindi Wikipedia (hiwiki)'
• 14:14 logmsgbot: reedy synchronizing Wikimedia installation... : sscapping for r114268
• 14:08 logmsgbot: reedy synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'r114268'
• 09:12 mutante: new URL pointing to Wikipedia Education Program - http://education.wikimedia.org
• 08:59 mutante: several srv's said they were unable to contact NTP server
• 08:57 mutante: apache-graceful-all to deploy changed redirects.conf
• 08:53 logmsgbot: tfinc synchronized wmf-deployment/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'Fixes file pages showing data charge warnings'
• 07:42 mutante: running authdns-update after adding education.wm for redirect RT:2634
• 06:21 logmsgbot: tstarling synchronized php-1.19/includes/User.php
• 05:35 logmsgbot: asher synchronized wmf-config/db.php 'pulling db36 durring db migration'
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 20 02:17:55 UTC 2012
• 00:25 logmsgbot: awjrichards synchronizing Wikimedia installation... : Reverting MobileFrontend to r113973
• 00:15 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114221'
• 00:07 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling zero rated mobile access everywhere'
• 00:05 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumpging version number for MobileFrontend resources'

March 19

• 23:54 logmsgbot: awjrichards synchronizing Wikimedia installation... : Redoing accidentally aborted scap, Pushing changes to MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#19_March.2C_2012
• 23:51 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing changes to MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#19_March.2C_2012
• 23:35 AaronSchulz: fixed a few files, on commons and other wikis, with empty oi_archive_name values even though the file was on NFS
• 23:20 Ryan_Lane: restarting all nginx servers
• 23:20 Ryan_Lane: added a new proxy to the ssl configuration to temporarily proxy access to wikimania videos being transcoded
• 21:38 binasher: creating "ops" db and related grants on prod db clusters 2-7 to prep rollout of ishmael / pt-digest beyond s1
• 21:17 binasher: started enwiki.revision sha1 alter on production side
• 20:57 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Removing debugging code from MobileFormatter'
• 20:54 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
• 20:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
• 20:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
• 20:31 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Adding debugging code to MobileFormatter'
• 20:07 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r114176'
• 19:41 Ryan_Lane: bringing virt3 instances back up
• 19:33 binasher: deploying new frontend squid conf to add support for mf_useformat cookie [rt 2645]
• 19:18 K4-713: CiviCRM 4.1.1 update script finished executing on prod.
• 19:12 Ryan_Lane: shutting down virt3 for memory reseating
• 19:09 K4-713: Started the CiviCRM 4.1.1 update script on prod.
• 19:08 mark: Rebuilding RAID arrays on brewster
• 18:58 K4-713: Put production civicrm / drupal instance in offline mode for upgrade
• 18:54 K4-713: Disabled all production CiviCRM Jenkins jobs, for CiviCRM upgrade.
• 18:54 cmjohnson1: brewster HDD replacement complete
• 18:42 mark: Shutting down brewster for HDD replacement
• 18:26 Jeff_Green: killed kill-slow-queries on db1008 for the duration of the civicrm upgrade
• 18:19 logmsgbot: nikerabbit synchronized php-1.19/includes/Linker.php 'i18ndeploy r114160'
• 18:19 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/resources/ext.webfonts.fontlist.js 'i18ndeploy r114160'
• 18:14 mark: Running smartctl -t long /dev/sdb on brewster
• 12:58 logmsgbot: hashar synchronized php-1.19/includes/SiteStats.php 'Reenable SiteStatsInit::articles() for bug 35169. SiteStatsInit::doAllAndCommit() still disabled since it breaks the site'
• 10:28 logmsgbot: tstarling synchronized wmf-config/PoolCounterSettings.php 'increased max queue from 50 to 100 on reports that the limit was reached on the enwiki main page in normal operation'
• 09:11 mutante: nomcom and langcom wikis look kind of broken , redirecting to pages on incubator with "Error: This page is unprefixed! "
• 08:49 mutante: making (almost) all private wikis https-only per RT-2565, vi remnant.conf,sync,graceful...
• 07:30 mutante: running sync-apache after making a change to remnant.conf to make grants.wm https-only
• 05:09 Ryan_Lane: bringing up most instances on virt3, doing so by project priority
• 04:42 Ryan_Lane: bringing up all instances on virt4, waiting 30 seconds between instances
• 04:25 Ryan_Lane: bringing up all instances on virt2, waiting 30 seconds between instances
• 04:09 Ryan_Lane: bringing up all instances on virt1, waiting 30 seconds between instances
• 04:00 Ryan_Lane: attempting to bring some instances up
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 19 02:17:17 UTC 2012
• 01:15 mutante: killed, updated, restarted wikibugs bot per request in RT:2656, should have fixed bugzilla:18831

March 18

• 23:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35308 - Install mw:Extension:DynamicPageList (Wikimedia) on Portuguese Wikipedia (ptwiki)'
• 19:20 Ryan_Lane: stopping all labs instances, manually recovering gluster volume
• 15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35295 - Missing a in abusefilter-hide-log permission for oversighters'
• 10:49 Ryan_Lane: rebooting virt4 thanks to defunct libvirt process
• 03:43 Ryan_Lane: bringing all labs instances up
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 18 02:18:51 UTC 2012
• 01:09 Ryan_Lane: rebooting all of the virt hosts, gluster is having major issues
• 00:43 Ryan_Lane: rebooting virt2
• 00:40 Ryan_Lane: restarting glusterfs on virt2
• 00:11 Ryan_Lane: rebooting virt3 libirt is non-responsive
• 00:00 Ryan_Lane: bringing up instances that were downed on virt3

March 17

• 23:50 Ryan_Lane: virt3 crashed, powercycling it
• 23:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove old comments'
• 23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove old comments'
• 23:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat'
• 23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat'
• 23:02 logmsgbot: catrope synchronizing Wikimedia installation... : Have to scap for that AFTv5 change to propagate i18n change
• 22:52 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ 'r114087'
• 21:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35289 - Add wikisource logo to mobile wikisource gateway'
• 02:21 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 17 02:21:03 UTC 2012
• 01:23 AaronSchulz: FindFilesMissingDBRows.php done, list under aaron/output/missingFileDBRows
• 00:11 AaronSchulz: Running FindFilesMissingDBRows.php on all wikis

March 16

• 21:21 binasher: running enwiki.revision sha1 schema migrations on eqiad side
• 20:12 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild moodbar messages
• 20:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable moodbar on enwiki'
• 19:53 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ 'r114030'
• 19:15 Reedy: Ran namespaceDupes on stewardwiki
• 17:11 RobH: hdd in search1017/1018 replaced per rt 2583
• 16:54 RobH: search1017 and search1018 coming down for hdd swap
• 16:53 RobH: cp1017 back in service pool
• 16:43 RobH: cp1019 back in full service
• 16:22 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r114021'
• 16:22 RobH: cp1017 memory error, coming down for troubleshooting.
• 16:18 RobH: cp1019 memory error cleared after reseating, notes on rt 2651
• 16:09 mark: Migrated all varnish3 packages to newer varnish packages from git
• 16:08 RobH: cp1019 coming down for memory error troubleshooting
• 15:58 RobH: cp1040 repaired per rt 2611
• 15:48 RobH: cp1040 down for memory replacement
• 15:09 logmsgbot: reedy synchronized stylize.php 'Test for hume'
• 15:04 logmsgbot: root synchronized ufg.sql 'test sync to see if hume is fixed'
• 14:55 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php
• 14:04 apergos: restarted swift-container-auditor on ms-be3, it had died for some reason
• 08:07 mutante: i reverted that (star cert for wikitech), no worries i "shred"ded the files
• 07:51 mutante: replaced self-signed cert on wikitech with the star cert
• 04:19 mutante: on stafford, deleting spence's puppet report files to free some disk space (they are like the largest report files of all)
• 03:09 mutante: stafford - - /var/lib/puppet/reports is getting quite large (18G), and we got the first disk space warning, do we want to keep those?
• 02:45 mutante: killing nrpe on several hosts where it was running as the wrong user again (somehow through the use of dsh)
• 02:21 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 16 02:21:35 UTC 2012
• 01:12 mutante: stopping nagios-wm temp. while changing nrpe config (will watch it manually until it's back)
• 00:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe'
• 00:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe'

March 15

• 23:17 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113974'
• 23:12 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/DisableTemplate.php 'r113973, fixes bug 35249'
• 23:10 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/modules/ext.articleFeedbackv5/ext.articleFeedbackv5.js 'r113972'
• 22:59 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 22:59 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Bump AFTv4 event logging percentage from 25% to 100%'
• 22:57 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js
• 22:48 mutante: purging Lucene monitoring on indexer from db9, remove duplicate service definitions manually anyways (still tons left), run purge script, reload Nagios..
• 22:24 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 22:23 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Raise AFTv4 event logging percentage from 5% to 25%'
• 22:21 mutante: getting rid of Swift HTTP checks on non production machines manually (come on spence _purge_ ;P)
• 22:07 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 22:04 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Raise AFTv4 event logging percentage from 1% to 5%'
• 21:44 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 21:28 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113961'
• 21:25 pgehres: K4-713 synchronized payments cluster to r113956
• 21:25 pgehres: disabled credit cards on donate.wikimedia.org
• 21:21 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'fix fatal'
• 21:20 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Bump AFTv4 event logging percentage from 0.27% to 1%'
• 21:19 Ryan_Lane: rebalancing instances gluster volume
• 21:18 RoanKattouw: That was r113959
• 21:18 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js
• 21:11 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r113958'
• 21:09 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ 'r113957'
• 20:46 mark: bits.pmtpa cluster back online
• 20:44 RobH: dns update for silver and zhen servers
• 20:37 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php
• 19:54 RobH: sq67-sq70 have been reinstalled, but not signed in puppet, not sure if they are ready for that or if there are other items mark needs to change first
• 19:11 RobH: working on sq67-sq70 reinstalls, disregard alerts
• 19:00 RobH: db1022 resetup and redeployed per rt 2537 and assigned back to asher
• 18:51 logmsgbot: reedy synchronizing Wikimedia installation... : Running scap to deal with message changes earlier
• 18:19 RobH: db1022 coming down for reinstall and resetup of raid per rt 2537
• 17:55 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/ 'r113940'
• 17:54 logmsgbot: reedy synchronized php-1.19/extensions/CheckUser/ 'r113940'
• 17:53 logmsgbot: reedy synchronized php-1.19/extensions/wikihiero/modules/ext.wikihiero.css 'r113940'
• 17:52 logmsgbot: reedy synchronized php-1.19/extensions/NewUserMessage/NewUserMessage.class.php 'r113940'
• 17:41 logmsgbot: reedy synchronized php-1.19/includes/RecentChange.php 'r113938'
• 17:38 logmsgbot: reedy synchronized php-1.19/resources/mediawiki/mediawiki.util.js 'r113936'
• 17:37 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialUndelete.php 'r113936'
• 17:32 logmsgbot: reedy synchronized php-1.19/languages/messages/ 'r113935'
• 17:31 logmsgbot: reedy synchronized php-1.19/resources/ 'r113935'
• 17:31 logmsgbot: reedy synchronized php-1.19/includes/ 'r113935'
• 17:16 logmsgbot: reedy synchronized php-1.19/includes/SkinTemplate.php 'r113932'
• 16:13 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php 'r113929'
• 15:15 mark: Created git repo operations/debs/varnish in gerrit
• 14:06 apergos: disabled moodbar temporarily on en wikii, see bug 35245
• 14:02 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disable of feedback dashboard (right config var this time?)'
• 13:51 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disable of feedback dashboard'
• 13:11 apergos: on screen as root on dataset1001, copying to gluster volume; if this causes problems feel free to shoot it. ( cp -a 20120211 /mnt/glusterpublicdata/public/enwiki/ )
• 09:08 mutante: ran puppet on mw1020
• 08:12 mutante: installing apache,apt,cron,mysql-client upgrades on spence
• 07:51 mutante: messed with /var/lib/dpkg/status on hume to fix broken packages/remove "marked for purging" on libmysql-php5 without removing a ton of other packages, rather hackish but seems fine anyways, like not broken anymore on simulated dist-upgrade etc
• 07:01 mutante: uprading apache and apt on hume
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 15 02:17:35 UTC 2012
• 01:26 Ryan_Lane: labsconsole was missing libapache2-mod-php5. puppet must have tried to upgrade a package unsuccessfully
• 01:22 mutante: planet back up (installed libapache2-mod-php5 which installed apache2-mpm-prefork and removed apache2-mpm-worker)
• 01:19 mutante: planet down - apache on singer, syntax error in site config "Invalid command 'php_admin_flag'"
• 01:03 mutante: fixing nrpe "unable to read output" raid check on srv197,207,243,,244,253.. (nrpe running as wrong user)

March 14

• 23:16 maplebed: installed the swiftcleaner to run daily from iron. see root's crontab for more info.
• 20:41 binasher: disabled log_queries_not_using_indexes on all core dbs
• 20:33 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
• 20:30 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
• 19:29 maplebed: rebooting ms-be1 to enable hyperthreading (and make it the same as all the other ms-be hosts)
• 19:06 preilly: pushing x-images header for vary support
• 19:06 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
• 19:05 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'zero needs to add x-images to vary header'
• 18:58 maplebed: ms-be5 is back in rotatino
• 18:31 preilly: push zero change for carrier testing
• 18:31 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
• 16:19 RobH: updating dns for new domain wikimediacommons.pt (nameservers not yet pointed at us)
• 16:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'add vcs for extdist updates'
• 13:03 RobH: cp1029-cp1035 all installed and ready for varnish deployment, puppet has been run
• 08:24 mutante: running "apt-get -f install" on snapshot3 to fix dpkg, which installed mysql-client- and client-core-5.1
• 08:02 mutante: stop/start memcached on srv254,srv255,srv257
• 07:51 mutante: restarting mecached on marmontel
• 07:51 mutante: fixing owa[1-3] Swift HTTP commands manually
• 03:44 mutante: ekrem - user agent "AppleDictionaryService" requests cause temp. WAP outage ..it seems
• 03:38 mutante: free some disk space on spence - deleted user.log.1 on spence, compressing messages.1, apt-get clean,...
• 02:52 RobH: cp1032-cp1035 reinstall issue wiped mbr causing issues, will reinstall in my AM
• 02:49 RobH: revoked, cp1032 is some reason in grub error, and its too late at night for me to work on it, will troubleshoot tomorrow
• 02:48 RobH: realized i forgot to log hours ago that cp1029-cp1036 are installed with puppet run, ready for varnish deployment tomorrow
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 14 02:17:13 UTC 2012

March 13

• 23:51 mutante: upgrading bugzilla to 4.0.5
• 23:42 logmsgbot: reedy synchronized php-1.19/resources/jquery/jquery.textSelection.js 'r113786'
• 23:14 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
• 22:47 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113779'
• 22:44 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r113774'
• 22:44 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/common.css 'r113774'
• 22:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113771'
• 22:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/api/ApiQueryExcerpts.php 'r113774'
• 22:27 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Removing moile URL template for tewtwiki'
• 21:44 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 21:44 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
• 21:31 logmsgbot: asher synchronized wmf-config/db.php 'replacing db18 with new s7 slave db56'
• 21:19 binasher: started slaving db56 from db37
• 20:30 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
• 19:27 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
• 19:17 RobH: iron updated to use ipmi_mgmt script
• 19:08 preilly: pushing changes for zero to mswiki
• 19:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
• 19:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 19:05 binasher: streaming hotbackup of db1041 to db56 (new s7 slave replacing db18)
• 18:10 maplebed: failover successful, restarted pybal on lvs4, failback successful.
• 18:09 binasher: power cycling db1020, which also froze this morning
• 18:08 maplebed: stopping pybal on lvs4 - should fail over to lvs3
• 17:47 maplebed: pybal restarted on lvs3
• 17:47 binasher: power cycling db1040, crashed again
• 17:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 35183 - p include extensions/Renameuser/Renameuser.php instead of extensions/Renameuser/SpecialRenameuser.php'
• 17:12 mark: Sending all normally-pmtpa upload traffic to upload-lb.eqiad
• 17:05 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 16:59 preilly: add disable images support to mswiki under zero domain
• 16:59 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'add disable images option for mswiki on zero domain'
• 16:58 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add disable images option for mswiki on zero domain'
• 16:46 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mswiki remove from mywiki'
• 16:44 mark: Sending traffic from Japan, India, Mexico to upload-lb.eqiad
• 16:37 LeslieCarr: reinstalling neon
• 16:23 apergos: stole some free space from the phys volume on ms1002 to give us more time for the rsync to keep going til after the move to swift etc
• 15:28 mark: Sending traffic from the USA to upload-lb.eqiad
• 15:27 mark: Rebooting lvs1005 with upgraded kernel/packages
• 15:12 LeslieCarr: manually deleted cp1025 info from nagios config file - nagios restored for now
• 14:51 mark: Sending traffic from Canada to upload-lb.eqiad
• 14:32 mark: Sending traffic from Brazil to upload-lb.eqiad
• 13:58 mark: Sending traffic from Argentina to upload-lb.eqiad
• 12:58 mark: Seeding the eqiad upload caches from live upload requests
• 11:59 mark: Setup squid logging to oxygen, with oxygen relaying to multicast 233.58.59.1
• 11:02 mark: Rebooting lvs1002 with kernel updates
• 10:17 mark: Rebooting manutius with newer 2.6.36 kernel to attempt avoiding i/o kernel bug with torrus
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 13 02:18:03 UTC 2012

March 12

• 22:55 K4-713: synchronized payments cluster to r113679, and tweaked the anti-fraud rules
• 21:51 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r113671'
• 21:51 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113671'
• 21:44 Reedy: Running foreachwiki extensions/WikimediaMaintenance/cleanupBug31576.php in screen as me on hume
• 21:39 RobH: search1014 repaired per rt 2483
• 20:26 RobH: cp1040 coming down for hardware stuffs
• 18:19 Nikerabbit: Assuming scap has finished
• 17:48 logmsgbot: nikerabbit synchronizing Wikimedia installation... : Deploying updated Translate
• 17:46 notpeter: restarting indexer on searchidx2
• 17:24 logmsgbot: nikerabbit synchronized php-1.19/includes/Title.php 'r113635'
• 17:22 logmsgbot: nikerabbit synchronized php-1.19/languages/ 'r113635'
• 17:14 logmsgbot: nikerabbit synchronized php-1.19/extensions/Narayam/ 'Updating Narayam'
• 17:13 mark: PXE booting cp1025-cp1028
• 17:11 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'Updating WebFonts'
• 15:16 mark: Rebooted manutius, stuck in a similar state as streber always did
• 06:10 mutante: turning off debug mode in nagios-nrpe, again had to kill it , restart fails
• 05:53 mutante: dunno, copper was stuck (no mgmt output after reboot) but powercycling it and back
• 05:43 mutante: rebooting copper to make sure grub update didnt break it and asked for restart anyways
• 05:37 mutante: copper - installing (security) updates (apt,grub,openssl,ruby,libc6..)
• 04:19 mutante: wanted to restart nagios-nrpe-server on spence with debug=1 to investigate permission issue. arr! "Address already in use" "cant write to pidfile", killed the one started on Feb18, and reordered allowed_hosts, spence talks to itself again now :p
• 03:40 mutante: same (and nscd) on fenari
• 03:35 mutante: upgrading libc6 and related packages on spence
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 12 02:17:28 UTC 2012

March 11

• 08:14 apergos: restarted lighttp on dataset2
• 07:49 apergos: removed current htcp log file, restarted purger, it seems to be logging normallynow
• 07:35 apergos: current ls shows 17416851456 2012-03-11 07:34 HTCPpurger.log while current du -sh shows 175M for /var/log. Sparse file that gets rotated badly? lots of leading nulls (many gb worth), why?
• 07:33 apergos: on ms1004 the HTCPpurger.log file after rotation was 17 gb, filling the disk. Removed it.
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 11 02:17:35 UTC 2012

March 10

• 22:09 Reedy: Make that wikimania2012, not wikimediawiki
• 22:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable anon page creation for wikimediawiki'
• 19:28 binasher: set sync_binlog = 1 on all current masters and eqiad dbs
• 19:22 binasher: reslaved db1033
• 07:03 mutante: ran puppet on db1022, another one that works fine manually but somehow did not by itself
• 05:11 mutante: doing more (cp*, db*, msbe-* ,mw*) by hand / for loop
• 05:01 mutante: starting nagios-nrpe-server on all via dsh (fail to restart on config change issue)
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 10 02:16:57 UTC 2012
• 01:07 maplebed: started swiftcleaner on owa1 looking for (and purging) bad objects
• 01:06 maplebed: rebalanced the swift rings to finish decreasing traffic sent to ms1 and ms2
• 00:18 Ryan_Lane: powercycling ssl1003
• 00:18 Ryan_Lane: powercycling ssl1001

March 9

• 20:34 notpeter: stopping search indexer on searchidx2 for fresh rsync to searchidx1001
• 19:58 preilly: pushed change to remove description from landing page
• 19:57 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 18:59 Ryan_Lane: sending test.m.wikipedia.org to the same place as test.wikipedia.org via squid
• 18:58 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Fixing wgMobileUrlTemplate settings for domains that do not have .m. domains configured'
• 18:48 logmsgbot: reedy synchronized php-1.19/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.css 'r113497'
• 18:40 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Changing the way in which wgMobileUrlTemplate is configurable by InitialiseSettings.php'
• 18:39 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling wgMobileUrlTemplate for testwiki - hopefully for real this time'
• 18:34 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Making wgMobileUrlTemplate configurable by InitialiseSettings.php'
• 18:34 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling wgMobileUrlTemplate for testwiki'
• 17:40 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113489'
• 17:32 maplebed: set swift storage device weight on ms2 to 0 and pushed out rings
• 15:52 apergos: cleared up a little bit of space on root partition of snapshot2, but that's about it. I hope we never have 3 versions of mw in test at the same time, the tmp caches will kill us
• 15:52 mark: Turned off vcc_err_unref on all varnish servers, so varnish doesn't complain when ACLs/probes/backends are unused
• 15:44 Jeff_Green: hume apt upgrades, puppetd --test, switch to mysql 5.1.53-fb3753-wm1
• 06:38 Ryan_Lane: reloading autofs on all labs instances
• 06:13 Tim: running svn cleanup on extdist trunk
• 04:18 Tim: switched php and wmf-deployment symlinks over to php-1.19 instead of php-1.18
• 04:18 Tim: restarted morebots
• 00:57 pp-pdf2: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
• 00:57 pp-pdf3: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
• 00:57 pp-pdf1: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
• 00:38 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mywiki'
• 00:32 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.js 'fixes to code push'
• 00:32 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.min.js 'fixes to code push'
• 00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fixes to code push'
• 00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.js 'fixes to code push'
• 00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.min.js 'fixes to code push'
• 00:01 RobH: oxygen install done, booting successfully after multiple tests, now running puppet for initial config
• 00:01 K4-713: updated the paypal IPN listener on aluminium to r1450

March 8

• 23:57 logmsgbot: awjrichards synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113428'
• 23:56 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'per-wiki memory limit configuration, with extra memory for zh* for converter tables'
• 23:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'per-wiki memory limit configuration, with extra memory for zh* for converter tables'
• 23:42 mutante: rebooting ms-be5
• 23:37 logmsgbot: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments
• 23:24 binasher: streaming hotbacking of db1017 to db1033 - no snapshots of enwiki in eqiad til db1033 is back
• 23:19 Tim: started changing the php symlink to 1.19 instead of 1.18, but then changed my mind and changed it back.
• 23:16 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 23:07 logmsgbot: tstarling synchronized php-1.19/extensions/ExtensionDistributor/svn-invoker.conf
• 23:01 logmsgbot: asher synchronized wmf-config/db.php 'returning db24 to service'
• 22:58 maplebed: powercycled ms-be3 - it crashed 2.5 hours ag.
• 22:52 logmsgbot: asher synchronized wmf-config/db.php 'pulling db18'
• 22:40 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend 'deployed r113413, r113414'
• 22:39 LeslieCarr: poked hole to allow labs machines to reach gluster machines in tampa
• 22:13 logmsgbot: catrope synchronized php-1.19/includes/MagicWord.php 'r113411'
• 22:13 logmsgbot: catrope synchronized php-1.19/includes/Cdb.php 'r113411'
• 22:13 logmsgbot: catrope synchronized php-1.19/includes/WebRequest.php 'r113411'
• 22:11 RobH: udpating dns for oxygen
• 22:03 RobH: oxygen coming down for reinstall
• 20:42 cmjohnson1: power to msw-c1-sdtpa restore
• 20:40 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 20:40 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.php 'changes for zero'
• 20:39 cmjohnson1: removing and relocating power to msw-c1-sdtpa
• 19:38 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
• 19:34 RoanKattouw: Running scap for ArticleFeedbackv5 updates
• 19:30 RoanKattouw: Running AFTv5 schema changes on enwiki
• 19:29 logmsgbot: catrope synchronized wmf-config/CommonSettings.php '$wgArticleFeedbackv5OversightEmails'
• 19:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '$wgArticleFeedbackv5OversightEmails'
• 19:26 RoanKattouw: Applying AFTv5 schema changes to en_labswikimedia
• 19:09 preilly: push zero rated changes
• 19:09 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
• 19:09 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 19:04 RoanKattouw: Clearing message blobs
• 18:53 RoanKattouw: Running rebuildLocalisationCache.php
• 18:49 binasher: power cycling cp1044
• 18:46 binasher: purging entire mobile varnish cache - the main mobile template included robots no-follow
• 18:43 preilly: needed to fix a google issue with robots
• 18:43 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ApplicationTemplate.php 'remove ROBOTS metatag'
• 18:40 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ApplicationTemplate.php 'remove ROBOTS metatag'
• 18:40 binasher: deploying new squid frontend.conf to fix epic fail - all googlebot traffic was being redirected to mobile. now just if it's mobilegooglebot.
• 18:29 RoanKattouw: Applying AFTv5 schema changes on testwiki
• 18:27 RoanKattouw: Pushing new AFTv5 code to testwiki, do not sync to the live site just yet
• 17:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'ptwikipedia to ptwiki'
• 17:14 cmjohnson1: shutting down db18 for memory testing
• 16:57 RobH: search1014 still down per rt2483
• 16:47 maplebed: took ms-be5 out of rotation in the swift cluster - it's crashed 3 times now.
• 16:36 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'r113368'
• 16:31 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Revert live hack because it works, will come in properly'
• 16:30 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Test for bug 27246'
• 16:16 RobH: search1008 repaired
• 15:52 RobH: mw1103 finally repaired and ready for os and such
• 14:48 pp-pdf1: installed python faulthandler 2.1
• 14:47 pp-pdf3: installed python faulthandler 2.1
• 14:47 pp-pdf2: installed python faulthandler 2.1
• 14:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35012 - Namespace aliases for wikipedia and wikipedia-talk namespaces on Sanskrit wiki'
• 09:17 mutante: running puppet on mw1010 - finished quickly without problems - uh, wonder why Nagios reported puppet freshness then
• 08:22 mutante: cp1019 - Hitting F1 to continue reboot ( "Alert! System fatal error during previous boot")
• 08:21 mutante: cp1019 went down, then rebooted by itself (i think) after showing "idrac-8W82BP1 Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted"
• 07:54 mutante: cadmium fixed by adding groups::wikidev
• 07:41 mutante: puppet on cadmium broken due to dependency Group[500] for User[catrope]
• 07:20 mutante: ms1004 ran out of disk - caused by 17G HTCPurger.log.1, trying to gzip it now
• 06:52 logmsgbot: tstarling synchronized multiversion/MWMultiVersion.php
• 06:51 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
• 03:04 Guest32353: powercycled ms-be5; it has been unresponsive for 2 hours.
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 8 02:18:02 UTC 2012
• 01:32 AaronSchulz: fixBug34995.php done
• 01:26 AaronSchulz: running fixBug34995 on all wikis
• 00:17 Ryan_Lane: adding zero cnames
• 00:16 Ryan_Lane: installing newer wikimedia-task-dns-auth on all dns servers
• 00:15 Ryan_Lane: added wikimedia-task-dns-auth_0.18 to the repo, to add support for zero

March 7

• 23:05 logmsgbot: aaron synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed r113319'
• 22:39 maplebed: set swift weight for ms1 to 0 initiating the process to move data off the host in preparation for decomissioning it.
• 21:17 Jeff_Green: running apt upgrades and puppetd --test on srv194, srv197, srv203, srv212, srv213, srv230, srv244, srv245, srv252, srv282 and manually restarting nrpe because they're reporting funky in nagios
• 20:20 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 20:17 Jeff_Green: yet another redirects.conf change, per RT#2498 redirect wikimedia.com-->wikimedia.org
• 20:05 binasher: reverted no-pagecache rsync on search nodes - without corresponding index warmup in lsearchd, it just pushes back the pain a bit and does more harm than good
• 20:04 binasher: deployed support for zero.wikipedia.org and carrier tagging to mobile varnish servers
• 19:38 logmsgbot: catrope synchronized php-1.19/includes/resourceloader/ 'r113278'
• 19:27 Jeff_Green: manual apt-upgrade, puppetd --refresh, and repeat on srv265 because it was running on outdated apache config
• 18:44 RobH: correction sq39
• 18:36 RobH: pulled sq39 from text pybal config, pulled sq46 from upload pybal config
• 18:36 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 18:36 logmsgbot: catrope synchronized php-1.19/extensions/CustomUserSignup/modules/AccountCreationUserBucket.js 'touch'
• 18:12 RobH: shutting down sq38 and sq46 per rt 2581 for testing
• 16:02 cmjohnson1: replacing hdd for disk 10 on db22
• 16:00 cmjohnson1: pulling disk 10 from db22
• 13:28 mark: Removed torrus from streber
• 13:00 pp-pdf2: updated mwlib to 0.13.6
• 13:00 pp-pdf3: updated mwlib to 0.13.6
• 13:00 pp-pdf1: updated mwlib to 0.13.6
• 11:29 logmsgbot: hashar synchronizing Wikimedia installation... : trigger a rebuild of l10n cache
• 04:53 mutante: added ms-be5 drives to swift cluster
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 7 02:18:01 UTC 2012
• 02:11 logmsgbot: catrope synchronized php-1.19/includes/api/ApiBase.php 'r113212'
• 01:58 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'bumped max file size to 4GiB'
• 00:27 maplebed: put ms-be4 into rotation as a new production swift backend storage node
• 00:21 maplebed: put ms-be3 into rotation as a new production swift backend storage node
• 00:05 maplebed: put ms-be2 into rotation as a new production swift backend storage node

March 6

• 23:54 logmsgbot: catrope synchronized php-1.19/extensions/CustomUserSignup/ 'Belated sync of r113056'
• 23:52 binasher: deploying new frontend squid config to include googlebot in mobile redirects
• 23:36 logmsgbot: reedy synchronized php-1.19/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php 'r113200 reverting r113198'
• 23:25 Tim: patched 5xx-filter.c live on locke and reloaded udp2log to stop the segfaults
• 23:20 logmsgbot: reedy synchronized php-1.19/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php 'r113198'
• 21:46 logmsgbot: catrope synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r113183'
• 21:41 notpeter: restarting puppet on brewster
• 21:03 Jeff_Green: pushing another change to redirects.conf and doing a graceful apache restart
• 20:32 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild message cache stuffs for r113129
• 20:31 Jeff_Green: disabled Global Connect nagios test (check_gcsip) on payments cluster because GC is down and nagios is spammy
• 20:25 notpeter: reimaging search1001-1020 with new partman recipe :/
• 20:22 notpeter: temp stopping puppet on brewster
• 20:21 logmsgbot: reedy synchronized php-1.19/resources/mediawiki.action/mediawiki.action.edit.js 'r113175'
• 20:20 logmsgbot: reedy synchronized php-1.19/maintenance/populateRevisionSha1.php 'r113175'
• 20:19 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialContributions.php 'r113175'
• 20:18 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialUserlogin.php 'r113176'
• 20:00 pp-pdf1: installed log-wikimedia-operations (which can be used for automated logging to #wikimedia-operations)
• 19:53 Ryan_Lane: restarting labs mysql to allow for more connections
• 19:26 Ryan_Lane: installing nova-api on virt0
• 19:09 Ryan_Lane: upping FLAGS.sql_max_pool_size for nova-api
• 18:47 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 18:46 Ryan_Lane: rebooting all instances
• 18:34 Ryan_Lane: restarting nova-network on virt2
• 18:19 Ryan_Lane: rebooting virt1
• 18:15 Ryan_Lane: rebooting virt2
• 18:11 Ryan_Lane: rebooting virt3
• 18:07 Ryan_Lane: rebooting virt4
• 17:57 Ryan_Lane: taking the opportunity to apply security updates to virt0-4
• 16:25 logmsgbot: catrope synchronized docroot/foundation/FrameResize.html 'Put Jobvite frame resize file in foundationwiki docroot per Erik'
• 11:40 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching sr* to 1.19
• 11:15 logmsgbot: hashar synchronized php-1.19/languages/messages/MessagesSa.php 'r1113039 for bug 34938 : title is sometime empty on Sanskrit wikis'
• 11:13 logmsgbot: tstarling synchronized php-1.19/includes/OutputPage.php 'r113128'
• 10:41 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching zh* from 1.18 to 1.19
• 08:36 mutante: on hooper: puppet broken due to dependency Package[libapache2-mod-php5] for Service[apache2]
• 03:33 mutante: rebooting bast1001 for kernel upgrade
• 03:32 mutante: upgrading apache2 packages, base-files, kernel, several libs on bast1001
• 03:27 mutante: installing a couple upgrades on fenari (apache2-utils, update-manager-core, cvs, ruby, libxml*, libopenssl-ruby*...)
• 02:37 logmsgbot: LocalisationUpdate completed (1.18) at Tue Mar 6 02:37:06 UTC 2012
• 02:36 logmsgbot: tstarling synchronizing Wikimedia installation... : updating to r113119
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 6 02:18:13 UTC 2012
• 01:27 Jeff_Green: manually updated packages and restarted apache on srv198, srv229, srv262, srv268, mw40 because their apache redirect configs failed to update after sync-apache and restart
• 01:07 Jeff_Green: another adjustment to redirects.conf and apache-graceful-all for RT#2488

March 5

• 22:24 Jeff_Green: modified redirects.conf per RT #2488
• 21:21 Reedy: Ran foreachwiki cleanupUploadStash.php
• 20:36 maplebed: enabled swift for 100% of thumbnails in production
• 18:18 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r113058'
• 18:11 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'WebFonts: bugwiki bug 34550; sawikisource bug 34159; amwiktionary amwikiquote bug 34700'
• 18:01 mark: Raised MTU between cr1-sdtpa - (csw1-sdtpa) - cr2-pmtpa to 9192
• 17:35 Jeff_Green: removed 3GB db30:/tmp/gmond.log and force-restarted gmond b/c the init script failed to restart it
• 17:16 Jeff_Green: adjusted LVS partitions on hume, moved /usr/local/apache to a new 5GB mount
• 15:18 mark: Fixed DNS resolving on the core routers by allowing DNS replies in the loopback filter
• 14:44 logmsgbot: reedy synchronized php-1.19/includes/Title.php 'r113036'
• 14:43 logmsgbot: reedy synchronized php-1.19/includes/AjaxResponse.php 'r113036'
• 14:35 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/ 'r113035'
• 14:34 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/ 'r113035'
• 13:50 mark: Set increased OSPF/OSPFv3 metric 30 on both directions of the link cr1-eqiad:xe-5/2/1 <--> cr1-sdtpa:xe-0/0/1, to combat higher than normal jitter and packet loss on the link
• 12:53 mark: Upgraded observium to latest version
• 09:41 mutante: restarting memcached on marmontel
• 09:40 mutante: restarting squid backend on knsq25
• 06:52 Ryan_Lane: all of the instances are accessing the file descriptors of files inside of the _base directory, and fuse has an issue with this. gluster can't recreate the base directory because of the processes holding open the old one.
• 06:50 Ryan_Lane: I've corrupted the _base directory on the instance's glusterfs share. I'm recovering the files from file descriptors using lsof. Not totally sure how I'm going to get the _base directory back, yet.
• 02:33 logmsgbot: LocalisationUpdate completed (1.18) at Mon Mar 5 02:33:04 UTC 2012
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 5 02:16:39 UTC 2012

March 4

• 21:48 logmsgbot: reedy synchronized wmf-config/ 'Bug 32726 - Set =true for Commons'
• 21:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix .'
• 21:41 logmsgbot: reedy synchronized wmf-config/ 'Bug 32726 - Set =true for Commons'
• 21:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34897 - Enable Special:Import on Catalan wikisource'
• 20:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34567 - New logo for Arabic Wiktionary'
• 20:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34715 - Please modify the import sources for the Spanish Wikiversity'
• 20:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34694 - Install the Quiz extension on de.wikibooks'
• 20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wmgMoodBarCutoffTime'
• 20:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Create wmgMoodBarCutoffTime'
• 20:14 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Variablise moodbarconfig infoUrl'
• 20:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Variablise moodbarconfig infoUrl'
• 20:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34618 - Install MoodBar on fr.wikisource'
• 20:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34766 - Logo of Sanskrit Wikisource'
• 19:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34867 - Switch Sango wiktionary logo'
• 19:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34931 - Add namespaces aliases on as.wikipedia.org'
• 19:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34690 - Changing the name in the title bar to Assamese'
• 02:35 logmsgbot: LocalisationUpdate completed (1.18) at Sun Mar 4 02:35:16 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 4 02:17:34 UTC 2012

March 3

• 18:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34728 - Categories added to user pages by Babel in pt.wiktionary'
• 13:04 logmsgbot: aaron synchronized php-1.19/includes/Revision.php 'deployed r112949'
• 02:35 logmsgbot: LocalisationUpdate completed (1.18) at Sat Mar 3 02:35:08 UTC 2012
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 3 02:18:04 UTC 2012

March 2

• 21:41 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'disabled logging hack'
• 20:47 Jeff_Green: added redirect/301 from http://static.wikimedia.org --> http://dumps.wikimedia.org now that archival static html dumps are located there
• 19:53 mark: Decommissioned csw5-pmtpa from AS14907 service. rest in pieces ;)
• 19:10 mark: Did a hot cut to remove csw5-pmtpa out of the path of cr1-sdtpa -> csw1-sdtpa -> csw5-pmtpa -> cr2-pmtpa
• 17:46 cmjohnson1: powering down msw1-pmtpa for relcocation to d1-pmtpa
• 17:40 cmjohnson1: disconnecting management fiber from msw1-pmtpa
• 16:59 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'r112904'
• 16:55 RobH: ms-be4 boot order fixed, fixing ms-be5 & ms-be2
• 16:49 RobH: fixed boot order on ms-be3, fixing ms-be4
• 16:33 RobH: poking at bios on ms-be3
• 16:05 RobH: wikitech outage resolved
• 15:20 RobH: shutdown frdev offsite vm per email to engineering last week
• 15:18 RobH: backing up wikitech in hopes of upgrading some of its software
• 08:36 apergos: on ms1004, low on space, HTCPpurger.log.1 had about 16 gb of nulls before any real content, I tailed off the real stuff and tossed the original. The current log file has the same problem, why?
• 02:34 logmsgbot: LocalisationUpdate completed (1.18) at Fri Mar 2 02:34:34 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 2 02:17:51 UTC 2012
• 01:36 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/lockmanager/LockManager.php 'deployed r112867'
• 00:41 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree 'deployed r112862'

March 1

• 23:33 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'log agent'
• 23:29 logmsgbot: reedy synchronizing Wikimedia installation... : Push message updates from r112848
• 23:22 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'logging fix'
• 23:22 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
• 23:20 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
• 23:17 logmsgbot: reedy synchronized php-1.19/includes/filerepo/backend/FSFileBackend.php 'r112850'
• 23:16 logmsgbot: reedy synchronized php-1.19/includes/Article.php 'r112850'
• 23:11 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
• 23:06 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ApiFeedbackDashboardResponse.php 'r112848'
• 23:05 logmsgbot: reedy synchronized php-1.19/extensions/Collection/Collection.body.php 'r112848'
• 22:12 logmsgbot: aaron synchronized php-1.19/includes/specials/SpecialContributions.php 'deployed r112844'
• 22:06 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend 'deployed r112841'
• 21:04 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'enabled FileBackend debug log'
• 19:57 cmjohnson1: replaced disk 3 labstore1 chassis
• 19:54 cmjohnson1: removing disk 3 from labstore1 chassis
• 19:47 Ryan_Lane: restarted memcached on virt0
• 19:15 logmsgbot: reedy synchronized php-1.19/cache/interwiki.cdb 'Updating interwiki cache'
• 17:39 Jeff_Green: Removed >5GB /tmp/gmond.log on db25, db32, db33, db37
• 17:36 logmsgbot: hashar synchronized php-1.19/includes/EditPage.php 'r112819 - Bug 34849 diff during editing an old version compares to the old version instead of the current one'
• 17:36 Jeff_Green: Removed >5GB /tmp/gmond.log on db13
• 17:35 Jeff_Green: Removed >5GB /tmp/gmond.log on db11
• 17:25 Jeff_Green: Removed 5.3GB /tmp/gmond.log on db1018
• 17:24 Jeff_Green: Removed 5.3GB /tmp/gmond.log on db1017
• 17:13 Jeff_Green: Removed 4.8GB /tmp/gmond.log on db1008. Tried to resist urge to make snarky comment about ganglia but failed.
• 14:54 RobH: strontium server rebooting to set HT to enabled
• 14:26 mark: Moving bits traffic back from pmtpa to eqiad
• 14:24 mark: Cleared dnsmasq cache on virt2
• 14:16 mark: csw5-pmtpa: Mar 1 14:01:42:A:Power Supply 2 , 2nd from left, bad
• 14:14 mark: mr1-pmtpa rebooted/lost power for some reason
• 14:07 mark: pmtpa/sdtpa management network went down
• 13:54 mark: Pooled new eqiad bits servers strontium and palladium
• 12:45 logmsgbot: hashar synchronized php-1.19/includes/specials/SpecialWatchlist.php 'r111882 for Bug 34835 - watchlist shows times in UTC'
• 10:53 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: reverting sr* wikis back to 1.18 per Siebrand's recommendation due to bug 34832
• 06:26 logmsgbot: tstarling synchronized php-1.19/extensions/SpamBlacklist/SpamBlacklist.php 'r112781'
• 05:46 maplebed: started swift deletion run on owa1, 2, and 3.
• 02:33 logmsgbot: LocalisationUpdate completed (1.18) at Thu Mar 1 02:33:53 UTC 2012
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 1 02:16:52 UTC 2012
• 02:15 Ryan_Lane: vlan tagged virt5's eth0 and eth1 ports on csw1-sdtpa
• 02:12 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'debug logging'
• 02:02 logmsgbot: reedy synchronized php-1.19/resources/mediawiki.action/mediawiki.action.history.diff.css 'r112750'
• 01:59 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: all zh wikis back to 1.18
• 01:50 logmsgbot: aaron synchronized php-1.19/extensions/WikiLove 'deployed r112758'
• 01:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last 265 wikipedias over to 1.19wmf1
• 01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: s7 to 1.19wmf1
• 01:23 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'r112754'
• 01:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: s2 to 1.19wmf1
• 00:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Meanwhile, on wikipedia.... Hello ruwiki!
• 00:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki to 1.19wmf1
• 00:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwiki to 1.19wmf1
• 00:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.19wmf1
• 00:05 logmsgbot: tstarling synchronized php-1.19/extensions/Collection/Collection.body.php 'r112745'

Archives

• Server admin log/Archive 1 (2004 Jun - 2004 Sep)
• Server admin log/Archive 2 (2004 Oct - 2004 Nov)
• Server admin log/Archive 3 (2004 Dec - 2005 Mar)
• Server admin log/Archive 4 (2005 Apr - 2005 Jul)
• Server admin log/Archive 5 (2005 Aug - 2005 Oct), with history 2004-06-23–2005-11-25
• Server admin log/Archive 6 (2005 Nov - 2006 Feb)
• Server admin log/Archive 7 (2006 Mar - 2006 Jun)
• Server admin log/Archive 8 (2006 Jul - 2006 Sep)
• Server admin log/Archive 9 (2006 Oct - 2007 Jan), with history 2005-11-25–2007-02-21
• Server admin log/Archive 10 (2007 Feb - 2007 Jun)
• Server admin log/Archive 11 (2007 Jul - 2007 Dec)
• Server admin log/Archive 12 (2008 Jan - 2008 Jul)
• Server admin log/2008-08
• Server admin log/2008-09
• Server admin log/Archive 13 (2008 Oct - 2009 Jun)
• Server admin log/Archive 14 (2009 Jun - 2009 Dec)
• Server admin log/Archive 15 (2010 Jan - 2010 Jun)
• Server admin log/Archive 16 (2010 Jul - 2010 Oct)
• Server admin log/Archive 17 (2010 Nov - 2010 Dec)
• Server admin log/Archive 18 (2011 Jan - 2011 Jun)
• Server admin log/Archive 19 (2011 Jul - 2011 Dec)
• Server admin log/Archive 20 (2011 Dec - 2012 Feb), with history 2007-02-21–2012-03-27