Server admin log/Archive 21

Revision as of 16:21, 1 May 2012

May 1

16:03 notpeter: rebuilding db52 and db53 as s2 slaves
15:47 logmsgbot_: asher synchronized wmf-config/db.php 's1: raising db59,60 weights, pulling db52/53 for reuse'
09:23 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'hewiki account creation high throttle limits'
04:04 Tim: on all apaches, running "chmod -R a+rX /usr/local/apache/common-local/" to clean up after killed rsyncs which left files unreadable
02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf2) at Tue May 1 02:23:29 UTC 2012
02:21 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileFeedback.php
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue May 1 02:14:06 UTC 2012
02:06 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileOptions.php
01:51 Ryan_Lane: bringing up all labs instances with a 60 second lag
01:40 Ryan_Lane: rebooting virt0
01:35 Ryan_Lane: rebooting virt3
01:33 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/HtmlFormatter.php
01:26 Ryan_Lane: rebooting virt5
01:18 Ryan_Lane: rebooting virt4
01:03 Ryan_Lane: rebooting virt2
00:51 LeslieCarr: restarted swift-container-auditor on ms-be5
00:38 logmsgbot_: tstarling synchronizing Wikimedia installation... :
00:26 Tim: removed large syslogs from mw60 and ran sync-common
00:18 Tim: on mw60 there was an actual directory at /usr/local/apache/common/php where a symlink should have been. fixed

April 30

23:58 logmsgbot_: aaron synchronized php
23:44 RoanKattouw: Started Apache back up on mw60
23:39 RoanKattouw: Running scap-1 on the Apaches with dsh
23:38 RoanKattouw: Moved /home/catrope/php-1.19 to /home/wikipedia/lazy-backups/php-1.19
23:38 Reedy: mediawiki.org to 1.20wmf2
23:37 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mw.org to 1.20wmf2
23:35 RoanKattouw: Strike that, instead moving /home/w/common/php-1.19 to /home/catrope/php-1.19
23:34 RoanKattouw: Removing /home/w/common/php-1.19 , NFS might freak out a bit
23:31 RoanKattouw: Removed php-1.19 from mw60 , synced it, and restarted Apache
23:28 RoanKattouw: Synced docroot and purged varnish for static-1.20wmf2, bits seems to be working for 1.20wmf2 now
23:27 RoanKattouw: mw60 has full disk, stopping Apache for now
22:50 Ryan_Lane: rebooting virt5
22:42 Ryan_Lane: rebooting virt3
22:35 Ryan_Lane: rebooting virt4
22:28 Ryan_Lane: rebooting virt1
22:23 Ryan_Lane: bringing down all instances (yay gluster)
21:12 pgehres: re-enabled Jenkins jobs on Aluminium after db1008 reboot
21:11 pgehres: CiviCRM back to normal after db1008 reboot
21:07 Jeff_Green: db1008 gets kernel update and reboot
21:00 pgehres: put CiviCRM on Aluminium in maintenance mode for db1008 reboot
20:59 logmsgbot_: reedy synchronized php-1.20wmf2/resources/startup.js 'touch'
20:57 pgehres: disabled all Jenkins jobs on Aluminium in prep for db1008 reboot
20:50 Jeff_Green: db1025 and storage3 get new kernels and reboot
20:28 notpeter: restarting, once again, innobackupex from db1034 to db57 for new s2 slave after fenari crash killed my screen
20:24 Reedy: Running ddsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 '/usr/bin/scap-1' in the hope it syncs all the files that would be nice to be on the app servers
20:18 logmsgbot_: reedy synchronized php-1.20wmf2/cache/ 'Synching whole cache directory'
19:59 notpeter: restarting nagios to get rid of some old checks
19:57 Jeff_Green: payments cluster gets kernel updates and reboots
19:55 logmsgbot_: reedy synchronizing Wikimedia installation... : Rebuiild l10n for 1.20wmf2
19:49 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.20wmf2.php 'Syncing file'
19:49 logmsgbot_: reedy synchronized php-1.20wmf2/LocalSettings.php 'Pushing LocalSettings.php'
19:48 paravoid: upgraded & rebooted ssl3001, ssl3002, ssl3003
19:45 logmsgbot_: reedy synchronizing Wikimedia installation... : Pushing out new symlinks etc, moving test2wiki to 1.20wmf2
19:30 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 live hack revisions'
19:28 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf1 live hack revisions'
19:26 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 for deployment'
19:18 Reedy: Syncing php-1.20wmf2 files from NFS to apaches. Likely to upset NFS (or the uplink for the switch nfs is on) for a little while...
19:14 paravoid: rebooting ssl1004
19:06 paravoid: rebooting ssl1003
19:00 paravoid: rebooting ssl1002
18:59 notpeter: starting innobackupex from db1034 to db57 for new s2 slave
18:50 paravoid: rebooting ssl1001
18:42 Jeff_Green: grosley gets new kernel + reboot
18:35 Jeff_Green: aluminium gets kernel update, yayyyyyyy!
18:34 paravoid: pooled back ssl1; depooling ssl3 and rebooting
18:29 binasher: rebooting mw45 for kernel upgrade
18:27 Jeff_Green: power cycling aluminium which faceplanted
18:22 binasher: rebooting mw45
18:21 notpeter: rebuilding db57 again, this time with more correct raid level!
18:19 logmsgbot_: asher synchronized wmf-config/db.php 'adding db59,60 to s1 with low weights'
18:16 paravoid: depooled & rebooting ssl1
18:09 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Sanity run after script changes.
18:00 logmsgbot_: aaron synchronized multiversion
17:58 logmsgbot_: reedy synchronized php-1.20wmf1/includes/MagicWord.php 'https://gerrit.wikimedia.org/r/6135'
17:44 logmsgbot_: aaron synchronized wikiversions.cdb
17:43 AaronSchulz: updating multiversion code
08:34 mutante: reinstalling srv266
08:08 mutante: upgraded mw1,mw2,mw35
07:59 mutante: reinstalling srv206
07:50 mutante: upgrading mw36
07:37 apergos: powercycling srv266, had this message on mgmt console: Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted
07:22 mutante: installing upgrades on srv212
07:19 apergos: reinstalled srv284, seems to be up now
07:17 mutante: powercycled mw8
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Mon Apr 30 02:13:59 UTC 2012

April 29

20:13 apergos: srv206 won't run puppet, see syslog, clearing out the yaml file didn't help, since it's not urgent I'm leaving it for tomorrow
19:51 Ryan_Lane: depooling ssl3004
19:51 Ryan_Lane: removed the ipv6 addresses from maerlant and added them to ssl3001, then restarted nginx
19:50 Ryan_Lane: repooling ssl3001
19:46 apergos: powercycled mw60, same reason as the rest
19:12 apergos: power cycled mw48 and mw52 (hung just like the others)
18:05 apergos: sll3002 and 3003 were rebooted and are the entire ssl esams pool right now
18:02 apergos: ok the ssl300x situation: ssl3001 is now disabled in the pybal conf file on fenari; it is picking up the ipv6and4labs tmplate and I don't know if that's right, anyways nginx doesn't want to bind to one of those addresses. ssl3004 isn't reachable or pingable even via mgmt but at leasy lvs sees it's gone
16:34 apergos: powercycling the ssl300x.esams hosts. 212 days of uptime... (and 3001 had gone out to lunch)
12:34 mutante: and finally mw1, so just leaving mw1102 and mw60 for having other issues for a while (->Nagios)
12:22 mutante: check_all_memcached recovered, but still same treatment for mw10 and 11 (8 and 15h ago)
12:15 mutante: powercycling mw32,mw33,mw44,mw46 one by one, they were all frozen and went down between like 17 and 24 hours ago approx.
12:07 mutante: powercycling mw30
02:56 paravoid: rebooting ssl2 (has 214 days uptime)
02:47 paravoid: powercycled ssl3
02:13 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sun Apr 29 02:13:58 UTC 2012

April 28

22:53 Reedy: Job queue logs on gdash seem to have stopped on the 26th...
22:29 logmsgbot_: reedy synchronized php-1.20wmf1/includes/EditPage.php 'https://gerrit.wikimedia.org/r/6088'
21:52 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
21:51 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
21:12 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
21:10 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
21:09 logmsgbot_: reedy synchronized common/php-1.20wmf1/extensions/cldr/LanguageNames.body.php 'more debugging'
20:51 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php 'Add debugging'
20:49 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Add debuglog group for language code not being a string'
19:04 logmsgbot_: reedy synchronized php-1.20wmf1/includes/ExternalEdit.php 'https://gerrit.wikimedia.org/r/6077'
19:03 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api/ApiParse.php 'https://gerrit.wikimedia.org/r/6076'
02:24 Ryan_Lane: rebooting all mediawiki boxes that have uptimes affected by the bug are being rebooted at 8 minute intervals
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 28 02:14:14 UTC 2012
01:33 paravoid: powecycled mw29
01:21 paravoid: powercycled mw38
00:17 notpeter: db12 is sooooo sloooooow, starting innobackupex from db1017 to db60 for new s1 slave

April 27

22:15 paravoid: upgraded ssl4 to nginx 0.7.65-5wmf1 and added it back to the pool
21:45 paravoid: rebooting ssl4 after upgrading (incl. a kernel update)
20:00 notpeter: starting innobackupex from db1040 to db1022 for new eqiad s6 snapshot slave, again
19:59 notpeter: starting innobackupex from db12 to db60 for new s1 slave, again
19:58 notpeter: starting innobackupex from db1017 to db59 for new s1 slave, again
19:49 paravoid: de-pooling ssl4
19:30 mutante: test - added new gerrit interwiki prefix for SAL/wikitech - gerrit:6002
19:14 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Fix rights for afttest and afttest-hide groups'
18:25 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Cleanup enotif related settings'
18:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnotifWatchlist to true for all wikis. Leaving wgShowUpdatedMarker set to false for all the big wikis'
16:50 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Simplify enotif code'
16:45 notpeter: starting innobackupex from db1040 to db1022 for new eqiad s6 snapshot slave
16:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgEnotifWatchlist defaulting to true. Big wikis explicitly set to false'
12:25 mutante: fixing integration.mw testswarm and applying fixed erb template by hashar
04:35 Tim: added an account for myself on observium
04:22 logmsgbot_: tstarling synchronized wmf-config/mc.php 'increased wgMemCachedTimeout from 500ms to 3000ms for bug 35900'
02:13 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 27 02:13:51 UTC 2012
00:12 Ryan_Lane: upgrading gluster on all instances
00:09 Ryan_Lane: upgrading gluster on labstore1-4

April 26

23:46 logmsgbot_: asher synchronized wmf-config/db.php 'raising db58 weight'
23:09 Reedy: Recreated resources directory symlinks in bits docroot
21:21 LeslieCarr: started deletion script on ms-be4
19:20 notpeter: restarting puppet on db59
19:18 Ryan_Lane: made LiquidThreads disabled by default on labsconsole, now users must add the special string to a page to enable it there.
19:18 Ryan_Lane: enabled NewUserMessage on labsconsole
19:06 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Add group permissions settings for AFTv5'
18:33 logmsgbot_: catrope synchronizing Wikimedia installation... : Deploy AFTv5 updates
17:17 LeslieCarr: reloaded varnish on mobile caches
14:19 notpeter: cleaned log space on search1017 and search1018 and started lucene
14:04 notpeter: stopping lucene on search1017 and 1018 to take that out of the equation
13:57 mutante: installing some (security) upgrades on fenari (apt,cron,samba,...)
13:54 notpeter: restartin lucene on search1017 and search1018
13:27 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayamon tewiki bug 33480'
13:23 logmsgbot_: nikerabbit synchronized php-1.20wmf1/extensions/Narayam/ 'Updating Narayam'
13:03 notpeter: (re)starting innobackupex from db1017 to db59 for new s1 slave
12:56 mark: Created precise-wikimedia APT distribution
08:27 mark: Power cycled mw40
06:57 binasher: restart pybal on amlvs1 with bgp disabled
06:57 binasher: restarted pybal on amlvs2 with bgp enabled
06:47 binasher: restarting pybal on amslvs2
06:26 binasher: shifting all traffic out of esams
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Thu Apr 26 02:14:03 UTC 2012
01:42 Ryan_Lane: starting mysql on db46
01:40 Tim: on professor: restarted udpprofile collector
01:37 Ryan_Lane: powercycling db46
01:33 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db46, host down'
00:44 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php

April 25

22:14 LeslieCarr: restarted swift-container-auditor on ms-be3
21:55 RobH: pushing dns update for scs-c1-eqiad and ps1-c#-eqiad
21:22 LeslieCarr: reloading varnish on mobile caches cp1041 cp1042 cp1043 cp1044
21:21 LeslieCarr: clearing mobile varnish cache
19:38 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'Attempted fatal fix'
19:33 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/Math/ 'Deploying 4c9e7dbe761c798ce15d7e2acef829a1582c058b'
19:14 notpeter: starting innobackupex from db12 to db59 for new s1 slave, per mr. feldman's directions
18:56 notpeter: starting innobackupex from db1017 to db60 for new s1 slave
18:49 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/FeaturedFeeds/SpecialFeedItem.php 'Deployed 4fb14a7b2ca9be715b820a9847d999f21c7d2cfc'
18:36 logmsgbot_: aaron synchronized php-1.20wmf1/img_auth.php 'Deployed f7e49bd71bd8356751242c5ce1cbae076a27cf7a'
18:10 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moving all remaining wikis to php-1.20wmf1
17:07 LeslieCarr: reloaded mobile varnish configs
17:06 LeslieCarr: purging mobile cache
16:40 LeslieCarr: starting delete script on ms-be3
16:14 RobH: done moving mgmt connections and serial connections in s8-eqiad for now
16:05 RobH: reshuffling cables in eqiad for serial and mgmt connections in a8, this may affect all eqiad mgmt and serial connections for the next 5 minutes
15:29 hashar: hashar: gallium: MySQL had issues most probably because of the mysql configuration snippets. https://gerrit.wikimedia.org/r/5796 might solve that.
14:03 mutante: gallium - don't start puppet unless the erb template fix for mysql has been merged
13:52 mutante: gallium stopped puppet, moved log_slow_queries config, re-setting up mysql again
13:41 mutante: gallium/testswarm - back up after mysql upgrade and issue starting the service
13:36 mutante: gallium - dpkg-reconfigure mysql-server-5.1, mysql does not start right
13:27 mutante: running apt-get upgrade on gallium
12:29 mark: Sending US, Brazil, Indian traffic to upload.eqiad
11:39 mutante: running authdns-update to add analytics100x and labsdb100x mgmt names
05:35 paravoid: powercycled lvs6, was dead and not responding to serial
03:43 logmsgbot_: asher synchronized wmf-config/db.php 'adding db58 to s7 as a new slave with a low weight'
03:24 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db58'
03:23 logmsgbot_: asher synchronized wmf-config/db.php 'adding db58 to s7 as a new slave with a low weight'
02:28 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Wed Apr 25 02:28:47 UTC 2012
02:14 logmsgbot_: LocalisationUpdate completed (1.19) at Wed Apr 25 02:14:46 UTC 2012
00:02 binasher: profiling collector was pegged at 100% cpu and graphs were turned to swiss cheese due to a bad stats call in 1.20, now fixed

April 24

23:59 binasher: powering off db16
23:55 binasher: streaming hot backup of db1041 to db58 (building a new s7 slave)
23:48 logmsgbot_: aaron synchronized php-1.19/includes/Setup.php 'Hacked out session request stats.'
23:46 logmsgbot_: aaron synchronized php-1.20wmf1/includes/Setup.php 'Deployed 42fcd43299246ecd1b265fcfcdd01a60319cf378'
23:19 AaronSchulz: Running 'mwscriptwikiset maintenance/populateRevisionSha1.php all.dblist' on hume
22:43 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Enabled file change journal on wikis using the new backend config.'
22:20 AaronSchulz: Tables added
22:18 binasher: rebooting db16 with updated kernel. it's probably still hopeless (dimm errors)
22:18 AaronSchulz: Creating the filejournal table on all wikis
21:59 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched commonswiki to the new backend config format.'
21:48 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db16, memory errors'
20:13 apergos: re-enabled replication via cron on ms7, it should catch up within an hour or so
20:10 binasher: reimaged db58 with fixed raid setup, imaging db59
19:51 notpeter: starting innobackupex from db1034 to db57 for new s2 slave
19:50 Ryan_Lane: repooling ssl3001
19:28 Ryan_Lane: depooling ssl3001
18:18 LeslieCarr: deploying to frontend
17:48 notpeter: deploying new squid conf to cp1001 frontend. is just a udp2log port change.
17:19 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Using newer backend for shared repos for testwiki, test2wiki, and mediawikiwiki.'
16:55 logmsgbot_: nikerabbit synchronized wmf-config/CommonSettings.php 'Translate extension configuration changes'
11:54 apergos: after much cursing and kicking zfs, a manual snapshot replication is running in screen as root on ms7 to ms8, expect it to take at least a day
11:44 mark: Sending all non-european upload traffic back to pmtpa to prepare for eqiad varnish storage rework
08:56 mutante: updated blog theme per guillaume (April commits)
08:05 apergos: temporarily disabled automatic zfs replication from ms7 -> ms8, cleared out space on ms8, catching up by hand
04:00 Ryan_Lane: powercycling ssl1
02:47 logmsgbot_: aaron synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Fixed notice spam.'
02:45 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/MobileFrontend/MobileFrontend.body.php 'Fixed notice spam.'
02:37 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Restructed filerepo a config a bit; nothing changed yet.'
02:28 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue Apr 24 02:28:47 UTC 2012
02:15 logmsgbot_: LocalisationUpdate completed (1.19) at Tue Apr 24 02:15:00 UTC 2012
00:15 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/stylesheets/common.css '0be2dc1288361c51f91533f1f77e78d9279b86e0'
00:13 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/common.css 'r115019'

April 23

23:35 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bumpging MobileFrontend resource version'
23:07 logmsgbot_: awjrichards synchronizing Wikimedia installation... :
23:02 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Add code for new URL scheme based on version_compare() logic'
22:51 logmsgbot_: awjrichards synchronizing Wikimedia installation... : MobileFrontend updates per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#23_April.2C_2012
22:33 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobile frontend resource version'
21:49 logmsgbot_: catrope synchronized php-1.20wmf1/resources/mediawiki/mediawiki.js 'Deploy 6e55a770b26b17b8fc9b5b4fe943dcc2867df4f3'
21:27 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'Deploy 93d470b'
20:41 mutante: neon - upgraded libssl, started icinga after adding monitor group
20:32 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/FileRepo.php 'Disabled write checks in the cleanDir() function.'
20:31 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/FileRepo.php 'Disabled write checks in the quickImport/quickPurge functions.'
19:43 logmsgbot_: catrope synchronized php-1.20wmf1/includes/specials/SpecialListgrouprights.php 'Deploy 047543b6805a268c8d689a7a1ce12ec545ef79a9'
18:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
18:43 logmsgbot_: reedy synchronized flaggedrevs.dblist 'Seems I never added ukwiki to the dblist... Oh well'
18:32 logmsgbot_: aaron synchronized wikiversions.dat
18:32 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf1
18:28 logmsgbot_: aaron synchronized php-1.20wmf1/includes/specials/SpecialContributions.php 'Deployed 72969cf8c9a403430c8c93fc20ab3118328c4d9c'
17:06 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use the newer backend config.'
14:33 notpeter: stopping puppet on cp1041 as well
14:17 notpeter: temp stopping puppet on cp1042-1044
13:09 mutante: powercycling frozen mw25, looks like mw21 above but no console output to paste here
13:07 mutante: fix puppet run on spence by removing searchidx1 resources from db9 (was in weird state being in site but also decommissioned)
11:23 mutante: mw21 powercycling mw21 - it died with this http://etherpad.wikimedia.org/mw21
10:55 mutante: force-reload ircecho on manganese to make gerrit-wm rejoin #mediawiki
10:48 hashar: banned CIA bots from #mediawiki IRC channel. It started spamming us with notifications from KDE and mandriva projects. See http://permalink.gmane.org/gmane.science.linguistics.wikipedia.technical/60905
10:30 mutante: searchidx1 was in site.pp and decom.pp at the same time. breaks puppet runs on spence. cannot override local resource. removing from site
10:27 mutante: killed a couple morebots processes on wikitech and it came back by itself :p

April 21

02:29 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 21 02:29:40 UTC 2012
02:15 logmsgbot_: LocalisationUpdate completed (1.19) at Sat Apr 21 02:15:20 UTC 2012

April 20

22:03 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched test2wiki to use the new LocalRepo config style.'
22:01 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched testwiki to use the new LocalRepo config style.'
21:52 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Added NFS backends for local/shared repos; they are not used yet.'
21:12 LeslieCarr: starting swift delete script on ms-be2
20:02 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/file/LocalFile.php 'deployed c77fbd394cda701758ad4523113f567bff7ede66'
19:45 apergos: powercycled mw4, it was unresponsive to pings and via mgmt
18:48 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449'
18:48 logmsgbot_: catrope synchronized php-1.19/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449'
18:07 notpeter: restarting nginx on ssl1002 and ssl1004 as they are not back up
18:01 logmsgbot_: awjrichards synchronizing Wikimedia installation... :
17:31 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Remoev wgArticleFeedbackv5OversightEmails override that was messing things up'
17:15 notpeter: stopping puppet on locke and emery. just to be safe...
17:11 RoanKattouw: Fixed ownership of /h/w/common/php-1.20wmf1/cache/l10n , should be owned by l10nupdate but was owned by reedy
17:01 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36124 - Deploy ProofreadPage extension on test2'
17:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Giving test2wiki moar namespaces'
16:11 mutante: add missing memcached servicegroup to nagios, restarted
15:10 mutante: apache error log on stafford has ruby exceptions re: phusion_passenger
15:01 mark: Converted OSPF directly connected redistributed routes from type 2 to type 1
14:51 mutante: starting swift-container-auditor on ms-be1
14:30 mark: Disabled down-pref of Tampa AS2828 routes
13:14 logmsgbot_: demon synchronized php-1.20wmf1/maintenance/backupTextPass.inc 'Pushing out Idb58ce27 for Ariel/Chris for dumps'
13:10 mark: Sending India upload traffic to upload-lb.eqiad
12:40 mark: Disabled iptables firewalls on internal prod swift cluster servers as it's dropping packets
12:22 mutante: restarted pdns on ns2
11:19 mark: Sending US upload traffic to eqiad as well
10:27 mark: Sending Brazil upload traffic to eqiad
08:39 hashar: Gave up running l10nupdate script it has some file permissions issues. Opened bug 36119 and bug 36120
08:36 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 08:36:53 UTC 2012
08:27 logmsgbot_: LocalisationUpdate completed (1.19) at Fri Apr 20 08:27:36 UTC 2012
08:13 hashar: rerunning l10nupdate for bug 34938
08:02 hashar: running l10nupdate for bug 34938
06:27 pgehres: re-eanabled PayPal on donatewiki and wmfwiki and resumed queue consumer on Aluminium
05:32 LeslieCarr: flushing mobile varnish cache
04:56 pgehres: disabled paypal on donatewiki and disabled queue consumer for duration of PayPal outage
02:33 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 02:33:02 UTC 2012
02:23 logmsgbot_: LocalisationUpdate completed (1.19) at Fri Apr 20 02:23:57 UTC 2012
01:47 logmsgbot_: awjrichards synchronizing Wikimedia installation... : r114983 on wikis still running 1.19

April 19

23:33 binasher: powercycled es1004
21:08 Jeff_Green: changed nagios contactgroup fundraising from tfinc/awrichards --> jgreen
21:03 RoanKattouw: Scap is broken in some weird way, it just stops running after the scap1-skins step. Doesn't run scap-1 (which does the actual sync), doesn't log "sync done", doesn't update graphite
21:01 logmsgbot_: catrope synchronizing Wikimedia installation... : Running scap again, AFTv5 is acting up
19:34 logmsgbot_: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
19:29 RoanKattouw: Running scap to deploy AFTv5 updates, and running AFTv5 schema changes on enwiki at the same time
18:50 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Set wmgArticleFeedbackv5OversightEmails for enwiki'
18:25 notpeter: nothing obvious in logs on db1005, starting mysql
18:15 notpeter: rebooting db1005. it's dead, jim.
17:52 RoanKattouw: Running schema changes for AFTv5 on testwiki
17:51 Jeff_Green: discovered nfs1 had ~1K redundant iptables rules, removed extras and reloaded
17:42 Jeff_Green: discovered sanger had ~7K redundant iptables rules, removed extras and reloaded
13:56 mutante: adding refreshLinks cron jobs to hume per RT-2355 (via puppet). if there should be any performance issues, schedule can be changed like <cluster>@<hour> in mediawiki.pp (and/or remove mediawiki::refreshlinks from hume and clear out the jobs of user mwdeploy)
08:35 mutante: emery - "udp2log_age" says some squid logfiles have not been written to in 6 hours, but from the filenames looks like this isnt a reason to worry, right
07:49 mutante: stat1 - this also needs udp2log stuff fixed. currently Could not find class misc::udp2log::udp-filter
07:47 mutante: gilman - what's up with it? closes SSH, does not like mgmt pass, was running jenkins but broken
07:43 mutante: owa[1-3] They dont have real puppet freshness issues, it's rather firewalling and the snmp traps
02:30 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Thu Apr 19 02:30:33 UTC 2012
02:21 logmsgbot_: LocalisationUpdate completed (1.19) at Thu Apr 19 02:21:31 UTC 2012

April 18

22:55 LeslieCarr: updating exim4.conf on mchenry to not allow old ranges
21:03 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
20:47 logmsgbot_: catrope synchronized php-1.20wmf1/resources/startup.js 'touch'
20:46 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/SyntaxHighlight_GeSHi/ 'Deploying GeSHi fix https://gerrit.wikimedia.org/r/#change,4949'
20:04 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: specieswiki and foundationwiki to 1.20wmf1
19:56 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/LiquidThreads/classes/Hooks.php 'Avoid fatals on invalid title in API'
19:51 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All *wiki wikis to 1.20wmf1
19:25 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote and wikiversity projects to 1.20wmf1
19:22 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikibooks to 1.20wmf1
19:18 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikinewses to 1.20wmf1
19:07 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisources to 1.20wmf1
19:07 logmsgbot_: catrope synchronized wmf-config/mc.php 'Swap out 10.0.2.251 (down) with 10.0.11.24 (spare). This is the last spare, there are now NO SPARES LEFT in mc.php'
19:00 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf1
18:57 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/LiquidThreads/classes/Dispatch.php 'Added type hint for better fatals'
18:44 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikiversity to 1.20wmf1
18:43 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikiquote to 1.20wmf1
18:41 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikibooks to 1.20wmf1
18:40 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf1
18:39 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwiktionary to 1.20wmf1
18:32 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikisource to 1.20wmf1
17:20 logmsgbot_: catrope synchronized docroot/bits/ 'Remove static-1.00 again'
16:57 logmsgbot_: catrope synchronized docroot/bits 'Add docroot/bits/static-1.00 for testing'
16:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wmfUseRevSha1Columns to true for enwiki'
13:30 mutante: applied a patch to etherpad that allows admins to delete pads
12:53 mutante: restarting/fixing etherpad issue
11:08 mark: Sending European bits traffic back to esams
02:30 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Wed Apr 18 02:30:50 UTC 2012
02:21 logmsgbot_: LocalisationUpdate completed (1.19) at Wed Apr 18 02:21:49 UTC 2012
02:13 logmsgbot_: catrope synchronized php-1.20wmf1/README 'Dummy sync to capture which hosts time out on sync-file'
00:52 K4-713: updated production civi to r1631
00:41 Ryan_Lane: adding interface for per-project sudo on OpenStackManager

April 17

23:36 K4-713: updated production civi to r1628
23:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Fixes for cswiktionary changes per Danny B'
22:49 RoanKattouw: That was bug 34885 of course
22:43 logmsgbot_: catrope synchronized php-1.19/extensions/WikiEditor/ 'Deploy fix for bug 348885'
22:38 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Deploy fix for bug 348885'
22:05 K4-713: updated prod civi to r1625
21:51 logmsgbot_: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero needed for carrier testing'
21:42 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'use $wmgUseMathJax'
21:41 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'use $wmgUseMathJax'
21:38 K4-713: queue consumer re-enabled
21:35 K4-713: updated prod civi to r1623
21:32 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php
21:29 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/templates/ApplicationTemplate.php 'ec7c5cc'
21:28 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114947'
21:24 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgUseMathJax on mediawikiwiki'
20:33 logmsgbot_: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.flagging.php
20:26 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/VisualEditor/ 'Deploy VisualEditor beta warning'
19:52 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bump mobile resource version'
19:52 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/
19:51 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/
19:50 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobile frontend resource version'
19:01 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php
18:55 logmsgbot_: reedy synchronized php-1.19/includes/api/ApiQueryBlocks.php 'r114941'
18:53 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
18:47 binasher: returning sq68
18:36 binasher: pulling sq68 from pybal for a bit
18:29 RoanKattouw: Did a graceful restart of all job runners using dsh about 15 mins ago
18:29 RoanKattouw: Restarted morebots
07:44 apergos: morebots test
07:44 apergos: restarted varnish service manually a bit a go on sq67 and sq70, the cron job didn't seem to have gone off. restarted morebots too while I was at it
03:37 Jeff_Green: dist-upgrade arsenic
03:29 LeslieCarr: restarting varnish on arsenic again
03:12 maplebed: started a script to delete old objects on ms-be1 for swift truncated object cleaning
02:53 Jeff_Green: dist-upgrade on strontium
02:43 LeslieCarr: restarted varnish on arsenic
02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue Apr 17 02:26:40 UTC 2012
02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Tue Apr 17 02:17:24 UTC 2012
01:44 LeslieCarr: restarting varnish on niobium
00:52 LeslieCarr: reloading amslvs4
00:27 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo 'deployed 552ff0f482f3e65e9795fe304dd810e9ae1b03fb'

April 16

23:31 logmsgbot_: catrope synchronizing Wikimedia installation... : Now with a touch of the specific WikiEditor.i18n.php file
23:11 logmsgbot_: catrope synchronizing Wikimedia installation... : Hopefully fixing the WikiEditor messages this time, now with MessagesEn.php touch
23:07 logmsgbot_: catrope synchronizing Wikimedia installation... : Hopefully fixing the WikiEditor messages this time
22:58 logmsgbot_: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend to r114934
22:49 logmsgbot_: catrope synchronizing Wikimedia installation... : Need to run scap for this WikiEditor change, contains i18n changes
22:39 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Deploy WikiEditor revert'
20:53 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Actually deploy the recent WikiEditor fixes'
18:58 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Commons Wiki to 1.20wmf1
18:47 logmsgbot_: reedy synchronized php-1.20wmf1/resources/mediawiki/mediawiki.js
18:46 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/WikiEditor
18:37 mutante: manually added iptables nat rules on nfs2
18:13 notpeter: upgrade of udp2log on nfs1/2 complete. should be operating normally now.
17:41 mutante: LDAP on nfs2 warnings - opendj was _just_ started there when puppet was fixed with an unrelated issue
17:38 mutante: restarting opendj on nfs2 because it refused connections
17:08 logmsgbot_: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ 'zero and mobile changes'
16:07 notpeter: upgrading and restarting udp2log on nfs1/2
15:04 mutante: puppet fresh on nfs[12] after removing nonexistent misc::mediawiki-logger class
14:46 mark: Shutdown db24 for memory testing by Chris
13:27 mark: Sending European bits traffic back to pmtpa
12:24 mark: Sending European bits traffic back to esams
12:06 mark: Testing sess_leak_fix2 patch with a snapshot varnish build on cp3001
11:56 Reedy: Ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -- "cd /usr/local/apache/common && sudo -u mwdeploy ln -s php php-1.18" to create symlink for php-1.18
11:51 Reedy: Killing php-1.18 again
11:48 mutante: sq34 - System halted! Error: Internal Storage Slot, powered down, -> RT
11:45 logmsgbot_: reedy synchronized php-1.18/ 'Symlink php-1.18 back to php (our current main running version) as lots of requests on bits are for 1.18 resources'
11:44 mutante: sq34 was broken and died when connecting to mgmt, powercycling
11:37 mutante: nfs1 - Could not find class misc::mediawiki-logger for nfs1
10:57 Krinkle: bits.wikimedia.org back up, mark fixed it.
10:33 Krinkle: bits.wikimedia.org serving Error 503 Service Unavailable on all load.php requests for mediawiki.org and nl.wikipedia.org, maybe more
09:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnableJavaScriptTest to true for test2wiki'
02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Mon Apr 16 02:26:58 UTC 2012
02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Mon Apr 16 02:17:57 UTC 2012

April 15

17:35 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api '/me whistles'
17:20 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api
02:25 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sun Apr 15 02:25:58 UTC 2012
02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Sun Apr 15 02:17:19 UTC 2012

April 14

18:14 mark: Shifting european bits traffic back from esams to pmtpa, session leak is still there
17:08 mark: Shifting european bits traffic back from pmtpa to esams
15:31 mark: Reverted varnish to 3.0.2-2wm4 on cp3001; the race condition patch did not fix the problem
14:56 mark: Sending European bits traffic to pmtpa for testing
13:52 mark: Backported varnish bug #897 patch to varnish 3.0.2, testing a snapshot build on cp3001
11:37 mark: Raised session_max to 300000 (runtime) on cp3001/cp3002
05:58 K4-713: re-enabled the queue consumer on aluminium
02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 14 02:26:55 UTC 2012
02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Sat Apr 14 02:17:34 UTC 2012
02:16 K4-713: updated prod civi to r1616
01:36 K4-713: turned off queue consumption on prod civicrm
01:36 K4-713: updated production civicrm to r1614

April 13

20:53 mark: Rebooting cp3002
20:37 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114889'
17:54 Jeff_Green: created new repo operations/debs/wikimedia-search-qa to stay within package naming conventions
17:31 notpeter: upgrading udplog on locke to 1.8-2 and restarting, etc
17:27 Jeff_Green: created new operations/debs/search-qa repo for packaging search qa scripts
17:17 notpeter: restarting udp2log on emery
12:53 notpeter: restopping puppet on locke/emery
12:09 mark: Deploying varnish 3.0.2-2wm4 and enabling persistent storage on all even numbered eqiad upload varnish hosts
11:46 mark: Imported varnish 3.0.2-2wm4 into the Wikimedia APT repository
02:48 logmsgbot: tstarling synchronizing Wikimedia installation... :
02:39 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Fri Apr 13 02:39:01 UTC 2012
02:20 logmsgbot: LocalisationUpdate completed (1.19) at Fri Apr 13 02:20:35 UTC 2012
01:40 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fix robots file'
01:18 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ 'zero and mobile changes'
01:06 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix html formatter'
00:56 logmsgbot: tstarling synchronizing Wikimedia installation... :
00:08 Ryan_Lane: rebooting ssl1004

April 12

23:39 logmsgbot: tstarling synchronizing Wikimedia installation... :
23:08 logmsgbot: preilly synchronizing Wikimedia installation... : zero rated mobile access changes and mobile frontend updates
21:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34923 - namespace required for PORTAL'
19:46 notpeter: stopping puppet on locke and emery
18:41 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 updates
18:22 Reedy: Ran namespaceDupes against bewiki
18:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34024 - Install ArticleFeedback on es.wikinews'
18:15 logmsgbot: reedy synchronized php-1.19/resources/startup.js 'touch'
18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34024 - Install ArticleFeedback on es.wikinews'
18:11 Reedy: Created AFT tables on eswikinews
17:54 RoanKattouw: Running schema updates for ArticleFeedbackv5 on enwiki
17:46 RoanKattouw: Deploying ArticleFeedbackv5 updates to testwiki and rebuilding localization cache
16:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Allow bnwiki crats to grant/remove import'
16:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35258 - Allow bureaucrats to remove sysop rights on fr.wikipedia'
16:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix imports for wm2012'
16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35917 - allow transwiki imports on wikimania2012'
16:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35666 - Renaming Namespace Wikisource:Author in gu.wikisource'
16:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35694 - Add enotif on page changes in watchlist (guwiki and source)'
16:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35818 - Change of Armenian Wikipedia namespace'
16:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35905 - Change namespaces configuration - pl.wikipedia'
16:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35261 - Add block permissions in rollback on Lusophone Wikipedia'
16:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35823 - Wikijunior and cookbook namespaces for the Vietnamese Wikibooks'
16:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35659 - Set logo for sl.wikiversity'
16:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35853 - Set a non-empty default value for wmgArticleFeedbackBlacklistCategories on WMF wikis'
15:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35878 - Enable e-mail notifications for watchlist (EnotifWatchlist) on tawiki'
15:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35852 - Add a category to $wgArticleFeedbackBlacklistCategories for Portuguese Wikipedia to remove AFT from disambiguation pages'
15:10 mutante: gallium - after files have been deleted/moved, puppet back to normal operation (and new clone directory in Apache)
13:23 mutante: killed puppets on gallium
12:33 mark: repooled ssl1002
12:27 mutante: powercycling frozen ssl1002
12:22 mark: Manually depooled down ssl1002 in pybal
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Thu Apr 12 02:24:29 UTC 2012
02:15 logmsgbot: LocalisationUpdate completed (1.19) at Thu Apr 12 02:15:54 UTC 2012

April 11

22:37 maplebed: deployed more log filters to emery: gerrit/r4758
21:35 LeslieCarr: restarted nrpe on db10
21:33 LeslieCarr: db1004 puppet is fubar
21:33 LeslieCarr: restarted puppet on db30
21:33 LeslieCarr: restarted puppet on mw1110
19:41 notpeter: reimaging bellin and blondel
19:28 logmsgbot: reedy synchronized live-1.5/ 'fix resources symlinks'
19:23 logmsgbot: reedy synchronized live-1.5/ 'fix resources symlinks'
16:54 notpeter: enabling notifications for eqiad lucene vips
16:31 mark: Sending Canadian upload traffic to the eqiad varnish upload cluster
15:59 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 4 to eqiad. for realz this time!'
15:45 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 1 and prefix pool to eqiad. for realz this time!'
15:31 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 2 to eqiad. for realz this time!'
15:15 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 3 to eqiad. for realz this time!'
14:40 notpeter: restarting indexer on searchidx2
13:48 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AbuseFilter/special/SpecialAbuseLog.php
13:35 mutante: applied patch-RT-2804.diff to bugzilla per RT:2804 re: XMLRPC content-type verification
12:07 mutante: moved another list: museum-l -> glam (http://lists.wikimedia.org/pipermail/glam/2012-April/000000.html)
11:58 mark: Setup cp1036 with the persistent storage backend
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed Apr 11 02:26:28 UTC 2012
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Wed Apr 11 02:17:55 UTC 2012
00:11 LeslieCarr: nagios down

April 10

23:50 RoanKattouw: Removed srv187-189 from /etc/dsh/group/job-runners , their jobrunner class has been commented out in puppet since October
23:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'bug 35869 - Add strategywiki as an import source on testwiki'
22:53 RoanKattouw: Trying a graceful restart of the job runner on mw1 by sending SIGHUP to the jobs-loop.sh process
22:53 logmsgbot: catrope synchronized php-1.19/extensions/WikimediaMaintenance/jobs-loop.sh 'r114834'
22:24 logmsgbot: reedy synchronized php-1.20wmf1/extensions/CentralAuth/ 'g4102'
22:23 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AntiSpoof/ 'g4103'
21:20 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling mobile URL template for mediawiki.org (using "mediawikiwiki" this time)'
21:18 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling mobile URL template for mediawiki.org'
21:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf1
21:04 logmsgbot: reedy synchronized php-1.20wmf1/extensions/MobileFrontend/javascripts 'minified JS'
20:55 logmsgbot: reedy synchronized docroot/ 'Fix symlinks'
20:45 logmsgbot: reedy synchronized docroot/
20:35 logmsgbot: reedy synchronized docroot/
20:31 logmsgbot: reedy synchronized live-1.5/
20:24 logmsgbot: reedy synchronized php-1.20wmf1/ 'Resyncing for apaches with no space'
20:23 logmsgbot: reedy synchronized live-1.5 'Fix symlinks'
20:18 Reedy: Deleting php-1.18 from all apaches due to lack of space
20:14 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PrefSwitch/ 'PrefSwitch is needed by SimpleSurvey'
19:35 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding localisation cache for test2/1.20wmf1
19:24 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf1.php 'Sync ExtensionMessages'
19:23 logmsgbot: reedy synchronized php-1.20wmf1/extensions/ 'Would you like some extensions to go with that, sir?'
19:21 LeslieCarr: restarting gmond on db1004 after removing it's 5gig log
19:07 logmsgbot: reedy synchronized php-1.20wmf1/LocalSettings.php 'Push LocalSettings out'
19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf1
19:00 logmsgbot: reedy synchronized php-1.20wmf1/ 'Pushing files for 1.20wmf1'
18:03 logmsgbot: aaron synchronized wmf-config/swift.php 'Catch e bogus empty file names from listings'
14:17 robh: search in eqiad is being reinstalled, no need to be alarmed (thats a pun!)
14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgLanguageConverterCacheType for git deployment later'
11:50 mutante: pxe boot / reinstall cp1029 - cp1036
11:24 mark: Imported varnish 3.0.2-2wm3 into the Wikimedia APT repository
09:30 apergos: restarted slaving on es1003, it will be a bit before it catches up. patience, young nagios
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Tue Apr 10 02:16:58 UTC 2012
01:33 Tim: on sodium: enabling mod_auth on lists.wikimedia.org by running puppet

April 9

23:14 mutante: migrated foundation-l to wikimedia-l (users/passwords/archive urls/settings stay, old mail address & siteinfo redirect)
22:32 logmsgbot: asher synchronized wmf-config/db.php 'returning db12 as enwiki recentchange/watchlist db'
21:39 LeslieCarr: restarted mysql on es1004 and cleared out its disk space
17:49 LeslieCarr: moving es monitoring to nrpe and variables, may cause false pages if i did it wrong :)
17:36 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 35426 - WebFonts on mr.wikisource.org'
14:54 RobH: i killed eqiad search nodes, woooo
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Apr 9 02:17:22 UTC 2012

April 8

08:45 Nemo_bis: Servers have been very slow, almost unresponsive, and network had a drop of ~0.3 Gb/s, at ~8.35-40.
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sun Apr 8 02:16:58 UTC 2012

April 7

17:55 logmsgbot: reedy synchronized wmf-config/codereview.php 'Remove deferred paths'
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sat Apr 7 02:16:54 UTC 2012

April 6

22:23 LeslieCarr: deploying new squid config to all squids
22:14 LeslieCarr: added neon into tiertwo of squid allowed hosts
22:13 LeslieCarr: deploying new squid config to amssq35
21:55 LeslieCarr: restarted puppet on spence
21:35 LeslieCarr: moved jenkins_1.458_all.deb to /srv/wikimedia/incoming/ on brewster
21:32 LeslieCarr: restarted squid on brewster
18:27 Ryan_Lane: updating OpenStackManager to r114758 on virt0
17:33 mark: Sending Japanese upload traffic to varnish in eqiad
17:15 mark: Power cycled down host lvs5
16:43 mutante: changed master and started slave on es1004
15:55 mutante: used gerrit create-project to create operations/debs/wikistats.git
14:13 mutante: manganese (gerrit) now sends SSL CA certificate on https, (curl -vvv says verify ok), should resolve RT:2777 and BZ:35709
11:51 mutante: es1004 - rsync was finished, deleted all binlogs from old host, mysqld_safe& , but did not "change master.." and "start slave" (see mail)
11:39 notpeter: restarting lsearchd on search3... again...
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Apr 6 02:17:37 UTC 2012
01:21 Ryan_Lane: updating OpenStackManager to r114757 on virt0
00:18 Ryan_Lane: updating OpenStackManager to r114754 on virt0

April 5

23:49 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Change guwikisource logo to point to the unscaled file instead'
21:46 notpeter: halting db15 for it to await decom
21:39 binasher: started enwiki.revision sha1 migration on db12
21:32 notpeter: restarting lsearchd on search18
21:22 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12, moving enwiki watchlist,recentchange,etc to db53'
21:19 logmsgbot: asher synchronized wmf-config/db.php 'returning db53'
21:17 logmsgbot: py synchronized wmf-config/lucene.php 'pushing all search traffic back to pmtpa'
18:34 Ryan_Lane: updating OpenStackManager to r114746 on virt0
18:19 Ryan_Lane: updating OpenStackManager to r114744 on virt0
16:49 RobH: brewster puppet running again, cisco installs wont work again until i finish puppetizing the files later today
15:41 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool4 to eqiad. this is the smaller wikis shard'
15:40 notpeter: pointing search pool4 to eqiad (this is the "smaller languages" shard)
15:14 Rob_H: puppet daemon being halted on brewster, i need to make local test changes to dhcp
14:52 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search prefix pool live in eqiad'
14:51 notpeter: pushing search prefix pool live in eqiad
14:51 mutante: gallium - disabled incompatible GitTool plugin on jenkins and restarted it
14:34 mutante: importing jenkins_1.458_all.deb to wikipedia apt repo and upgrading it on gallium
14:08 apergos: started rsync in screen session as root on es1003 copying snapshot from es1001 to /a/
14:04 andrewbogott: created labs account for cneubauer
14:02 logmsgbot: py synchronized wmf-config/lucene.php 'pointing enwiki search and enwiki.prefix at eqiad'
14:00 notpeter: pointing enwiki and enwiki.prefix at eqiad search cluster
13:48 mutante: gallium - upgraded all pear packages
13:45 mutante: gallium - upgraded phpunit and php_codesniffer via pear (have been installed via pear before, distro outdated)
13:43 mutante: gallium - upgrading pear
13:33 mutante: installing package upgrades on gallium. apache,apt,postgres,php5-*,ruby,...various libs
13:24 logmsgbot: py synchronized wmf-config/lucene.php 'pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad'
13:21 notpeter: pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad
12:27 notpeter: search1 and search4 seem to be dead. restarting lsearchd
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Thu Apr 5 02:16:52 UTC 2012
00:33 Ryan_Lane: updating OpenStackManager to r114730 on virt0
00:24 Ryan_Lane: updating OpenStackManager to r114729 on virt0
00:19 Ryan_Lane: updating OpenStackManager to r114728 on virt0
00:12 Ryan_Lane: updating OpenStackManager to r114726 on virt0
00:00 Ryan_Lane: updating OpenStackManager to r114724 on virt0

April 4

22:16 maplebed: deployed (3rd time's the charm!) udp-filter changes to emery for diederik
22:14 logmsgbot: py synchronized wmf-config/lucene.php 'pointing all search back to pmtpa'
22:13 notpeter: flipping all search back to pmtpa (until tomorrow...)
22:00 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback 'r114717'
21:24 cmjohnson1: replacing power cable to psu1 (bottom) es1
21:22 cmjohnson1: replacing power cable to psu1 (top) es1
21:14 logmsgbot: py synchronized wmf-config/lucene.php 'pointing de, fr, and ja search at lvs pool in eqiad for live testing'
21:12 notpeter: moving de, fr, and ja search to eqiad
21:04 cmjohnson1: replacing power cable on labstore2 array psu2 (right side)
21:00 cmjohnson1: replacing power cable on labstore1 array psu1 (left side)
20:57 cmjohnson1: removing power from bottom power supply labstore 2
20:54 cmjohnson1: removing power from top power supply on labstore2
19:44 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wmgArticleFeedbackv5AbuseFiltering on enwiki'
19:40 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable wmgArticleFeedbackv5AbuseFiltering on enwiki'
19:14 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114716'
19:12 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wmgArticleFeedbackv5AbuseFiltering on enwiki'
19:04 RobH: dns update for zhen mgmt
18:54 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying AFTv5 update
18:52 logmsgbot: py synchronized wmf-config/lucene.php 'pointing ru, nl, pl, pt, zh, and sv search at lvs pool in eqiad for live testing'
18:51 notpeter: moving ru, nl, pl, pt, zh, and sv search to eqiad
18:27 mutante: nuked /a contents on es1004, started rsync from es1001
18:16 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add code for wmgArticleFeedbackv5AbuseFiltering'
18:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgArticleFeedbackv5AbuseFiltering, enabled on testwiki only'
17:55 RoanKattouw: Running AFTv5 schema changes on enwiki
17:47 RobH: i didnt crash the site, weeee
17:46 RobH: gracefully restarting apaches
17:46 RobH: pushing out redirects change to apaches for wikipedia.org/com.il redirect to he.wikipedia.org
17:41 binasher: started enwiki.revision sha1 migration on db53
17:38 logmsgbot: asher synchronized wmf-config/db.php 'returning db52, pulling db53'
17:32 RobH: update done, all nameservers still online
17:31 RobH: dns update for wikipedia.org/com.il being resolved
17:08 RoanKattouw: Applying AFTv5 schema change on testwik
15:30 logmsgbot: py synchronized wmf-config/lucene.php 'pointing eswiki search at lvs pool in eqiad for live testing'
15:28 notpeter: pointing eswiki search at eqiad
12:51 mutante: db1007 - add mysql startup via 'update-rc.d mysql defaults'
12:42 apergos: started mysqld on db1007 via /etc/init.d/mysql (this doesn't seem to point to a special fb build, and can't seem to find one on this host, what's up with that?)
12:31 apergos: rebooted bd1007, it was dead in the water (also no helpful messages on console, bah)
11:16 mutante: enabled Renameuser extension on wikitech, renamed tchay per RT request, disabled extension again (it was installed but disabled)
02:19 logmsgbot: LocalisationUpdate completed (1.19) at Wed Apr 4 02:19:03 UTC 2012
01:50 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileOp.php 'deployed r114697'
01:39 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'

April 3

23:17 LeslieCarr: updating bgp policies on cr1.sdtpa
22:44 LeslieCarr: reinstalling neon
22:04 maplebed: rolled back changes to emery in udp-filter due to the new binary crashing.
21:50 maplebed: ran /etc/init.d/udp2log reload on emery to enact the puppetted changes
21:41 maplebed: deploying new udp-filter and teahouse filters to emery for diederik
20:13 notpeter: restarting lsearchd on search7. was taosted
18:37 logmsgbot: root synchronized wmf-config/mc.php
18:37 RobH: syncing new mc.php, forgot to check for all three of the servers i took down, opps.
18:28 RobH: shutting down mw28, mw49, & mw58 for rack relocation due to power overload in d2-pmtpa, relocation to d1-sdtpa per rt 2692
17:59 K4-713: Synchronized payments cluster to r114642
17:52 logmsgbot: reedy synchronized php-1.19/extensions/MobileFrontend/
17:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix commas in guwikisource namespaces'
17:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix commas in guwikisource namespaces'
16:38 RobH: bringing down srv237 for phase balancing
16:37 RobH: srv230 back in rotation
16:26 RobH: shutting down srv230 for power phase move per rt 2759
16:10 RobH: updating brewster to use new dhcp files for cisco, no more local hackin.
15:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35482 - Add Patroller & Autopatroller groups on ml.wikisource'
15:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35482 - Add Patroller & Autopatroller groups on ml.wikisource'
15:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35545 - Grant the abusefilter-log-detail right to patrollers on Commons'
15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35545 - Grant the abusefilter-log-detail right to patrollers on Commons'
15:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35624 - Subject namespace for the Vietnamese Wikibooks'
15:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35603 - Enable Transwiki import on KN:WP'
15:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35581 - Closure of nz.wikimedia.org'
15:15 logmsgbot: reedy synchronized closed.dblist 'Bug 35581 - Closure of nz.wikimedia.org'
13:35 Tim: manually reloaded rsyslogd on all apaches
06:16 Tim: deploying limited/split apache syslog (https://gerrit.wikimedia.org/r/#change,4149)
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Tue Apr 3 02:16:32 UTC 2012
00:37 logmsgbot: aaron synchronized php-1.19/includes/Block.php 'deployed r114672'

April 2

23:54 Tim: restarting all apaches with apache-restart-all-hard
23:51 logmsgbot: tstarling synchronized php-1.19/extensions/ConfirmEdit/FancyCaptcha.class.php
23:37 logmsgbot: tstarling synchronizing Wikimedia installation... :
23:36 maplebed: cleared the varnish cache for preilly
23:34 Tim: on all apaches: running logrotate -f and deleting the resulting backup syslog files, to free up disk space
23:32 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114673'
23:21 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version number'
23:05 logmsgbot: awjrichards synchronizing Wikimedia installation... : Deploying MobileFrontend changes at r114671 per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#2_April.2C_2012
21:43 maplebed: reverted changes to emery's logging due to a broken package in the deploy.
21:30 LeslieCarr: turned down ms7's secondary ethernet port to prevent the flapping (stupid sun boxes)
19:51 maplebed: deploying new udp-filter to emery rt-2501 gerrit/r4120
19:51 notpeter: running authdns-update on dobson
18:30 RobH: brewster puppet daemon stopped, doing local hacks
18:17 RobH: removed old bin files on db1004 and prolly borked it by removing the wrong files
17:54 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php '35436 - Enable Narayam at Hindi Wikipedia'
17:47 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add disable images option for default on zero domain'
17:45 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 35328 - Enable WebFonts for fr.wikisource.org'
17:40 logmsgbot: nikerabbit synchronized php-1.19/languages/Names.php 'I18ndeploy r114656'
17:15 preilly: carrier testing push for DIGI
17:15 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
16:46 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Mon Apr 2 02:16:47 UTC 2012

April 1

02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Apr 1 02:17:22 UTC 2012

March 31

10:22 mutante: srv222,225 were also upgraded but stopping there for now in favor of reinstalls
09:58 mutante: nuked /usr/shared/doc on a couple srv's, hey at least 700MB or something, and yes we really should reinstall with a decent partitioning scheme as M ark said
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 31 02:18:10 UTC 2012

March 30

19:37 hashar: configured jenkins on gallium to use smtp.pmtpa.wmnet as outgoing SMTP server
19:28 RobH: puppet daemon restarted on brewster
18:13 RobH: killing puppet daemon on brewster, i need to hack at local configuration for cisco server stuff
12:56 mutante: db1047 - added system startup for /etc/init.d/mysql
12:47 mutante: powercycling db1047
12:28 mutante: deleted old kernel sources on upgraded srvs for that little extra space during peaks, suggesting to nuke /usr/share/doc if there should be more disk space warnings
10:41 mutante: same for srv223
09:18 mutante: srv224,srv219,srv220, upgrade apache, dist-upgrading w/ kernel, disabling ureadahead, rebooting one by one
08:06 mutante: storage3 - gmond unable to find the metric information for any mysql_* .."module has not been loaded", starting mysql, running puppet ...
07:57 mutante: powercycling storage3
07:03 Tim: running bug 35578 cleanup script in screen on fenari
06:41 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
06:40 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
06:39 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
06:15 Tim: killed vi on fenari owned by awjrichards, locking CommonSettings.php for two days
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 30 02:17:56 UTC 2012
01:13 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove more crap'
01:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove some dupe code'
01:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove wmgUsabilityPrefSwitch'
00:59 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove wmgUsabilityPrefSwitch'
00:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove unused wmgUseUsabilityInitiativeAlpha'

March 29

23:49 logmsgbot: aaron synchronized php-1.19/includes/revisiondelete/RevisionDeleteUser.php 'deployed r114619'
21:20 LeslieCarr: rebooting db47
20:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Swap wgUseCommaCount to wgArticleCountMethod'
20:07 notpeter: restarting lsearchd on search2 to del the logfile to end all logfiles
20:05 RoanKattouw: Stopping and starting Gerrit on manganese to apply Chad's change of the -1 text in the DB
20:02 notpeter: restarting lsearchd on search7 to del the logfile to end all logfiles
18:11 logmsgbot: catrope synchronized php-1.19/extensions/ClickTracking/ClickTracking.hooks.php
17:59 RobH: search1021 coming back up, done with tests
17:53 RobH: search1021 coming down for ssd fit test
17:07 notpeter: disabling notifications for search lvs nagios checks for 24 hours to test fix
15:42 notpeter: finished clearning up all pmtpa search hosts. hey look! they all have lots of space now!
15:15 notpeter: restarting lsearchd on search3
15:02 RobH: brewster puppet re-enabled
15:02 RobH: virt1001 pxe boots via dhcp and fails tftp download, i have to hold off on further troubleshooting until i have a network admin
14:47 RobH: did virt1001 wrong, reupdating dns
14:39 RobH: all nameservers still online after udpate
14:37 RobH: updating dns for virt1001 testing
14:29 RobH: stopping puppet runs on brewster so my hacking at the dhcpd.conf file won't get overwritten until I have it working right
14:01 Jeff_Green: restarted varnish on on cp3002 because it was thrashing futiley
13:45 notpeter: rebooting (mostly) down cp3001
13:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add participation namespace to metawiki per request'
13:11 notpeter: trimming logs and such on search1-20
09:59 mutante: srv221, disabling ureadahead, installing package upgrades and new kernel, rebooting
09:40 mutante: kill and start lsearchd on search7
09:36 mutante: restarted defunct lsearchd on search6
09:10 mutante: gallium - added demon,hashar,reedy to group jenkins as it's a problem using puppet when users and groups already exist
06:25 mutante: powercycling sq40
06:21 mutante: installed more package upgrades on sodium
05:58 mutante: installed security upgrades on brewster, cadmium, capella (apache,mysql,ruby,apt..)
05:49 mutante: db42 - mysql did not autostart after boot, added using update-rc.d
05:42 mutante: db42 - reboot worked despite the grub warning about unreliable blocklists
05:37 mutante: rebooting db42 to finish upgrades
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 29 02:17:53 UTC 2012

March 28

23:27 Tim: running apt-get upgrade on mw22,mw66,srv193,srv250,srv253,srv236
23:25 Tim: cleaned up stuck apt-get process on srv236
23:22 Tim: cleaned up stuck apt-get processes on mw22,mw66,srv193,srv250,srv253
21:44 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobile frontend resrouce version'
21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/application.min.js 'r114576'
21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/banner.min.js 'r114576'
21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114576'
21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114576'
21:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/application.js 'r114576'
21:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/banner.js 'r114576'
20:43 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 28 20:43:20 UTC 2012
20:29 notpeter: restarted search1020. nothing conspicuous in logs
19:56 RoanKattouw: Running a patched version of l10nupdate that rebuilds the localization cache
18:49 logmsgbot: catrope synchronizing Wikimedia installation... : Bugfixes for ArticleFeedbackv5, ArticleFeedback and ClickTracking
16:47 cmjohnson1: msw1-d1-pmtpa replacement complete
16:34 cmjohnson1: replacing msw-d1-pmtpa per rt2639
15:36 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech'
15:34 Reedy: srv221 is full
15:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech'
14:39 RobH: restarted morebots in screen on wikitech, no longer as catrope, as roan has root on that box
14:36 RobH: got virt1001 to pxe, but dhcp doesnt know how to handle, need subnet details.
14:34 notpeter: lucene hosed on search9 and search15. restarting, then will look after cause
13:14 Jeff_Green: restarting puppet/puppetmaster on stafford to experiment with report settings
02:10 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 28 02:10:34 UTC 2012

March 27

23:12 logmsgbot: tstarling synchronized php-1.19/cache/trusted-xff.cdb
20:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33789 - Enable botadmin usergroup on ml.wikipedia'
19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix lezwiki namespace'
19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove ruwiki arbcom talk from namespaceprotection'
19:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
18:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
18:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35290 - Create Slovenian Wikiversity'
18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35290 - Create Slovenian Wikiversity'
17:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
16:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
16:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
16:48 logmsgbot: reedy ran sync-common-all
16:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'prep work for new wikis'
16:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34527 - Create a Arbcom namespace on Russian Wikipedia'
16:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34527 - Create a Arbcom namespace on Russian Wikipedia'
15:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35161 - Incubator configuration updates'
15:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32825 - Favicon for siwiki'
14:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35516 - Add Skin: namespace to MW.org'
08:15 apergos: test you silly morebot

07:59:56 hashar: archived old server admin logs since the old page was too long for my connection to download :-/
06:59:02 apergos: !log powercycled emery, it was unresponsive via the mgmt console and not pingable
02:17:52 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 27 02:17:52 UTC 2012
00:56:51 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114507'
00:55:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/LeaveFeedbackTemplate.php 'r114508'
00:42:50 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bmping resource version for MobileFrontend'
00:41:58 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r114509'
00:37:30 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version #'
00:36:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/MobileFrontendTemplate.php 'r114507'
00:36:09 logmsgbot: awjrichards[00:36:36] synchronized php/extensions/MobileFrontend/templates/LeaveFeedbackTemplate.php 'r114508'
00:35:50 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r114508'
00:08:55 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114506'

March 26

23:18:17 logmsgbot: awjrichards synchronizing Wikimedia installation... : Syncing MobileFrontend to r114504 changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#26_March.2C_2012
22:44:53 RobH: !log also rolling firmware to ps1-d[1|2|3]-pmtpa
22:28:10 RobH: !log pushing firmware updates to servertechs in sequence: ps1-[a2|a3|a4|a5|b2|b3|b4|b5|c1|c2|c3|d1|d2|d3]-sdtpa, disregard any errors from rebooting alerts
19:55:09 notpeter: !log stopping puppet on search6 and search15 for 24 hours to test new log rotation script
19:19:35 RobH: !log cp1019 memory replaced per rt 2651
19:07:14 apergos: rebooting ms1001 (new kernel)
17:53:34 RobH: cp1019 coming down for memory replacement per rt 2651
17:51:39 RobH: fluorine disk upgrade done, os install pending, details on rt 2350
17:43:48 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r114492'
17:36:51 RobH: fluorine coming down for new disks
17:14 notpeter: backingup plwiki.nspart1 index on search7, deleting working copy, and restarting lsearchd. (note: this will probably cause some downtime on some languages while the proc restarts...)
15:18 RobH: db59 has errors, but as it was a fusion io testbed server, it is more than likely tweaked for such, it is not in any rotation
14:54 RobH: db59 shutting down for io card removal per rt 2589
13:37 mutante: while on it, installing a whole bunch of package updates on db42
13:25 mutante: db42 was out of disk , caused by ~5G citations.csv in /tmp, gzipped the file
09:59 mutante: ..and on ms-be-3. running puppet on db59
09:43 mutante: another corrupted .yaml file on ssl2
09:33 mutante: brewster - delete puppet lock file, restart lighttpd, puppet ...
09:05 mutante: brewster was out of disk - deleted lighttpd access.log.1, gzipped access.log
08:24 mutante: on several mw* boxes puppet did not run because .yaml files on the puppetmaster became corrupted. need to delete the $hostname files in /var/lib/puppet/yaml/node on stafford and re-run. puppet bug similar to http://projects.puppetlabs.com/issues/7836
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 26 02:18:03 UTC 2012

March 25

22:26 RobH: row b servertech firmware in eqiad all updated, should clear alarms as they come back online
22:18 RobH: firmware updates on servertechs in row b eqiad, disregard alarms
20:14 RobH: to fellow ops, you can disregard those observium errors, as I caused them
20:13 RobH: firmware updated on all power strips in row a eqiad.
16:22 RobH: ps1-a1-sdtpa firmware update complete
16:15 RobH: updating firmware on ps1-a1-sdtpa
16:14 RobH: ps1-b1-sdtpa firmware updated successfully
16:14 RobH: ps1-a1-eqiad firmware updated successfully
16:09 RobH: updating firmware on ps1-s1-eqiad and ps1-b1-sdtpa
16:07 RobH: updated firmware successfully on ps1-a8-eqiad, if it has observium alarms now then there are bigger issues.
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 25 02:17:21 UTC 2012
00:59 LeslieCarr: admin down asw-a-eqiad xe-1/1/2 and cr2-eqiad xe-5/0/0 due to framing errors causing packet loss and lacp sporadic timeouts. source of the issue

March 24

19:46 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
19:46 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
17:35 mark: Migration from br1-knams to cr2-knams completed.
17:09 mark: Migrated second knams-esams dark fiber link from br1-knams to cr2-knams
16:36 mark: Corrected MTU setting on cr2-knams's AMS-IX interface
16:20 Reedy: Some european users reporting oruting issues
16:01 mark: Cleared OSPF session between csw1-esams and csw2-esams which magically made some internal routes reappear
15:40 mark: Brought up AMS-IX ipv4 BGP sessions
15:30 mark: Brought up AMS-IX ipv6 BGP sessions
15:25 mark: Moved AMS-IX connection to cr2-knams:xe-1/1/0
15:22 mark: Shutdown all AMS-IX BGP sessions
15:06 mark: Disabled BFD on OSPF3 between cr2-knams and csw1-esams
14:49 mark: Moved AS6908 and AS1257 PIs to cr2-knams
14:18 mark: Brought up AS13030 and AS1299 BGP sessions on cr2-knams
13:57 mark: Shutdown AS1299 BGP session on br1-knams
13:14 mark: Established full iBGP mesh with added router cr2-knams. cr2-knams now has full Internet connectivity.
12:48 mark: Moved fiber from br1-knams:e1/2 to cr2-knams:xe-0/0/0
12:44 mark: Disabled br1-knams:e1/2 (DF leg 1 to esams)
12:43 mark: Rack mounted and powered up cr2-knams
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 24 02:17:02 UTC 2012

March 23

23:49 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ 'r114466'
23:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia'
23:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia'
23:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce'
23:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce'
23:07 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce'
23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce empty arrays'
22:24 RobH: scs-a1-eqiad back online
21:58 RobH: scs-a8-eqiad coming down for re-grounding
19:51 RobH: all power strips in eqiad are now properly grounded
18:12 maplebed: removed ms1 and most of ms2 from the production swift rings. no effect expected.
18:04 logmsgbot: asher synchronized wmf-config/db.php 'returning db32, pulling db52 for migration'
16:44 RobH: cp1019 in middle of firmware update, please dont touch
16:44 RobH: cp1017 memory error seems ot have cleared post firmware update, will keep an eye on it for the rest of the day
16:09 RobH: raid rebuilding on magnesium, however swift stuff is kind of black box mystery right now to me, need Ben to review magnesium later for that
15:53 RobH: magnesium coming back online
15:44 RobH: shutting down magnesium for disk swap
15:37 RobH: firmware updating on cp1017, no one touch it please
15:30 RobH: db1020 can go back into whatever rotation Asher wants it in
15:29 RobH: db20 memory error on raid controller resolved with firmware updarte
06:39 logmsgbot: tstarling synchronized php-1.19/includes/filerepo/file/LocalFile.php 'r114442'
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 23 02:18:35 UTC 2012
01:55 mutante: deleting puppet report files older than 60hours on stafford to free disk space

March 22

23:30 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
23:18 RobH: db1020 firmware still updating, will check on it later tonight. offline until then
22:19 notpeter: all 3 dns servers are responding to digs after reload
22:10 notpeter: pushing a new zone file to add 2 more search-related vips for eqiad
20:52 notpeter: stopping puppet on brewster temporarily
20:25 notpeter: rebuilding search1015 and 1016 for disk shuffles
20:01 RobH: magnesium goign down and up again, troubleshooting the disks
19:47 apergos: rebooting ms1002, had stuck rsyncs, and kswapds at 100% cpu, weirdness like "ls /export/upload/wikipedia/am/0/00" hanging.
18:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
15:45 RobH: search 1015 and search1016 back up with added disks
15:08 RobH: shutting down search1015 & search1016 for hdd additions
14:45 RobH: db1020 still offline, requires firmware update on raid controller per rt 2621, will perform later today
14:33 logmsgbot: reedy synchronizing Wikimedia installation... :
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 22 02:17:47 UTC 2012
01:14 K4-713: Re-enabled the donations queue consumer in Jenkins
00:28 binasher: started enwiki.revision alter on db32
00:26 binasher: disabled lvm snapshots and puppet on db32 for revision sha1 alter
00:24 logmsgbot: asher synchronized wmf-config/db.php 'pullin db32 for revision alter'

March 21

22:27 ^demon|away: wmf-deployed extensions now r/o in SVN
21:52 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php
21:27 Ryan_Lane: bringing up all instances on virt3
21:08 cmjohnson1: swapped 2 DIMMS in virt3 (b2 and b5)
21:01 Ryan_Lane: shutting down virt3 to replace dimms
20:47 ^demon: /trunk/phase3 is now r/o in SVN
20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable prefswitch'
20:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgArticleFeedbackv5OversightEmails on enwiki'
18:59 maplebed: rebooted ms-be3 after it crashed.
18:51 binasher: brought db24 back up after hang, and reslaving, but leaving out of db.php. just replicating until a replacement s2 snapshot host is built
18:51 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 update
18:46 logmsgbot: asher synchronized wmf-config/db.php 'returning db36'
18:35 logmsgbot: asher synchronized wmf-config/db.php 'pulling db24, failing hw'
18:03 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php
18:01 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily disable ShortUrl on testwiki because we think it might conflict with ArticleFeedbackv5'
17:59 K4-713: updated and synchronized payments cluster to r114382
17:47 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
12:25 notpeter: disabling notifications for search-pool1
08:58 mutante: rebooting ms-be4
08:37 mutante: stopped/started lsearchd on search9
08:05 mutante: ms-be4 down but cant powercycle it yet..Unable to establish LAN session / ipmitool /ipmi_mgmt
07:58 mutante: restarted lsearchd on search3 and 9
05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/CoreParserFunctions.php
05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/Parser.php
05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/StripState.php
05:22 logmsgbot: tstarling synchronized php-1.19/tests/parser/parserTests.txt
03:51 mutante: added "lez" to langlist and running authdns-update, for lez.wikipedia per RT-2665
03:29 mutante: magnesium - shutting down, has existing RT-2669 to replace disk
03:18 mutante: magnesium - "..drive on port B of the Srial ATA controller is operating outsde of normal specifications.. Strike F1 key to continue"..
03:16 mutante: powercycling magnesium - down and just "init: tty4 main" on mgmt, frozen
03:10 mutante: running puppet on aluminium
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 21 02:18:10 UTC 2012
01:06 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114342'
00:25 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo'
00:03 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo'

March 20

23:19 Ryan_Lane: fixing the zero redirect
22:46 logmsgbot: reedy synchronized wikipedia.dblist 'test'
22:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/api/ApiQueryExtracts.php 'r114319'
22:09 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping resrouce version # for MobileFrontend'
21:54 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#20_March.2C_2012
21:46 binasher: stopped eqiad bits servers from udplogging to emery, packet loss is back to zero
20:59 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
20:17 binasher: killed enwiki.revision sha1 migrator (upgrade-1.19wmf1-2.php). after db36 completes, will run the rest by hand
19:52 Ryan_Lane: pushing change for zero.wikipedia.org to redirect to the english message
19:41 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page'
19:16 cmjohnson1: pulling disk 5 on virt1 for reseating
18:34 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page'
18:02 pgehres: flipped Template:CC-status on wmfwiki since credit cards are still disabled on payments.wikimedia.org
17:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35193 - Enable sub page feature in Telugu Wikisource'
17:49 notpeter: restarting lsearchd on search10
17:30 logmsgbot: aaron synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed r114285'
17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that then'
17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Test something for sewikimedia'
16:42 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Bug 35355 - Reset of permissions for Hindi Wikipedia'
16:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove hiwiki botadmin from whGRoupsRemoveFromSelf'
15:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
15:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
15:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
15:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35161 - Incubator configuration updates'
15:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31209 - Enable the WikiLove extension for incubator'
14:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove more group dupes'
14:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35355 - Reset of permissions for Hindi Wikipedia (hiwiki)'
14:14 logmsgbot: reedy synchronizing Wikimedia installation... : sscapping for r114268
14:08 logmsgbot: reedy synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'r114268'
09:12 mutante: new URL pointing to Wikipedia Education Program - http://education.wikimedia.org
08:59 mutante: several srv's said they were unable to contact NTP server
08:57 mutante: apache-graceful-all to deploy changed redirects.conf
08:53 logmsgbot: tfinc synchronized wmf-deployment/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'Fixes file pages showing data charge warnings'
07:42 mutante: running authdns-update after adding education.wm for redirect RT:2634
06:21 logmsgbot: tstarling synchronized php-1.19/includes/User.php
05:35 logmsgbot: asher synchronized wmf-config/db.php 'pulling db36 durring db migration'
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 20 02:17:55 UTC 2012
00:25 logmsgbot: awjrichards synchronizing Wikimedia installation... : Reverting MobileFrontend to r113973
00:15 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114221'
00:07 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling zero rated mobile access everywhere'
00:05 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumpging version number for MobileFrontend resources'

March 19

23:54 logmsgbot: awjrichards synchronizing Wikimedia installation... : Redoing accidentally aborted scap, Pushing changes to MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#19_March.2C_2012
23:51 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing changes to MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#19_March.2C_2012
23:35 AaronSchulz: fixed a few files, on commons and other wikis, with empty oi_archive_name values even though the file was on NFS
23:20 Ryan_Lane: restarting all nginx servers
23:20 Ryan_Lane: added a new proxy to the ssl configuration to temporarily proxy access to wikimania videos being transcoded
21:38 binasher: creating "ops" db and related grants on prod db clusters 2-7 to prep rollout of ishmael / pt-digest beyond s1
21:17 binasher: started enwiki.revision sha1 alter on production side
20:57 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Removing debugging code from MobileFormatter'
20:54 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
20:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
20:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
20:31 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Adding debugging code to MobileFormatter'
20:07 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r114176'
19:41 Ryan_Lane: bringing virt3 instances back up
19:33 binasher: deploying new frontend squid conf to add support for mf_useformat cookie [rt 2645]
19:18 K4-713: CiviCRM 4.1.1 update script finished executing on prod.
19:12 Ryan_Lane: shutting down virt3 for memory reseating
19:09 K4-713: Started the CiviCRM 4.1.1 update script on prod.
19:08 mark: Rebuilding RAID arrays on brewster
18:58 K4-713: Put production civicrm / drupal instance in offline mode for upgrade
18:54 K4-713: Disabled all production CiviCRM Jenkins jobs, for CiviCRM upgrade.
18:54 cmjohnson1: brewster HDD replacement complete
18:42 mark: Shutting down brewster for HDD replacement
18:26 Jeff_Green: killed kill-slow-queries on db1008 for the duration of the civicrm upgrade
18:19 logmsgbot: nikerabbit synchronized php-1.19/includes/Linker.php 'i18ndeploy r114160'
18:19 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/resources/ext.webfonts.fontlist.js 'i18ndeploy r114160'
18:14 mark: Running smartctl -t long /dev/sdb on brewster
12:58 logmsgbot: hashar synchronized php-1.19/includes/SiteStats.php 'Reenable SiteStatsInit::articles() for bug 35169. SiteStatsInit::doAllAndCommit() still disabled since it breaks the site'
10:28 logmsgbot: tstarling synchronized wmf-config/PoolCounterSettings.php 'increased max queue from 50 to 100 on reports that the limit was reached on the enwiki main page in normal operation'
09:11 mutante: nomcom and langcom wikis look kind of broken , redirecting to pages on incubator with "Error: This page is unprefixed! "
08:49 mutante: making (almost) all private wikis https-only per RT-2565, vi remnant.conf,sync,graceful...
07:30 mutante: running sync-apache after making a change to remnant.conf to make grants.wm https-only
05:09 Ryan_Lane: bringing up most instances on virt3, doing so by project priority
04:42 Ryan_Lane: bringing up all instances on virt4, waiting 30 seconds between instances
04:25 Ryan_Lane: bringing up all instances on virt2, waiting 30 seconds between instances
04:09 Ryan_Lane: bringing up all instances on virt1, waiting 30 seconds between instances
04:00 Ryan_Lane: attempting to bring some instances up
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 19 02:17:17 UTC 2012
01:15 mutante: killed, updated, restarted wikibugs bot per request in RT:2656, should have fixed bugzilla:18831

March 18

23:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35308 - Install mw:Extension:DynamicPageList (Wikimedia) on Portuguese Wikipedia (ptwiki)'
19:20 Ryan_Lane: stopping all labs instances, manually recovering gluster volume
15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35295 - Missing a in abusefilter-hide-log permission for oversighters'
10:49 Ryan_Lane: rebooting virt4 thanks to defunct libvirt process
03:43 Ryan_Lane: bringing all labs instances up
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 18 02:18:51 UTC 2012
01:09 Ryan_Lane: rebooting all of the virt hosts, gluster is having major issues
00:43 Ryan_Lane: rebooting virt2
00:40 Ryan_Lane: restarting glusterfs on virt2
00:11 Ryan_Lane: rebooting virt3 libirt is non-responsive
00:00 Ryan_Lane: bringing up instances that were downed on virt3

March 17

23:50 Ryan_Lane: virt3 crashed, powercycling it
23:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove old comments'
23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove old comments'
23:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat'
23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat'
23:02 logmsgbot: catrope synchronizing Wikimedia installation... : Have to scap for that AFTv5 change to propagate i18n change
22:52 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ 'r114087'
21:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35289 - Add wikisource logo to mobile wikisource gateway'
02:21 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 17 02:21:03 UTC 2012
01:23 AaronSchulz: FindFilesMissingDBRows.php done, list under aaron/output/missingFileDBRows
00:11 AaronSchulz: Running FindFilesMissingDBRows.php on all wikis

March 16

21:21 binasher: running enwiki.revision sha1 schema migrations on eqiad side
20:12 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild moodbar messages
20:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable moodbar on enwiki'
19:53 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ 'r114030'
19:15 Reedy: Ran namespaceDupes on stewardwiki
17:11 RobH: hdd in search1017/1018 replaced per rt 2583
16:54 RobH: search1017 and search1018 coming down for hdd swap
16:53 RobH: cp1017 back in service pool
16:43 RobH: cp1019 back in full service
16:22 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r114021'
16:22 RobH: cp1017 memory error, coming down for troubleshooting.
16:18 RobH: cp1019 memory error cleared after reseating, notes on rt 2651
16:09 mark: Migrated all varnish3 packages to newer varnish packages from git
16:08 RobH: cp1019 coming down for memory error troubleshooting
15:58 RobH: cp1040 repaired per rt 2611
15:48 RobH: cp1040 down for memory replacement
15:09 logmsgbot: reedy synchronized stylize.php 'Test for hume'
15:04 logmsgbot: root synchronized ufg.sql 'test sync to see if hume is fixed'
14:55 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php
14:04 apergos: restarted swift-container-auditor on ms-be3, it had died for some reason
08:07 mutante: i reverted that (star cert for wikitech), no worries i "shred"ded the files
07:51 mutante: replaced self-signed cert on wikitech with the star cert
04:19 mutante: on stafford, deleting spence's puppet report files to free some disk space (they are like the largest report files of all)
03:09 mutante: stafford - - /var/lib/puppet/reports is getting quite large (18G), and we got the first disk space warning, do we want to keep those?
02:45 mutante: killing nrpe on several hosts where it was running as the wrong user again (somehow through the use of dsh)
02:21 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 16 02:21:35 UTC 2012
01:12 mutante: stopping nagios-wm temp. while changing nrpe config (will watch it manually until it's back)
00:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe'
00:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe'

March 15

23:17 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113974'
23:12 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/DisableTemplate.php 'r113973, fixes bug 35249'
23:10 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/modules/ext.articleFeedbackv5/ext.articleFeedbackv5.js 'r113972'
22:59 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
22:59 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Bump AFTv4 event logging percentage from 25% to 100%'
22:57 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js
22:48 mutante: purging Lucene monitoring on indexer from db9, remove duplicate service definitions manually anyways (still tons left), run purge script, reload Nagios..
22:24 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
22:23 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Raise AFTv4 event logging percentage from 5% to 25%'
22:21 mutante: getting rid of Swift HTTP checks on non production machines manually (come on spence _purge_ ;P)
22:07 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
22:04 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Raise AFTv4 event logging percentage from 1% to 5%'
21:44 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
21:28 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113961'
21:25 pgehres: K4-713 synchronized payments cluster to r113956
21:25 pgehres: disabled credit cards on donate.wikimedia.org
21:21 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'fix fatal'
21:20 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Bump AFTv4 event logging percentage from 0.27% to 1%'
21:19 Ryan_Lane: rebalancing instances gluster volume
21:18 RoanKattouw: That was r113959
21:18 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js
21:11 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r113958'
21:09 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ 'r113957'
20:46 mark: bits.pmtpa cluster back online
20:44 RobH: dns update for silver and zhen servers
20:37 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php
19:54 RobH: sq67-sq70 have been reinstalled, but not signed in puppet, not sure if they are ready for that or if there are other items mark needs to change first
19:11 RobH: working on sq67-sq70 reinstalls, disregard alerts
19:00 RobH: db1022 resetup and redeployed per rt 2537 and assigned back to asher
18:51 logmsgbot: reedy synchronizing Wikimedia installation... : Running scap to deal with message changes earlier
18:19 RobH: db1022 coming down for reinstall and resetup of raid per rt 2537
17:55 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/ 'r113940'
17:54 logmsgbot: reedy synchronized php-1.19/extensions/CheckUser/ 'r113940'
17:53 logmsgbot: reedy synchronized php-1.19/extensions/wikihiero/modules/ext.wikihiero.css 'r113940'
17:52 logmsgbot: reedy synchronized php-1.19/extensions/NewUserMessage/NewUserMessage.class.php 'r113940'
17:41 logmsgbot: reedy synchronized php-1.19/includes/RecentChange.php 'r113938'
17:38 logmsgbot: reedy synchronized php-1.19/resources/mediawiki/mediawiki.util.js 'r113936'
17:37 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialUndelete.php 'r113936'
17:32 logmsgbot: reedy synchronized php-1.19/languages/messages/ 'r113935'
17:31 logmsgbot: reedy synchronized php-1.19/resources/ 'r113935'
17:31 logmsgbot: reedy synchronized php-1.19/includes/ 'r113935'
17:16 logmsgbot: reedy synchronized php-1.19/includes/SkinTemplate.php 'r113932'
16:13 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php 'r113929'
15:15 mark: Created git repo operations/debs/varnish in gerrit
14:06 apergos: disabled moodbar temporarily on en wikii, see bug 35245
14:02 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disable of feedback dashboard (right config var this time?)'
13:51 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disable of feedback dashboard'
13:11 apergos: on screen as root on dataset1001, copying to gluster volume; if this causes problems feel free to shoot it. ( cp -a 20120211 /mnt/glusterpublicdata/public/enwiki/ )
09:08 mutante: ran puppet on mw1020
08:12 mutante: installing apache,apt,cron,mysql-client upgrades on spence
07:51 mutante: messed with /var/lib/dpkg/status on hume to fix broken packages/remove "marked for purging" on libmysql-php5 without removing a ton of other packages, rather hackish but seems fine anyways, like not broken anymore on simulated dist-upgrade etc
07:01 mutante: uprading apache and apt on hume
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 15 02:17:35 UTC 2012
01:26 Ryan_Lane: labsconsole was missing libapache2-mod-php5. puppet must have tried to upgrade a package unsuccessfully
01:22 mutante: planet back up (installed libapache2-mod-php5 which installed apache2-mpm-prefork and removed apache2-mpm-worker)
01:19 mutante: planet down - apache on singer, syntax error in site config "Invalid command 'php_admin_flag'"
01:03 mutante: fixing nrpe "unable to read output" raid check on srv197,207,243,,244,253.. (nrpe running as wrong user)

March 14

23:16 maplebed: installed the swiftcleaner to run daily from iron. see root's crontab for more info.
20:41 binasher: disabled log_queries_not_using_indexes on all core dbs
20:33 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
20:30 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
19:29 maplebed: rebooting ms-be1 to enable hyperthreading (and make it the same as all the other ms-be hosts)
19:06 preilly: pushing x-images header for vary support
19:06 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
19:05 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'zero needs to add x-images to vary header'
18:58 maplebed: ms-be5 is back in rotatino
18:31 preilly: push zero change for carrier testing
18:31 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
16:19 RobH: updating dns for new domain wikimediacommons.pt (nameservers not yet pointed at us)
16:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'add vcs for extdist updates'
13:03 RobH: cp1029-cp1035 all installed and ready for varnish deployment, puppet has been run
08:24 mutante: running "apt-get -f install" on snapshot3 to fix dpkg, which installed mysql-client- and client-core-5.1
08:02 mutante: stop/start memcached on srv254,srv255,srv257
07:51 mutante: restarting mecached on marmontel
07:51 mutante: fixing owa[1-3] Swift HTTP commands manually
03:44 mutante: ekrem - user agent "AppleDictionaryService" requests cause temp. WAP outage ..it seems
03:38 mutante: free some disk space on spence - deleted user.log.1 on spence, compressing messages.1, apt-get clean,...
02:52 RobH: cp1032-cp1035 reinstall issue wiped mbr causing issues, will reinstall in my AM
02:49 RobH: revoked, cp1032 is some reason in grub error, and its too late at night for me to work on it, will troubleshoot tomorrow
02:48 RobH: realized i forgot to log hours ago that cp1029-cp1036 are installed with puppet run, ready for varnish deployment tomorrow
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 14 02:17:13 UTC 2012

March 13

23:51 mutante: upgrading bugzilla to 4.0.5
23:42 logmsgbot: reedy synchronized php-1.19/resources/jquery/jquery.textSelection.js 'r113786'
23:14 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
22:47 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113779'
22:44 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r113774'
22:44 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/common.css 'r113774'
22:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113771'
22:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/api/ApiQueryExcerpts.php 'r113774'
22:27 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Removing moile URL template for tewtwiki'
21:44 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
21:44 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
21:31 logmsgbot: asher synchronized wmf-config/db.php 'replacing db18 with new s7 slave db56'
21:19 binasher: started slaving db56 from db37
20:30 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
19:27 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
19:17 RobH: iron updated to use ipmi_mgmt script
19:08 preilly: pushing changes for zero to mswiki
19:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
19:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
19:05 binasher: streaming hotbackup of db1041 to db56 (new s7 slave replacing db18)
18:10 maplebed: failover successful, restarted pybal on lvs4, failback successful.
18:09 binasher: power cycling db1020, which also froze this morning
18:08 maplebed: stopping pybal on lvs4 - should fail over to lvs3
17:47 maplebed: pybal restarted on lvs3
17:47 binasher: power cycling db1040, crashed again
17:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 35183 - p include extensions/Renameuser/Renameuser.php instead of extensions/Renameuser/SpecialRenameuser.php'
17:12 mark: Sending all normally-pmtpa upload traffic to upload-lb.eqiad
17:05 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
16:59 preilly: add disable images support to mswiki under zero domain
16:59 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'add disable images option for mswiki on zero domain'
16:58 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add disable images option for mswiki on zero domain'
16:46 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mswiki remove from mywiki'
16:44 mark: Sending traffic from Japan, India, Mexico to upload-lb.eqiad
16:37 LeslieCarr: reinstalling neon
16:23 apergos: stole some free space from the phys volume on ms1002 to give us more time for the rsync to keep going til after the move to swift etc
15:28 mark: Sending traffic from the USA to upload-lb.eqiad
15:27 mark: Rebooting lvs1005 with upgraded kernel/packages
15:12 LeslieCarr: manually deleted cp1025 info from nagios config file - nagios restored for now
14:51 mark: Sending traffic from Canada to upload-lb.eqiad
14:32 mark: Sending traffic from Brazil to upload-lb.eqiad
13:58 mark: Sending traffic from Argentina to upload-lb.eqiad
12:58 mark: Seeding the eqiad upload caches from live upload requests
11:59 mark: Setup squid logging to oxygen, with oxygen relaying to multicast 233.58.59.1
11:02 mark: Rebooting lvs1002 with kernel updates
10:17 mark: Rebooting manutius with newer 2.6.36 kernel to attempt avoiding i/o kernel bug with torrus
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 13 02:18:03 UTC 2012

March 12

22:55 K4-713: synchronized payments cluster to r113679, and tweaked the anti-fraud rules
21:51 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r113671'
21:51 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113671'
21:44 Reedy: Running foreachwiki extensions/WikimediaMaintenance/cleanupBug31576.php in screen as me on hume
21:39 RobH: search1014 repaired per rt 2483
20:26 RobH: cp1040 coming down for hardware stuffs
18:19 Nikerabbit: Assuming scap has finished
17:48 logmsgbot: nikerabbit synchronizing Wikimedia installation... : Deploying updated Translate
17:46 notpeter: restarting indexer on searchidx2
17:24 logmsgbot: nikerabbit synchronized php-1.19/includes/Title.php 'r113635'
17:22 logmsgbot: nikerabbit synchronized php-1.19/languages/ 'r113635'
17:14 logmsgbot: nikerabbit synchronized php-1.19/extensions/Narayam/ 'Updating Narayam'
17:13 mark: PXE booting cp1025-cp1028
17:11 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'Updating WebFonts'
15:16 mark: Rebooted manutius, stuck in a similar state as streber always did
06:10 mutante: turning off debug mode in nagios-nrpe, again had to kill it , restart fails
05:53 mutante: dunno, copper was stuck (no mgmt output after reboot) but powercycling it and back
05:43 mutante: rebooting copper to make sure grub update didnt break it and asked for restart anyways
05:37 mutante: copper - installing (security) updates (apt,grub,openssl,ruby,libc6..)
04:19 mutante: wanted to restart nagios-nrpe-server on spence with debug=1 to investigate permission issue. arr! "Address already in use" "cant write to pidfile", killed the one started on Feb18, and reordered allowed_hosts, spence talks to itself again now :p
03:40 mutante: same (and nscd) on fenari
03:35 mutante: upgrading libc6 and related packages on spence
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 12 02:17:28 UTC 2012

March 11

08:14 apergos: restarted lighttp on dataset2
07:49 apergos: removed current htcp log file, restarted purger, it seems to be logging normallynow
07:35 apergos: current ls shows 17416851456 2012-03-11 07:34 HTCPpurger.log while current du -sh shows 175M for /var/log. Sparse file that gets rotated badly? lots of leading nulls (many gb worth), why?
07:33 apergos: on ms1004 the HTCPpurger.log file after rotation was 17 gb, filling the disk. Removed it.
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 11 02:17:35 UTC 2012

March 10

22:09 Reedy: Make that wikimania2012, not wikimediawiki
22:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable anon page creation for wikimediawiki'
19:28 binasher: set sync_binlog = 1 on all current masters and eqiad dbs
19:22 binasher: reslaved db1033
07:03 mutante: ran puppet on db1022, another one that works fine manually but somehow did not by itself
05:11 mutante: doing more (cp*, db*, msbe-* ,mw*) by hand / for loop
05:01 mutante: starting nagios-nrpe-server on all via dsh (fail to restart on config change issue)
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 10 02:16:57 UTC 2012
01:07 maplebed: started swiftcleaner on owa1 looking for (and purging) bad objects
01:06 maplebed: rebalanced the swift rings to finish decreasing traffic sent to ms1 and ms2
00:18 Ryan_Lane: powercycling ssl1003
00:18 Ryan_Lane: powercycling ssl1001

March 9

20:34 notpeter: stopping search indexer on searchidx2 for fresh rsync to searchidx1001
19:58 preilly: pushed change to remove description from landing page
19:57 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
18:59 Ryan_Lane: sending test.m.wikipedia.org to the same place as test.wikipedia.org via squid
18:58 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Fixing wgMobileUrlTemplate settings for domains that do not have .m. domains configured'
18:48 logmsgbot: reedy synchronized php-1.19/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.css 'r113497'
18:40 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Changing the way in which wgMobileUrlTemplate is configurable by InitialiseSettings.php'
18:39 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling wgMobileUrlTemplate for testwiki - hopefully for real this time'
18:34 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Making wgMobileUrlTemplate configurable by InitialiseSettings.php'
18:34 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling wgMobileUrlTemplate for testwiki'
17:40 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113489'
17:32 maplebed: set swift storage device weight on ms2 to 0 and pushed out rings
15:52 apergos: cleared up a little bit of space on root partition of snapshot2, but that's about it. I hope we never have 3 versions of mw in test at the same time, the tmp caches will kill us
15:52 mark: Turned off vcc_err_unref on all varnish servers, so varnish doesn't complain when ACLs/probes/backends are unused
15:44 Jeff_Green: hume apt upgrades, puppetd --test, switch to mysql 5.1.53-fb3753-wm1
06:38 Ryan_Lane: reloading autofs on all labs instances
06:13 Tim: running svn cleanup on extdist trunk
04:18 Tim: switched php and wmf-deployment symlinks over to php-1.19 instead of php-1.18
04:18 Tim: restarted morebots
00:57 pp-pdf2: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
00:57 pp-pdf3: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
00:57 pp-pdf1: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
00:38 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mywiki'
00:32 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.js 'fixes to code push'
00:32 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.min.js 'fixes to code push'
00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fixes to code push'
00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.js 'fixes to code push'
00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.min.js 'fixes to code push'
00:01 RobH: oxygen install done, booting successfully after multiple tests, now running puppet for initial config
00:01 K4-713: updated the paypal IPN listener on aluminium to r1450

March 8

23:57 logmsgbot: awjrichards synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113428'
23:56 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'per-wiki memory limit configuration, with extra memory for zh* for converter tables'
23:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'per-wiki memory limit configuration, with extra memory for zh* for converter tables'
23:42 mutante: rebooting ms-be5
23:37 logmsgbot: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments
23:24 binasher: streaming hotbacking of db1017 to db1033 - no snapshots of enwiki in eqiad til db1033 is back
23:19 Tim: started changing the php symlink to 1.19 instead of 1.18, but then changed my mind and changed it back.
23:16 logmsgbot: tstarling synchronizing Wikimedia installation... :
23:07 logmsgbot: tstarling synchronized php-1.19/extensions/ExtensionDistributor/svn-invoker.conf
23:01 logmsgbot: asher synchronized wmf-config/db.php 'returning db24 to service'
22:58 maplebed: powercycled ms-be3 - it crashed 2.5 hours ag.
22:52 logmsgbot: asher synchronized wmf-config/db.php 'pulling db18'
22:40 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend 'deployed r113413, r113414'
22:39 LeslieCarr: poked hole to allow labs machines to reach gluster machines in tampa
22:13 logmsgbot: catrope synchronized php-1.19/includes/MagicWord.php 'r113411'
22:13 logmsgbot: catrope synchronized php-1.19/includes/Cdb.php 'r113411'
22:13 logmsgbot: catrope synchronized php-1.19/includes/WebRequest.php 'r113411'
22:11 RobH: udpating dns for oxygen
22:03 RobH: oxygen coming down for reinstall
20:42 cmjohnson1: power to msw-c1-sdtpa restore
20:40 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
20:40 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.php 'changes for zero'
20:39 cmjohnson1: removing and relocating power to msw-c1-sdtpa
19:38 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
19:34 RoanKattouw: Running scap for ArticleFeedbackv5 updates
19:30 RoanKattouw: Running AFTv5 schema changes on enwiki
19:29 logmsgbot: catrope synchronized wmf-config/CommonSettings.php '$wgArticleFeedbackv5OversightEmails'
19:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '$wgArticleFeedbackv5OversightEmails'
19:26 RoanKattouw: Applying AFTv5 schema changes to en_labswikimedia
19:09 preilly: push zero rated changes
19:09 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
19:09 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
19:04 RoanKattouw: Clearing message blobs
18:53 RoanKattouw: Running rebuildLocalisationCache.php
18:49 binasher: power cycling cp1044
18:46 binasher: purging entire mobile varnish cache - the main mobile template included robots no-follow
18:43 preilly: needed to fix a google issue with robots
18:43 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ApplicationTemplate.php 'remove ROBOTS metatag'
18:40 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ApplicationTemplate.php 'remove ROBOTS metatag'
18:40 binasher: deploying new squid frontend.conf to fix epic fail - all googlebot traffic was being redirected to mobile. now just if it's mobilegooglebot.
18:29 RoanKattouw: Applying AFTv5 schema changes on testwiki
18:27 RoanKattouw: Pushing new AFTv5 code to testwiki, do not sync to the live site just yet
17:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'ptwikipedia to ptwiki'
17:14 cmjohnson1: shutting down db18 for memory testing
16:57 RobH: search1014 still down per rt2483
16:47 maplebed: took ms-be5 out of rotation in the swift cluster - it's crashed 3 times now.
16:36 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'r113368'
16:31 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Revert live hack because it works, will come in properly'
16:30 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Test for bug 27246'
16:16 RobH: search1008 repaired
15:52 RobH: mw1103 finally repaired and ready for os and such
14:48 pp-pdf1: installed python faulthandler 2.1
14:47 pp-pdf3: installed python faulthandler 2.1
14:47 pp-pdf2: installed python faulthandler 2.1
14:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35012 - Namespace aliases for wikipedia and wikipedia-talk namespaces on Sanskrit wiki'
09:17 mutante: running puppet on mw1010 - finished quickly without problems - uh, wonder why Nagios reported puppet freshness then
08:22 mutante: cp1019 - Hitting F1 to continue reboot ( "Alert! System fatal error during previous boot")
08:21 mutante: cp1019 went down, then rebooted by itself (i think) after showing "idrac-8W82BP1 Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted"
07:54 mutante: cadmium fixed by adding groups::wikidev
07:41 mutante: puppet on cadmium broken due to dependency Group[500] for User[catrope]
07:20 mutante: ms1004 ran out of disk - caused by 17G HTCPurger.log.1, trying to gzip it now
06:52 logmsgbot: tstarling synchronized multiversion/MWMultiVersion.php
06:51 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
03:04 Guest32353: powercycled ms-be5; it has been unresponsive for 2 hours.
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 8 02:18:02 UTC 2012
01:32 AaronSchulz: fixBug34995.php done
01:26 AaronSchulz: running fixBug34995 on all wikis
00:17 Ryan_Lane: adding zero cnames
00:16 Ryan_Lane: installing newer wikimedia-task-dns-auth on all dns servers
00:15 Ryan_Lane: added wikimedia-task-dns-auth_0.18 to the repo, to add support for zero

March 7

23:05 logmsgbot: aaron synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed r113319'
22:39 maplebed: set swift weight for ms1 to 0 initiating the process to move data off the host in preparation for decomissioning it.
21:17 Jeff_Green: running apt upgrades and puppetd --test on srv194, srv197, srv203, srv212, srv213, srv230, srv244, srv245, srv252, srv282 and manually restarting nrpe because they're reporting funky in nagios
20:20 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
20:17 Jeff_Green: yet another redirects.conf change, per RT#2498 redirect wikimedia.com-->wikimedia.org
20:05 binasher: reverted no-pagecache rsync on search nodes - without corresponding index warmup in lsearchd, it just pushes back the pain a bit and does more harm than good
20:04 binasher: deployed support for zero.wikipedia.org and carrier tagging to mobile varnish servers
19:38 logmsgbot: catrope synchronized php-1.19/includes/resourceloader/ 'r113278'
19:27 Jeff_Green: manual apt-upgrade, puppetd --refresh, and repeat on srv265 because it was running on outdated apache config
18:44 RobH: correction sq39
18:36 RobH: pulled sq39 from text pybal config, pulled sq46 from upload pybal config
18:36 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
18:36 logmsgbot: catrope synchronized php-1.19/extensions/CustomUserSignup/modules/AccountCreationUserBucket.js 'touch'
18:12 RobH: shutting down sq38 and sq46 per rt 2581 for testing
16:02 cmjohnson1: replacing hdd for disk 10 on db22
16:00 cmjohnson1: pulling disk 10 from db22
13:28 mark: Removed torrus from streber
13:00 pp-pdf2: updated mwlib to 0.13.6
13:00 pp-pdf3: updated mwlib to 0.13.6
13:00 pp-pdf1: updated mwlib to 0.13.6
11:29 logmsgbot: hashar synchronizing Wikimedia installation... : trigger a rebuild of l10n cache
04:53 mutante: added ms-be5 drives to swift cluster
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 7 02:18:01 UTC 2012
02:11 logmsgbot: catrope synchronized php-1.19/includes/api/ApiBase.php 'r113212'
01:58 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'bumped max file size to 4GiB'
00:27 maplebed: put ms-be4 into rotation as a new production swift backend storage node
00:21 maplebed: put ms-be3 into rotation as a new production swift backend storage node
00:05 maplebed: put ms-be2 into rotation as a new production swift backend storage node

March 6

23:54 logmsgbot: catrope synchronized php-1.19/extensions/CustomUserSignup/ 'Belated sync of r113056'
23:52 binasher: deploying new frontend squid config to include googlebot in mobile redirects
23:36 logmsgbot: reedy synchronized php-1.19/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php 'r113200 reverting r113198'
23:25 Tim: patched 5xx-filter.c live on locke and reloaded udp2log to stop the segfaults
23:20 logmsgbot: reedy synchronized php-1.19/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php 'r113198'
21:46 logmsgbot: catrope synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r113183'
21:41 notpeter: restarting puppet on brewster
21:03 Jeff_Green: pushing another change to redirects.conf and doing a graceful apache restart
20:32 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild message cache stuffs for r113129
20:31 Jeff_Green: disabled Global Connect nagios test (check_gcsip) on payments cluster because GC is down and nagios is spammy
20:25 notpeter: reimaging search1001-1020 with new partman recipe :/
20:22 notpeter: temp stopping puppet on brewster
20:21 logmsgbot: reedy synchronized php-1.19/resources/mediawiki.action/mediawiki.action.edit.js 'r113175'
20:20 logmsgbot: reedy synchronized php-1.19/maintenance/populateRevisionSha1.php 'r113175'
20:19 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialContributions.php 'r113175'
20:18 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialUserlogin.php 'r113176'
20:00 pp-pdf1: installed log-wikimedia-operations (which can be used for automated logging to #wikimedia-operations)
19:53 Ryan_Lane: restarting labs mysql to allow for more connections
19:26 Ryan_Lane: installing nova-api on virt0
19:09 Ryan_Lane: upping FLAGS.sql_max_pool_size for nova-api
18:47 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
18:46 Ryan_Lane: rebooting all instances
18:34 Ryan_Lane: restarting nova-network on virt2
18:19 Ryan_Lane: rebooting virt1
18:15 Ryan_Lane: rebooting virt2
18:11 Ryan_Lane: rebooting virt3
18:07 Ryan_Lane: rebooting virt4
17:57 Ryan_Lane: taking the opportunity to apply security updates to virt0-4
16:25 logmsgbot: catrope synchronized docroot/foundation/FrameResize.html 'Put Jobvite frame resize file in foundationwiki docroot per Erik'
11:40 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching sr* to 1.19
11:15 logmsgbot: hashar synchronized php-1.19/languages/messages/MessagesSa.php 'r1113039 for bug 34938 : title is sometime empty on Sanskrit wikis'
11:13 logmsgbot: tstarling synchronized php-1.19/includes/OutputPage.php 'r113128'
10:41 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching zh* from 1.18 to 1.19
08:36 mutante: on hooper: puppet broken due to dependency Package[libapache2-mod-php5] for Service[apache2]
03:33 mutante: rebooting bast1001 for kernel upgrade
03:32 mutante: upgrading apache2 packages, base-files, kernel, several libs on bast1001
03:27 mutante: installing a couple upgrades on fenari (apache2-utils, update-manager-core, cvs, ruby, libxml*, libopenssl-ruby*...)
02:37 logmsgbot: LocalisationUpdate completed (1.18) at Tue Mar 6 02:37:06 UTC 2012
02:36 logmsgbot: tstarling synchronizing Wikimedia installation... : updating to r113119
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 6 02:18:13 UTC 2012
01:27 Jeff_Green: manually updated packages and restarted apache on srv198, srv229, srv262, srv268, mw40 because their apache redirect configs failed to update after sync-apache and restart
01:07 Jeff_Green: another adjustment to redirects.conf and apache-graceful-all for RT#2488

March 5

22:24 Jeff_Green: modified redirects.conf per RT #2488
21:21 Reedy: Ran foreachwiki cleanupUploadStash.php
20:36 maplebed: enabled swift for 100% of thumbnails in production
18:18 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r113058'
18:11 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'WebFonts: bugwiki bug 34550; sawikisource bug 34159; amwiktionary amwikiquote bug 34700'
18:01 mark: Raised MTU between cr1-sdtpa - (csw1-sdtpa) - cr2-pmtpa to 9192
17:35 Jeff_Green: removed 3GB db30:/tmp/gmond.log and force-restarted gmond b/c the init script failed to restart it
17:16 Jeff_Green: adjusted LVS partitions on hume, moved /usr/local/apache to a new 5GB mount
15:18 mark: Fixed DNS resolving on the core routers by allowing DNS replies in the loopback filter
14:44 logmsgbot: reedy synchronized php-1.19/includes/Title.php 'r113036'
14:43 logmsgbot: reedy synchronized php-1.19/includes/AjaxResponse.php 'r113036'
14:35 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/ 'r113035'
14:34 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/ 'r113035'
13:50 mark: Set increased OSPF/OSPFv3 metric 30 on both directions of the link cr1-eqiad:xe-5/2/1 <--> cr1-sdtpa:xe-0/0/1, to combat higher than normal jitter and packet loss on the link
12:53 mark: Upgraded observium to latest version
09:41 mutante: restarting memcached on marmontel
09:40 mutante: restarting squid backend on knsq25
06:52 Ryan_Lane: all of the instances are accessing the file descriptors of files inside of the _base directory, and fuse has an issue with this. gluster can't recreate the base directory because of the processes holding open the old one.
06:50 Ryan_Lane: I've corrupted the _base directory on the instance's glusterfs share. I'm recovering the files from file descriptors using lsof. Not totally sure how I'm going to get the _base directory back, yet.
02:33 logmsgbot: LocalisationUpdate completed (1.18) at Mon Mar 5 02:33:04 UTC 2012
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 5 02:16:39 UTC 2012

March 4

21:48 logmsgbot: reedy synchronized wmf-config/ 'Bug 32726 - Set =true for Commons'
21:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix .'
21:41 logmsgbot: reedy synchronized wmf-config/ 'Bug 32726 - Set =true for Commons'
21:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34897 - Enable Special:Import on Catalan wikisource'
20:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34567 - New logo for Arabic Wiktionary'
20:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34715 - Please modify the import sources for the Spanish Wikiversity'
20:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34694 - Install the Quiz extension on de.wikibooks'
20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wmgMoodBarCutoffTime'
20:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Create wmgMoodBarCutoffTime'
20:14 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Variablise moodbarconfig infoUrl'
20:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Variablise moodbarconfig infoUrl'
20:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34618 - Install MoodBar on fr.wikisource'
20:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34766 - Logo of Sanskrit Wikisource'
19:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34867 - Switch Sango wiktionary logo'
19:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34931 - Add namespaces aliases on as.wikipedia.org'
19:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34690 - Changing the name in the title bar to Assamese'
02:35 logmsgbot: LocalisationUpdate completed (1.18) at Sun Mar 4 02:35:16 UTC 2012
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 4 02:17:34 UTC 2012

March 3

18:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34728 - Categories added to user pages by Babel in pt.wiktionary'
13:04 logmsgbot: aaron synchronized php-1.19/includes/Revision.php 'deployed r112949'
02:35 logmsgbot: LocalisationUpdate completed (1.18) at Sat Mar 3 02:35:08 UTC 2012
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 3 02:18:04 UTC 2012

March 2

21:41 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'disabled logging hack'
20:47 Jeff_Green: added redirect/301 from http://static.wikimedia.org --> http://dumps.wikimedia.org now that archival static html dumps are located there
19:53 mark: Decommissioned csw5-pmtpa from AS14907 service. rest in pieces ;)
19:10 mark: Did a hot cut to remove csw5-pmtpa out of the path of cr1-sdtpa -> csw1-sdtpa -> csw5-pmtpa -> cr2-pmtpa
17:46 cmjohnson1: powering down msw1-pmtpa for relcocation to d1-pmtpa
17:40 cmjohnson1: disconnecting management fiber from msw1-pmtpa
16:59 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'r112904'
16:55 RobH: ms-be4 boot order fixed, fixing ms-be5 & ms-be2
16:49 RobH: fixed boot order on ms-be3, fixing ms-be4
16:33 RobH: poking at bios on ms-be3
16:05 RobH: wikitech outage resolved
15:20 RobH: shutdown frdev offsite vm per email to engineering last week
15:18 RobH: backing up wikitech in hopes of upgrading some of its software
08:36 apergos: on ms1004, low on space, HTCPpurger.log.1 had about 16 gb of nulls before any real content, I tailed off the real stuff and tossed the original. The current log file has the same problem, why?
02:34 logmsgbot: LocalisationUpdate completed (1.18) at Fri Mar 2 02:34:34 UTC 2012
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 2 02:17:51 UTC 2012
01:36 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/lockmanager/LockManager.php 'deployed r112867'
00:41 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree 'deployed r112862'

March 1

23:33 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'log agent'
23:29 logmsgbot: reedy synchronizing Wikimedia installation... : Push message updates from r112848
23:22 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'logging fix'
23:22 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
23:20 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
23:17 logmsgbot: reedy synchronized php-1.19/includes/filerepo/backend/FSFileBackend.php 'r112850'
23:16 logmsgbot: reedy synchronized php-1.19/includes/Article.php 'r112850'
23:11 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
23:06 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ApiFeedbackDashboardResponse.php 'r112848'
23:05 logmsgbot: reedy synchronized php-1.19/extensions/Collection/Collection.body.php 'r112848'
22:12 logmsgbot: aaron synchronized php-1.19/includes/specials/SpecialContributions.php 'deployed r112844'
22:06 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend 'deployed r112841'
21:04 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'enabled FileBackend debug log'
19:57 cmjohnson1: replaced disk 3 labstore1 chassis
19:54 cmjohnson1: removing disk 3 from labstore1 chassis
19:47 Ryan_Lane: restarted memcached on virt0
19:15 logmsgbot: reedy synchronized php-1.19/cache/interwiki.cdb 'Updating interwiki cache'
17:39 Jeff_Green: Removed >5GB /tmp/gmond.log on db25, db32, db33, db37
17:36 logmsgbot: hashar synchronized php-1.19/includes/EditPage.php 'r112819 - Bug 34849 diff during editing an old version compares to the old version instead of the current one'
17:36 Jeff_Green: Removed >5GB /tmp/gmond.log on db13
17:35 Jeff_Green: Removed >5GB /tmp/gmond.log on db11
17:25 Jeff_Green: Removed 5.3GB /tmp/gmond.log on db1018
17:24 Jeff_Green: Removed 5.3GB /tmp/gmond.log on db1017
17:13 Jeff_Green: Removed 4.8GB /tmp/gmond.log on db1008. Tried to resist urge to make snarky comment about ganglia but failed.
14:54 RobH: strontium server rebooting to set HT to enabled
14:26 mark: Moving bits traffic back from pmtpa to eqiad
14:24 mark: Cleared dnsmasq cache on virt2
14:16 mark: csw5-pmtpa: Mar 1 14:01:42:A:Power Supply 2 , 2nd from left, bad
14:14 mark: mr1-pmtpa rebooted/lost power for some reason
14:07 mark: pmtpa/sdtpa management network went down
13:54 mark: Pooled new eqiad bits servers strontium and palladium
12:45 logmsgbot: hashar synchronized php-1.19/includes/specials/SpecialWatchlist.php 'r111882 for Bug 34835 - watchlist shows times in UTC'
10:53 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: reverting sr* wikis back to 1.18 per Siebrand's recommendation due to bug 34832
06:26 logmsgbot: tstarling synchronized php-1.19/extensions/SpamBlacklist/SpamBlacklist.php 'r112781'
05:46 maplebed: started swift deletion run on owa1, 2, and 3.
02:33 logmsgbot: LocalisationUpdate completed (1.18) at Thu Mar 1 02:33:53 UTC 2012
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 1 02:16:52 UTC 2012
02:15 Ryan_Lane: vlan tagged virt5's eth0 and eth1 ports on csw1-sdtpa
02:12 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'debug logging'
02:02 logmsgbot: reedy synchronized php-1.19/resources/mediawiki.action/mediawiki.action.history.diff.css 'r112750'
01:59 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: all zh wikis back to 1.18
01:50 logmsgbot: aaron synchronized php-1.19/extensions/WikiLove 'deployed r112758'
01:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last 265 wikipedias over to 1.19wmf1
01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: s7 to 1.19wmf1
01:23 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'r112754'
01:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: s2 to 1.19wmf1
00:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Meanwhile, on wikipedia.... Hello ruwiki!
00:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki to 1.19wmf1
00:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwiki to 1.19wmf1
00:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.19wmf1
00:05 logmsgbot: tstarling synchronized php-1.19/extensions/Collection/Collection.body.php 'r112745'

@@ Line 1: / Line 1: @@
 == May 1 ==
 * 16:03 notpeter: rebuilding db52 and db53 as s2 slaves
-* 15:47 logmsgbot_: asher synchronized wmf-config/db.php  's1: raising db59,60 weights, pulling db53/54 for reuse'
+* 15:47 logmsgbot_: asher synchronized wmf-config/db.php  's1: raising db59,60 weights, pulling db52/53 for reuse'
 * 09:23 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php  'hewiki account creation high throttle limits'
 * 04:04 Tim: on all apaches, running "chmod -R a+rX /usr/local/apache/common-local/" to clean up after killed rsyncs which left files unreadable

Server admin log/Archive 21

Revision as of 16:21, 1 May 2012

May 1

April 30

April 29

April 28

April 27

April 26

April 25

April 24

April 23

April 21

April 20

April 19

April 18

April 17

April 16

April 15

April 14

April 13

April 12

April 11

April 10

April 9

April 8

April 7

April 6

April 5

April 4

April 3

April 2

April 1

March 31

March 30

March 29

March 28

March 27

March 26

March 25

March 24

March 23

March 22

March 21

March 20

March 19

March 18

March 17

March 16

March 15

March 14

March 13

March 12

March 11

March 10

March 9

March 8

March 7

March 6

March 5

March 4

March 3

March 2

March 1

Archives

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Ops documentation

Wiki

Toolbox