Server admin log/Archive 21

Revision as of 10:45, 16 August 2012

August 16

03:57 mutante: nagios back up after adding missing monitor groups misc_pmtpa in appserver role (srv194)
02:56 K4-713: synchronized payments cluster to 37e31eddf5a4
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 16 02:28:33 UTC 2012
02:12 Ryan_Lane: pushed in large puppet change for ldap, openstack, gerrit and ldap pdns to make it more modular and to add support for eqiad region
02:10 Ryan_Lane: fixed arecord issues with labsconsole by adding an exception handling live hack for the jobs
01:21 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'remove wikimedia philippines event exemption'
00:40 mutante: installing package upgrades on singer
00:36 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed c51a9a288b6dd5c0023a77f324c04707b23501c6'
00:24 binasher: running "mwscript purgeParserCache.php --wiki=$db --age=1814400" instead
00:19 binasher: running mwscript purgeParserCache.php --wiki=enwiki --age=1209600
00:02 RoanKattouw: find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec /root/fixownership2 \{\} \; where fixownership2 = chown -h apache $1
00:02 RoanKattouw: Restarting the find-chown, this time with -h so symlinks are handled correctly (for some reason there are a bunch of broken symlinks with weird characters out there...)

August 15

23:52 binasher: powercycled srv259
23:44 mutante: brought nagios back up by removing "misc_pmtpa" host and servicegroups from srv194 manually
23:35 mutante: restarted pdns-recursor on ns0
23:11 mutante: authdns-update, moving zirconium to public IP and removing selenium
22:33 logmsgbot: reedy synchronized php-1.20wmf9/extensions/OpenSearchXml/
21:19 RoanKattouw: find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec chown apache \{\} \;
21:17 RoanKattouw: chown on ms7 has finished, running another find to find remaining bad files
20:57 K4-713: synchronized payments cluster to 79275e152b76f
20:26 ottomata: staring puppet on brewster. PARTMAN IS THE ENEMY. will continue the fight tomorrow.
20:22 mutante: http://en.wikipedia.org/w/ now redirects to Main_page on all languages. It was a 403 before.
20:20 mutante: apache-graceful-all
20:15 mutante: sync-apache pushing out new redirects from /w/ to Main_Page
20:04 ottomata: stopping puppet on brewster to experiment with partman on new analytics dells
19:29 mark: Removed OSPF/OSPFv3 metric 60 on cr1-sdtpa:xe-1/1/0 (eqiad link)
19:17 mark: Moved VRRP mastership from cr2-pmtpa to cr1-sdtpa by reassigning VRRP priorities, relieving the sdtpa-pmtpa link
18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remaining wikis over to 1.20wmf9
17:11 maplebed: ms-be1 dropped offline yesterday; powercycling.
16:11 mark: Setup SnapMirror replication from nas1-a:fr_archive to nas1001-a:fr_archive
16:10 mark: Created 4 TB volumes 'fr_archive' in aggregate prod1 on both nas1-a nad nas1001-a
16:10 mark: Increased aggregate 'prod1' by 4 drives on both nas1-a and nas1001-a
16:06 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 15 16:06:26 UTC 2012
15:40 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 15 15:40:21 UTC 2012
14:33 cmjohnson1: authdns-update for es9 and es10
13:46 RobH: authdns-update for wikipedia.co.za
12:15 mark: Started rsync of /home to nas1-a:/vol/home_pmtpa on nfs1
12:11 mark: Restricted NFS mounting of /vol/root on all NetApps
11:39 mark: Destroyed aggregate 'labs' on nas1-a
10:00 Tim: deploying bugfix version of wmerrors
09:51 Tim: moved old MW debug logs out of /home/wikipedia/logs to avoid confusion, live logs are now on fluorine
05:22 Tim: moving MediaWiki logs to fluorine
05:17 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:15 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
04:38 Tim: added rate limit exemption for 27.108.200.111
04:38 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
03:00 Tim: built a udplog package for precise and uploaded it
02:35 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'enabling Scribunto and CodeEditor on test2wiki'
02:28 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
02:15 logmsgbot: tstarling Finished syncing Wikimedia installation... :
02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
01:46 logmsgbot: tstarling Started syncing Wikimedia installation... :
01:00 logmsgbot: tstarling Started syncing Wikimedia installation... :
00:47 logmsgbot: tstarling Started syncing Wikimedia installation... :
00:39 Tim: running scap to for extension-list update

August 14

23:53 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Move VE Parsoid from cadmium to wtp1'
21:25 logmsgbot: kaldari synchronized php-1.20wmf9/extensions/PageTriage/tools/cleanupPageTriage.php 'adding PageTriage maintainence script'
21:05 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:45 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:38 mutante: killed duplicate nagios-wm
20:20 cmjohnson1: db44 swapping out disk1 rt 3424
20:08 ottomata: installed precise and puppetized stat1001
19:26 cmjohnson1: search23 shutting down to replace DIMM A2 rt 3423
19:00 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Deploying https://gerrit.wikimedia.org/r/19399'
18:52 RoanKattouw: Running /root/fixownership < /root/badownershipfiles in a screen on ms7
18:14 notpeter: stopping puppet on brewster for partman experimentation
17:34 logmsgbot: catrope synchronized php-1.20wmf9/.gitmodules
17:33 RoanKattouw: Yesterday's find on ms7 is done, found 1.8 million files with bad ownership. Will run a batch chown on them soon
17:32 logmsgbot: catrope synchronized php-1.20wmf9/extensions/UploadWizard 'Deploy 9d9189458ad8d966d952d295918847f00e304cb5'
09:17 logmsgbot: nikerabbit synchronized php-1.20wmf9/extensions/Translate/ 'I18ndeploy: Translate bugs'
07:46 logmsgbot: tstarling Finished syncing Wikimedia installation... :
07:07 logmsgbot: tstarling Started syncing Wikimedia installation... :
06:48 Tim: running scap to push out CodeEditor and Scribunto sources, not enabled yet
05:45 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/ 'Update E3Experiments to master'
03:34 Tim: restarting all apaches to get LuaSandbox extension
03:26 Tim: on mw8: removed broken package php5-memcached
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Aug 14 02:45:30 UTC 2012
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 14 02:28:13 UTC 2012
01:06 Tim: restarting apaches to get new wmerrors configuration

August 13

23:53 Tim: uploaded a new version of the php5-wmerrors package for lucid and precise
23:18 mutante: removed #wikimedia-tech from /usr/local/bin/start-nagios-bot
23:01 mutante: killing nagios-wm process which isn't supposed to be on #wikimedia-tech anymore
21:58 cmjohnson1: running authdns-update for labsdb1 & 2
21:19 RoanKattouw: ...piping result into ms7:/root/badownershipfiles
21:18 RoanKattouw: ms7# find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache
21:17 RoanKattouw: Searching for files with wrong ownership on ms7 using
21:13 RobH: reinstalling virt1004 to see if the raid issue re-appears
20:55 notpeter: updating root auth keys file on ms7 by hand.
20:36 logmsgbot: catrope synchronized php-1.20wmf9/resources/jquery/jquery.tablesorter.js 'Deploy 1fafaef3aa157f49cbbf4b9cffb03544bcf72a08'
20:04 preilly: updating zero for custom banner colors
20:03 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
20:03 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
19:42 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
19:41 preilly: update to zero for banner text link
19:41 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
19:36 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
19:32 RobH: powering off analytics1011-1014 to remove the add on nics (disrupting installer)
18:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf9
17:21 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Switched backend reads to swift for testwikis and mw.org'
15:18 paravoid: reprovisioning srv281 with a new fs layout
13:05 mark: Doing NDU of IOM3 shelf firmware on nas1001-a
12:59 mark: Doing NDU of IOM3 shelf firmware on nas1-a
12:47 mark: Setup NTP on all NetApps
12:24 mark: Restored SnapMirror relationship between nas1-a and nas1001-a, 70 days lagged, transfer initiated
12:04 paravoid: apt: include php-memcached 2.0.1-6~wmf+lucid2 @ lucid-wikimedia
11:58 paravoid: apt: include php5 5.3.10-1ubuntu3.2+wmf1 @ precise-wikimedia
11:08 paravoid: apt: remove wikidiff2/0.0.1wm1 (obsolete), copysrc php-wikidiff2 {lucid,precise}-wikimedia
11:08 paravoid: apt: remove php-wikidiff2/1.1.0-2 from lucid-wikimedia/universe (lucid-wikimedia/main has 1.1.2-1)
10:49 apergos: starting initial rsync from ms8 to ms10 (should take approximately forever)
06:26 Tim: uploaded czmq and zpubsub packages to brewster for Ori
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Aug 13 02:37:53 UTC 2012
02:20 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 13 02:20:27 UTC 2012

August 12

03:56 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Aug 12 03:56:33 UTC 2012
03:38 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 12 03:38:33 UTC 2012
03:19 Reedy: Running localisation update manually in screen session on fenari
02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

August 11

02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

August 10

21:54 Ryan_Lane: live hacked nova on virt2 with parts of nova git commit 4584e552a653904c36cf04cb295a7bf09d2def28
21:22 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
21:22 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
21:21 preilly: update to zero needed to support carrier testing
21:21 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero'
21:21 preilly: update to zero needed to support carrier testing
21:20 logmsgbot: preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero'
18:42 Jeff_Green: adding frack.eqiad.wmnet hosts to DNS (wmnet and 10..in-addr.arpa)
18:11 K4-713: Payments cluster updated to 133b2b6fb5
17:07 notpeter: temporarily stopping puppet on brewster
16:43 cmjohnson1: enabling pxe boot for 10GB Nic cards mc1-16
16:32 mark: Started OnTap upgrade to 8.1 on nas1-a and nas1-b
14:39 mark: OnTap upgrade on nas1001-a and -b complete
14:11 mark: Started OnTap upgrade to 8.1 on nas1001-b
13:55 mark: Started OnTap upgrade to 8.1 on nas1001-a
13:10 mark: Upgraded SP firmware to 1.3 on nas1001-a and nas1001-b
11:55 mark: nas1001-b disk fw upgrade completed
11:51 mark: nas1001-a disk firmware upgrade completed. Started disk firmware upgrade on nas1001-b
11:46 mark: Started disk firmware update on nas1001-a
02:39 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Aug 10 02:39:57 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 10 02:22:22 UTC 2012
01:07 Tim: on virt0: enabling mysql general log to debug "detached session" errors on nova
00:05 Tim: added myself as a labsconsole sysop, added myself to the packaging project, created a "php" instance for building php-related packages

August 9

22:05 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Added swift to multiwrite backend for all remaining wikis'
22:02 RobH: db1046 back online
21:40 RobH: shutting down db1046 to migrate its position in the rack
21:27 logmsgbot: olivneh synchronized php-1.20wmf8/extensions/E3Experiments/experiments/communityClicks.js
21:20 binasher: resumed enwiki replication in eqiad
21:13 binasher: stopping mysql on db1017 for a minute (and all enwiki eqiad replication with it)
20:52 logmsgbot: kaldari synchronized wmf-config/extension-list 'Removing deprecated CustomUserSignup extension'
20:52 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Removing deprecated CustomUserSignup extension and turning on Curation Toolbar for en.wiki'
20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Removing deprecated CustomUserSignup extension'
20:09 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:48 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'VE bugfix'
19:37 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php
19:22 binasher: rebooting db1009, db1010 for upgrades
19:20 binasher: rebooting db1011 for upgrade
19:17 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak links in VE'
18:01 cmjohnson1: re-imaging db68-77
16:54 cmjohnson1: loading OS on db65-67
16:06 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
16:05 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
14:53 RobH: cp1001 returned to service, resolving rt 3212
14:30 RobH: cp1001 squid services going offline for troubleshooting
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Aug 9 02:37:33 UTC 2012
02:20 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 9 02:19:57 UTC 2012

August 8

21:30 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:49 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:02 binasher: streaming a hotbackup of db1017 to db1050
19:54 ottomata: updating DNS entries for analytics1011-1027, they are all in row-c
18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia to 1.20wmf9
18:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix $wgVisualEditorParsoidPrefix'
18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf9
18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiquote over to 1.20wmf9
18:35 binasher: completed metawiki centralnotice schema migration for fundraising
18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikisource to 1.20wmf9
18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, fishbowl, closed and private wikis to 1.20wmf9
18:25 Reedy: special, fishbowl, closed and private wikis to 1.20wmf9
18:16 apergos: shutting down ms10, going to move it to public ip
17:21 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
17:20 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
16:58 mutante: freeing disk space on wikitech
16:53 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
16:52 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
16:46 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero'
16:45 logmsgbot: preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero'
16:40 cmjohnson1: swapping out main board on search32
15:21 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/.git 'Sync .git metadata so Special:Version is accurate'
14:43 logmsgbot: reedy synchronized wmf-config/
11:49 mark: Upgraded Observium to latest SVN
02:40 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 8 02:40:06 UTC 2012
02:21 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 8 02:21:44 UTC 2012
01:04 mutante: a couple package upgrades on bast1001

August 7

23:22 logmsgbot: catrope synchronized php-1.20wmf8/resources/mediawiki/mediawiki.js 'Deploying 6b6466f948d29520e1e3ab2592b940ce52415300'
23:19 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'Update VisualEditor'
22:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Curation Toolbar'
22:47 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings.php for Curation Toolbar'
22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disabled echo due to bug 39085'
22:12 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
22:05 binasher: streaming a hotbackup of db1001 to db1049 (new enwiki snapshot host in eqiad)
21:48 logmsgbot: reedy synchronized php-1.20wmf9/extensions/EducationProgram/
21:45 Reedy: Updated EducationProgram schema on both enwiki and test2wiki
20:15 logmsgbot: reedy synchronized php-1.20wmf9/includes/api/ApiQuery.php
16:23 LeslieCarr: moving bits traffic from pmtpa to eqiad
15:45 mark: Moving text squid traffic from pmtpa to eqiad
15:08 first reports of sites being very slow
15:07 nagios-wm: Apache HTTP on ... is CRITICAL: CRITICAL - Socket timeout after 10 seconds
14:35 Jeff_Green: starting otrs dump on db49
07:11 paravoid: powercycled srv272, swapdeath
03:09 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 7 03:09:28 UTC 2012
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Aug 7 02:37:34 UTC 2012
00:53 mutante: planet (en and fr) configs Updated to revision 115645.
00:32 Tim: on cp3002: set tcp_tw_recycle back to zero

August 6

23:49 mutante: wikitech back to old version after failed upgrade attempt
23:48 mutante: test
22:27 mutante: dist-upgrading wikitech instance
22:19 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/backend/SwiftFileBackend.php
22:14 binasher: rebooting db1028, upgrading
22:13 binasher: rebooting db1027, upgrade to precise
22:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/SwiftFileBackend.php 'deployed 0a5c71d4dd6405502b1d1d0a02b5d0927d519986'
21:59 binasher: rebooting db1026, upgrading to precise
21:54 mutante: saving space on wikitech linode - gzipping old .sql dumps and stuff
21:32 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'Updating VisualEditor'
21:18 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix Parsoid URL'
21:09 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor 'Updating VisualEditor'
20:23 logmsgbot: catrope synchronized php-1.20wmf9/cache/l10n/ 'Sync out l10n cache again now that it is properly rebuilt'
19:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf9
19:45 RoanKattouw: Rebuilt ExtensionMessages-1.20wmf9.php and then l10n cache for wmf9 to pick up addition of FundraiserLandingPage.i18n.magic.php , somehow wasn't picked up during initial scap
19:44 RobH: asw-c8-eqiad PEM1 power reseated, cleared alarm rt 3204
19:39 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.20wmf9.php 'Updated'
19:30 RobH: db1047 back online
19:18 RobH: db1047 shutting down
19:15 RobH: db1047 mysql and system shutdown per rt 3084 for bad memory swap
19:14 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
19:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf9
19:11 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
18:20 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
18:12 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
18:06 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed b55e9652fce3051d621356c5c87c47f44515a367'
18:02 logmsgbot: aaron synchronized wmf-config/filebackend.php
17:59 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed b55e9652fce3051d621356c5c87c47f44515a367'
17:11 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Adding Swift to local multiwrite backend for testwikis+mw.org'
16:57 cmjohnson1: powering down mc1 to reseat DIMM
16:50 logmsgbot: reedy synchronized .
16:33 logmsgbot: reedy synchronized php-1.20wmf9/
15:08 paravoid: switched European bits traffic back to esams
14:50 Tim: on cp3002: temporarily set net.ipv4.tcp_tw_recycle=1
14:41 Tim: on cp3002 increasing tcp_max_tw_buckets
14:33 Tim: on cp3001: date -s `"date"` to fix leap second issue
13:26 Nemo_bis: Reedy says: Status: Down - routing issues to Tampa !Wikipedia !Wikimedia
13:17 Nemo_bis: more or less everything at !Wikimedia / !Wikipedia seems down
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Aug 6 02:27:00 UTC 2012

August 5

21:54 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Update per rearrangements etc'
21:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'more cp updates to $wgSquidServersNoPurge'
20:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add some eqiad internal ips for squids to wgSquidServersNoPurge'
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Aug 5 02:24:31 UTC 2012

August 4

02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sat Aug 4 02:27:51 UTC 2012

August 3

23:50 logmsgbot: reedy synchronized wmf-config/secure.php 'fix bits usages'
21:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38988 - Enable Extension:Collection for Wikimedia Sverige'
18:58 maplebed: updated DNS to fix *.svc.{esams,eqiad}.wmnet forward resolution
18:49 andrewbogott: turned off nova-compute on virt1, virt2, virt3, virt4
18:44 maplebed: updating DNS adding swift eqiad LVS address
17:42 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler 'Updated to master (0207962155b810b11100bc6d05c3562949e7c1a9)'
07:58 hashar: GlusterFS upgrade of labs project storage seems to have been completed. Fixed the issues I had with it on 'beta' project, yeah!!!!!
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Aug 3 02:25:37 UTC 2012
01:54 domas: bumped memcached timeout to be slightly above min_rto, 200ms->250ms, should eliminate most of stupid memcached errors because of datacenter network failures
01:53 logmsgbot: midom synchronized wmf-config/mc.php

August 2

22:48 Ryan_Lane: upgrading glusterfs for project storage to 3.3
22:05 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Moved all wikis to multiwrite backend'
22:05 mutante: dist-upgrading sodium
21:47 hashar: added Dsc to the Gerrit 'integration' group. David now got access to the various integration/* repositories.
20:34 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo/modules/base/ext.echo.base.css
20:30 logmsgbot: asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent back to false, no change observed to current issue'
20:12 logmsgbot: asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent = true as an experiment'
20:06 Andrew: Leaving Echo deployment there for now.
20:04 Andrew: Rolled back LQT updates, they seem to break other notifications. We'll figure out the bugs with that in another window.
20:02 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
19:56 Andrew: deployed Echo to mediawiki.org
19:56 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
19:55 logmsgbot: andrew synchronized wmf-config/CommonSettings.php
19:55 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo
19:54 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
19:44 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
19:44 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo
19:43 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
19:43 Andrew: Deploying Echo to test2wiki instead
18:38 Andrew: Turned Echo back on on testwiki
18:35 Andrew: Stealing E2's deployment window to have another crack at Echo
16:44 paravoid: sync-apache/apache-gracefull-all for BZ #38905 (ShortUrl on !Wikipedia)
16:20 RobH: authdns-update for smokeping
13:05 logmsgbot: nikerabbit synchronized php-1.20wmf8/extensions/Translate/specials/ 'Translate bug fixes'
04:45 logmsgbot: tstarling synchronized php-1.20wmf8/skins/common/commonElements.css
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Aug 2 02:27:17 UTC 2012
02:11 Ryan_Lane: deployed OATHAuth 8992e6f541cd7adf8111ccd87f25a481d3759b33 on labsconsole
02:01 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
01:26 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/
01:16 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php
00:55 logmsgbot: asher synchronized wmf-config/db.php 'moving watchlist/special contrib queries to db63, preparing to decom db12'
00:47 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php
00:43 logmsgbot: asher synchronized wmf-config/db.php 'adding db63 to s1 at a low weight'

August 1

23:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:27 mutante: rebuilding wikitech-l archives
23:16 logmsgbot: andrew Finished syncing Wikimedia installation... :
22:59 Andrew: pulling the plug on Echo deployment due to replication-lag related bug.
22:57 logmsgbot: andrew Started syncing Wikimedia installation... :
22:32 Andrew: Running scap to deploy Echo to test
22:31 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made mw.org use the multiwrite backend.'
22:23 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'logging for /home being used for thumbs for testwiki'
22:14 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed read-only flag from testwiki backends.'
22:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made testwikis use multiwrite backend.'
21:27 Andrew: Added Echo tables to testwiki
21:11 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Configured $wgTimedTextForeignNamespaces.'
21:08 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler/handlers/TextHandler/TextHandler.php 'Deployed a247b6264c99cff66e4e3ae3428a46f971fab3a2'
21:07 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler/TimedMediaHandler.php 'Deployed a247b6264c99cff66e4e3ae3428a46f971fab3a2'
21:01 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
20:48 logmsgbot: andrew synchronized wmf-config/CommonSettings.php
20:47 Andrew: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
20:47 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
20:46 Andrew: syncing {Common,Initialise}Settings.php
20:46 Andrew: Added configuration skeleton for Echo in {Common,Initialise}Settings.php
20:46 Andrew: Created tables for Echo on mediawikiwiki
19:16 MaxSem: Ran extensions/GeoData/updateIndexGranularity.php on testwiki
19:04 binasher: started hotbackup of db1017 to db63
19:04 mark: Ran reprepro --delete clearvanished on brewster to cleanup removed repositories karmic-wikimedia and oneiric-wikimedia
19:03 mutante: installing a couple lib upgrades on fenari
18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything else over to 1.20wmf8
18:04 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler 'Updating to 4b7c77d63d7798a024c7f8b4ba01e2f3e7203550'
17:37 maplebed: pushing dns typo correction
10:32 Tim: on dobson: restarted ntpd
10:30 Tim: on linne: restarted ntpd in an attempt to fix the leap indicator
02:49 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Aug 1 02:49:33 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 1 02:25:43 UTC 2012
01:53 logmsgbot: catrope synchronized php-1.20wmf8/cache/l10n
01:05 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayer.js 'touch'
00:33 Tim: leap second event plus one month is causing an apparent 1s step in time reported by linne/dobson as seen by some clients, causing nagios errors etc. Will step.

July 31

23:12 logmsgbot: aaron Finished syncing Wikimedia installation... :
22:32 logmsgbot: aaron Started syncing Wikimedia installation... :
22:19 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disable ogghandler for wikis using TMH.'
22:18 logmsgbot: catrope synchronized php-1.20wmf7/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php Deploying 3d45bc25577f33bc220e17a7671d391977f9fbf9 and 4c7db40395b6b2d273e08f6b0e59c5a74cb5ed34'
22:17 RobH: authdns-update for ms-be1006 and ms-be1012
22:04 binasher: added memcached_1.4.14-0wmf1_amd64 to precise-wikimedia
21:55 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
21:54 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
21:49 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
21:49 logmsgbot: aaron Finished syncing Wikimedia installation... :
21:45 logmsgbot: aaron Started syncing Wikimedia installation... :
21:08 logmsgbot: aaron Finished syncing Wikimedia installation... :
21:03 logmsgbot: aaron Started syncing Wikimedia installation... :
20:09 logmsgbot: mlitn Finished syncing Wikimedia installation... :
19:35 logmsgbot: mlitn Started syncing Wikimedia installation... :
19:26 RobH: authdns-update for wtp1 info
17:40 AaronSchulz: Initial swift originals migration for s1-s7 done
17:29 maplebed: finished swift deploy - image scaling requests now go straight to the rendering cluster
16:16 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'Recognize thumb_handler for testwiki and /home check.'
16:09 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Deployed handlerUrl for thumb zone.'
16:00 maplebed: beginning swift deploy to make thumbnail requests bypass ms5 and go straight from swift to the rendering cluster
14:33 mark: Repooled cp1042
14:33 mark: Reinstalled cp1042 with Precise
13:16 Tim: compiling phpllvm tests on bast1001
12:19 mark: Depooled cp1042 for reinstall with Precise
08:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
08:45 logmsgbot: reedy synchronized flaggedrevs.dblist 'Disable FR on eswikibooks'
02:54 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 31 02:54:35 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Jul 31 02:27:34 UTC 2012

July 30

23:49 binasher: adding revised enwiki.aft_article_feedback af_user_id_user_ip index via osc
23:48 binasher: dropping enwiki.aft_article_feedback af_user_id_user_ip index
21:48 logmsgbot: reedy synchronized php-1.20wmf7/extensions/Collection/.git 'Just for saper..'
20:51 binasher: adding enwiki.aft_article_feedback af_user_id_ip index
20:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
20:30 logmsgbot: reedy synchronized php-1.20wmf8/extensions/Collection
20:29 logmsgbot: reedy synchronized php-1.20wmf7/extensions/Collection
19:26 logmsgbot: reedy synchronized .
19:22 logmsgbot: reedy synchronized php-1.20wmf8/extensions/UploadWizard/
18:25 notpeter: removing srv194 from apaches pool due to logging issues
18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf8
17:42 notpeter: re-adding srv194 to pmtpa apaches pool for basic testing of precise
17:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove duplicate routing code'
16:55 logmsgbot: reedy synchronized wmf-config/
16:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable shorturl on tawikis'
16:31 Reedy: Created ShortUrl table on numerous tawiki* wikis
16:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ShortUrl on tawiki, hiwiki and orwiki'
16:26 Reedy: Add ShortUrl table to hiwiki, orwiki and tawiki
16:05 paravoid: sync-apache/apache-graceful-all for RT #2121
16:02 Reedy: Created FlaggedRevs tables on eswikiboooks
15:58 logmsgbot: reedy synchronized wmf-config/
15:53 Reedy: Created FlaggedRevs tables on eswiki
08:11 hashar: gallium/jenkins: deployed job for the WLMMobile nightly builds
07:40 hashar: gallium/jenkins: updating Android SDKs
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 30 02:45:58 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Jul 30 02:23:05 UTC 2012
01:00 Tim: installed mytop on fenari at faidon's request
00:52 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'docs'
00:49 Tim: killed all AFTv5 queries another few times
00:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabled properly'
00:45 Tim: killed ArticleFeedback queries on db12
00:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabled ArticleFeedback since it caused an overload on db12 and general site slowness'
00:32 Tim: on hume: kill -CONT
00:28 Tim: on hume: stopped populateRevisionSha1.php with kill -STOP due to excessive (800s) lag on db12

July 29

23:13 logmsgbot: reedy synchronized php-1.20wmf8/includes/WebRequest.php
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 29 02:45:34 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Jul 29 02:23:12 UTC 2012

July 28

06:29 logmsgbot: catrope synchronized php-1.20wmf8/extensions/Vector/modules/ext.vector.collapsibleNav.css 'touch'
06:29 logmsgbot: catrope synchronized php-1.20wmf7/extensions/Vector/modules/ext.vector.collapsibleNav.css 'touch'
02:51 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 28 02:51:29 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sat Jul 28 02:25:46 UTC 2012

July 27

21:16 RobH: updated dns for mgmt on wtp1.mgmt
20:42 cmjohnson1: setting db66
20:23 RobH: authdns-udpate for yttrium/wlm.w.o per rt 3221
20:12 cmjohnson1: setting up tmh2 rt3298
19:51 cmjohnson1: setting up tmh1 rt 3298
19:40 RobH: setting up yttrium per rt 3221
19:32 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
19:31 logmsgbot: aaron synchronized php-1.20wmf7/includes/DefaultSettings.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
19:31 logmsgbot: aaron synchronized php-1.20wmf7/includes/filerepo/FileRepo.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
19:26 logmsgbot: aaron synchronized php-1.20wmf8/thumb.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
19:25 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/FileRepo.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
19:24 logmsgbot: aaron synchronized php-1.20wmf8/includes/DefaultSettings.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
19:00 RobH: allocating potassium and zirconium to rt 3342 (misc db use) and shutting them down until they are installed.
18:37 RobH: another authdns-update, forgot to update asset tags to servernames for mgmt
18:33 RobH: authdns-update for tmh1/2
14:07 hashar: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
14:07 hashar: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
14:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'gerrit 16817 - fix nycwikimedia wgImportSources'
13:40 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Deploying I39d1ca7c: import sources for nycwikimedia'
12:51 Reedy: Created tgs_lang indexes on wikis that have translate installed
02:53 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Fri Jul 27 02:53:30 UTC 2012
02:31 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Jul 27 02:31:05 UTC 2012

July 26

22:26 logmsgbot: kaldari synchronized php-1.20wmf7/extensions/E3Experiments/Experiments.hooks.php 'syncing Experiments.hooks.php for wmf7'
22:25 logmsgbot: kaldari synchronized php-1.20wmf8/extensions/E3Experiments/Experiments.hooks.php 'syncing Experiments.hooks.php for wmf8'
22:00 logmsgbot: reedy synchronized wmf-config/
21:55 logmsgbot: reedy synchronized php-1.20wmf7/LocalSettings.php
21:48 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning E3Experiments on for en.wiki'
21:38 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:09 logmsgbot: kaldari synchronized php-1.20wmf8/extensions/E3Experiments 'sync E3Experiments stuff for en.wiki'
21:02 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on E3Experiments for en.wiki'
19:40 RobH: changed to calcium instead, as yttrium is a 610 and i only need a 310
19:37 RobH: claiming yttrium for smokeping install
19:23 RobH: authdns update for new services
18:59 logmsgbot: reedy synchronized php-1.20wmf8/extensions/WikiEditor
16:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
16:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Special -'
16:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'enable shorturl on testwiki and test2wiki'
16:42 Reedy: Created ShortUrl tables on test2wiki
13:44 hashar: deployment-prep rsync finished for both apache and upload6. Remounting and restarting apaches
13:23 cmjohnson1: search32 has hardware issues..powering down
08:43 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php '(bug 32516) Enable Narayam on pawiki.'
08:38 logmsgbot: nikerabbit synchronized php-1.20wmf7/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/16721'
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 26 02:46:34 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Jul 26 02:24:01 UTC 2012
00:21 logmsgbot: catrope synchronized php-1.20wmf8/skins/common/commonElements.css 'Deploy 0cc26386cec1955acf2e1f7341eaaa466ea220b2 and 162a25e96cf48cbd2545f6c3d54f0c20f1bcd520'
00:08 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Undo testing change'

July 25

23:46 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
23:30 binasher: completed abuse_filter_log migration for all wikis
22:54 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php
22:29 LeslieCarr: fixed puppetmaster on stafford (bad stafford, no cookie!)
22:20 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'url extraction fix'
22:15 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'debug logging'
22:08 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'url regex tweak'
22:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily set $wmgArticleFeedbackv5OversightEmails to my work e-mail for debugging'
22:05 binasher: started second step of abuse_filter_log migration via osc, indexes for new columns
21:56 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'debug logging'
21:08 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Re-implementing wgNoticeFundraisingUrl override for wmf7 wikis'
21:02 logmsgbot: aaron synchronized live-1.5/thumb_handler.php 'entry point to thumb.php'
20:43 binasher: started abuse_filter_log migration via osc. first step, adding afl_rev_id, afl_log_id columns. running in cluster order starting with s1 (enwiki)
20:30 logmsgbot: reedy synchronized php-1.20wmf8/includes/objectcache/MemcachedClient.php 'Bring in reversion to shut up warnings'
20:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all none wikipedia projects to 1.20wmf8
20:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:49 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/ 'Revert back to 0eeafeab44fb7a177ebf929c313999855656701f'
18:25 logmsgbot: reedy synchronized php-1.20wmf8/extensions/Quiz/
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, closed and special to 1.20wmf8
18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews to 1.20wmf8
18:02 binasher: streaming hot backup of db1033 to db63, intended db12 replacement and first precise enwiki db
17:10 mark: Added asw-c-eqiad to Torrus, RANCID, Observium
16:52 cmjohnson1: srv266 still working on it...going down
15:46 logmsgbot: reedy synchronized wmf-config/
15:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:33 Reedy: Created wikilove tables on svwikinews
15:24 Reedy: Created FlaggedRevs tables on cawikinews
15:23 logmsgbot: reedy synchronized wmf-config/
14:37 cmjohnson1: srv266 shutting down for attempted h/w repairs
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Jul 25 02:46:06 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 25 02:23:44 UTC 2012

July 24

23:23 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Synchronizing updates to MobileFrontend per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-06-24
22:52 logmsgbot: awjrichards Started syncing Wikimedia installation... : Synchronizing updates to MobileFrontend per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-06-24
22:26 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Fixing footer logo for enwiki on mobile view'
20:10 binasher: upgrading versions of xtrabackup and percona toolkit on all coredbs
19:45 logmsgbot: mlitn Finished syncing Wikimedia installation... :
19:10 logmsgbot: mlitn Started syncing Wikimedia installation... :
18:30 RoanKattouw: root@fenari:~# chmod g+w /home/wikipedia/common/php-1.20wmf8/cache/l10n/
18:30 RoanKattouw: root@fenari:~# chown l10nupdate:wikidev /home/wikipedia/common/php-1.20wmf8/cache/l10n
18:04 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
16:28 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
16:08 RobH: srv278 hardware worked on by chris, placing back in service to see if its going to stay fixed
14:30 cmjohnson1: correcting enviromental leads on powestrips in sdtpa
14:06 paravoid: force puppet run on all LVS servers for the ganglia change
09:10 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 36104 - Narayam on bnwikisource'
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Jul 24 02:48:21 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 24 02:26:52 UTC 2012
00:03 binasher: upgrading db1045 to precise for testing

July 23

23:11 AaronSchulz: Started originals migrations for remaining wikis
22:07 binasher: deploying new mobile redirector to eqiad squids
20:44 cmjohnson1: ms-be10 taking off-line for some hardware testing
20:13 logmsgbot: reedy synchronized php-1.20wmf7/includes/specials/SpecialUserlogin.php
20:12 logmsgbot: reedy synchronized php-1.20wmf8/includes/specials/SpecialUserlogin.php
19:31 RobH: authdns-update for transcode name updates
19:28 RobH: powering down old transcode1, was camera gateway sandbox, reclaiming name
18:57 maplebed: took ms-be10 out of rotation because it ate itself
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf8
17:03 RobH: reobooting potassium, shouldnt be in use, had tons of cruft ssh connections from month ago
16:26 Reedy: running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf5"
16:23 Reedy: running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf4"
16:22 logmsgbot: reedy Finished syncing Wikimedia installation... : 1.20wmf8 messages
16:04 paravoid: reinstalling srv281; disk full, hasn't run puppet for 300 days, depooled for ages
15:54 logmsgbot: reedy Started syncing Wikimedia installation... : 1.20wmf8 messages
15:44 logmsgbot: reedy Started syncing Wikimedia installation... : 1.20wmf8 messages
15:42 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf8 and rebuild localisation cache
15:23 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf8 and rebuild localisation cache
15:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rm chwikimedia we dont host it
15:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: chwikimedia to 1.20wmf7
15:14 Reedy: srv281 has a full / and hasn't had a puppet run in over 434298 minutes
15:10 logmsgbot: reedy synchronized php-1.20wmf8/cache/
15:06 logmsgbot: reedy synchronized php-1.20wmf8/ 'Initial sync out of 1.20wmf8'
13:44 RobH: authdns-update for ms-be eqiad hosts
12:46 maplebed: rebooting ms-be10 for xfs errors and a clean boot
12:38 maplebed: rebalanced swift rings moving more content to new object servers
12:26 logmsgbot: reedy synchronized flaggedrevs.dblist
12:26 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
12:25 Reedy: Created trwikiquote flagged revs tables
12:23 logmsgbot: reedy synchronized wmf-config/
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 23 02:22:45 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 23 02:22:33 UTC 2012

July 22

22:56 Ryan_Lane: restarted opendj on sanger. The process OOM'd due to heap size.
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 22 02:23:02 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 22 02:22:49 UTC 2012

July 21

17:42 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'deployed ffd469239a80b8bb9813eb029e4cd71a3ea52db9'
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 21 02:23:19 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 21 02:23:06 UTC 2012

July 20

18:57 paravoid: powerin on owa1/owa2 again, they're being used as ganglia aggregators for swift
18:11 Ryan_Lane: starting gerrit
17:59 binasher: db48 is now replicating from db1048 (and db1048 from db48)
17:26 RobHalsell: authdns-update run for new servers
15:29 paravoid: powering off the rest of mw10[0-9][0-9]
14:35 mark: Added server hydrogen to the dns_rec eqiad LVS pool
14:22 paravoid: powering off owa1/2/3, unused
13:54 paravoid: powering off all of mw1[0-9][0-9][0-9].eqiad.wmnet, unused
13:36 mark: Reinstalling hydrogen
13:19 paravoid: powercycling hydrogen, down since yesterday
12:59 paravoid: powercycling nescio/ns2, unresponsive network & console
10:38 mark: Built new varnish 3.0.3~rc1+persistent1-wm1 packages and inserted them into the precise-wikimedia APT repository
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 20 02:23:39 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Fri Jul 20 02:23:27 UTC 2012

July 19

22:48 logmsgbot: catrope Finished syncing Wikimedia installation... :
22:10 logmsgbot: catrope Started syncing Wikimedia installation... :
22:02 binasher: deploying new mobile redirector to pmtpa text squids (currently inactive)
21:39 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:36 binasher: deploying new mobile redirector to eqiad text squids
21:36 logmsgbot: catrope synchronized php-1.20wmf7/extensions/LastModified 'Remove E3Experiments cruft from LastModified'
21:07 binasher: deploying new mobile redirector to esams text squids
18:52 preilly: fixing subdomain check for zero vs m
18:52 logmsgbot: preilly synchronized wmf-config/mobile.php 'fix subdomain check'
18:21 RobH: updating dns with new zonefiles for legally won domain names
18:15 Jeff_Green: storage3 dist-upgrade and reboot
17:01 AaronSchulz: Running copyFileBackend.php for commons (shards c-f)
16:28 cmjohnson1: mw60 powering down to replace DIMM B1 rt3287
15:48 RobH: hydrogen repaired per rt 3243
15:37 RobH: rebooting hydrogen to set bios redirection
14:18 logmsgbot: reedy synchronized wmf-config/
13:42 Jeff_Green: dist-upgrade and reboot hosts in payments cluster
13:12 Reedy: Pointed /h/w/c/php to php-1.20wmf7
13:10 Reedy: Deleted php-1.20wmf6/cache/l10n from mediawiki-installation
02:31 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 19 02:31:02 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 19 02:23:27 UTC 2012

July 18

22:02 notpeter: ok, changed my mind one more time. removing srv194 from apaches pool for precise testing
21:41 notpeter: scratch that. removing srv289 from apaches pool for precise testing, not mw1
21:39 notpeter: removing mw1 from apaches pool to do precise test install
21:28 notpeter: adding php5 packages to precise-wikimedia repo
19:45 cmjohnson1: srv281 powering down for HW checks
19:09 cmjohnson1: srv278 powering down for hardware problems -- random reboot system already depooled
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 295 remaining wikis to 1.20wmf7
17:37 cmjohnson1: mw8 powering down to replace DIMM A3 rt 3273
17:25 cmjohnson1: search32 powering down to replace DIMM B1 rt 3076
16:55 logmsgbot: reedy synchronized php-1.20wmf7/includes/api/ApiQuerySiteinfo.php
12:52 pp-pdf1: upgraded mwlib to 0.13.11, restarted all services
07:58 hashar: gallium: added Firefox 14 to Testswarm, disabled Firefox 13.
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 18 02:47:23 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 18 02:24:14 UTC 2012

July 17

23:24 Tim: on srv193: removed core dump files, disabled core dumping, restarted apache
19:52 logmsgbot: mlitn Finished syncing Wikimedia installation... :
19:09 logmsgbot: mlitn Started syncing Wikimedia installation... :
15:18 mark: lvs1002 is back up and idling
15:15 mark: Fixed serial console redirection after boot to OFF on lvs1002
15:06 mark: lvs4 is back up and serving traffic
14:51 mark: Reinstalling lvs4 with Ubuntu Precise
14:48 mark: Stopped PyBal on lvs4, failing over traffic to lvs3
14:47 mark: Reinstalling lvs1002 with Ubuntu Precise
14:46 mark: Stopped PyBal on lvs1002, failing over traffic to lvs1005
14:44 mark: lvs1001 is back up and serving traffic
14:11 mark: Stopped PyBal on lvs1001, failing over traffic to lvs1004
13:09 maplebed: changed auth URL for swift to use load balancer rather than round robin DNS
13:08 mark: lvs1003 is back up and serving traffic
13:07 maplebed: changed order on ganglia swift view graphs to group by metric rather than host
13:02 logmsgbot: reedy synchronized wmf-config/
12:53 mark: Fixed boot order on lvs1003
12:34 mark: Reinstalling lvs1003 with Ubuntu Precise
12:28 mark: Stopped PyBal on lvs1003, failing over traffic to lvs1006
12:10 mark: amslvs1 is back up and serving traffic
11:44 mark: Reinstalling amslvs1 with Ubuntu Precise
11:40 mark: Stopped PyBal on amslvs1, failing over traffic to amslvs3
11:32 mark: amslvs2 is back up and serving traffic
10:47 mark: Reinstalling amslvs2 with Ubuntu Precise
10:36 mark: Stopped PyBal on amslvs2, failing over traffic to amslvs4
07:30 Tim: testing envvars/apache2.conf change on srv193
07:09 Tim: restarted apache on srv193
02:58 Tim: graceful restart of all apaches
02:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jul 17 02:47:06 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 17 02:23:41 UTC 2012
00:44 Tim: deploying new redirects.conf
00:28 Tim: testing new redirects.conf on mw1

July 16

23:03 pp-pdf1: restarted nserve
23:02 pp-pdf1: fixed nserve handling of filenames with whitespace
22:46 pp-pdf1: restart all services
22:45 pp-pdf2: restart all services
22:45 pp-pdf3: restart all services
22:45 pp-pdf1: update mwlib to 0.13.9
22:45 pp-pdf2: update mwlib to 0.13.9
22:45 pp-pdf3: update mwlib to 0.13.9
22:44 pp-pdf2: update greenlet to 0.4.0
22:44 pp-pdf1: update greenlet to 0.4.0
22:44 pp-pdf3: update greenlet to 0.4.0
22:44 pp-pdf2: upgrade qserve to 0.2.8
22:44 pp-pdf3: upgrade qserve to 0.2.8
22:43 pp-pdf1: upgrade qserve to 0.2.8
22:20 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Disabling epub generation from Collection extension on all wikis except for simple and test'
22:19 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Making epub generation for Collection extension configurable'
18:26 AaronSchulz: Running copyFileBackend.php for commons (shards 8-b)
18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf7
17:08 LeslieCarr: reactivating XO transit connections on cr1-sdtpa
17:03 LeslieCarr: draining cr2-eqiad to cr1-sdtpa link for moving of fiber
15:56 mark: lvs3 is back up, and idling
15:56 mutante: fixing pt.planet after missing locale has been added
15:48 mutante: installing package upgrades on singer (planet)
15:43 cmjohnson1: mw8 shutting down and taking offline to run Dell's dset program
15:38 mutante: planet - svn up the config
15:29 mark: Reinstalling lvs3 with Ubuntu Precise
15:17 mark: lvs5 is back up and serving traffic
14:49 mark: Reinstalling lvs5 with Ubuntu Precise
14:45 mark: Stopped PyBal on lvs5 to failover traffic to lvs1
14:35 maplebed: adjusted swift rings; set new object servers to 20, new container servers to 100
14:33 mark: lvs6 is back up and serving traffic
14:12 mark: Reinstalling lvs6 with Ubuntu Precise
14:02 mark: Stopped PyBal on lvs6 to failover traffic to lvs2
13:58 mutante: mw8 - alright, most likely just needs new DIMM per cmjohnson
13:50 mutante: mw8 PHP fatal errors, running out of memory
12:56 mutante: sync-apache, graceful-all to fix wikipedia.cz redirect
05:31 apergos: reboot to upgrade kernel etc on mw8 since it's been flapping anyways
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 16 02:46:14 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 16 02:23:04 UTC 2012

July 15

20:19 Aaron|home: Running copyFileBackend.php for commons (shard 7)
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 15 02:46:33 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 15 02:23:27 UTC 2012

July 14

18:46 LeslieCarr: powercycling frozen db1029
18:01 LeslieCarr: rebooting unresponsive mw1111
16:41 logmsgbot: midom synchronized php-1.20wmf6/includes/GlobalFunctions.php 'message key analysis'
16:18 logmsgbot: midom synchronized php-1.20wmf6/includes/GlobalFunctions.php 'message key analysis'
03:07 Aaron|home: Running copyFileBackend.php for commons (shards 5-6)
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 14 02:47:32 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 14 02:23:14 UTC 2012
00:25 logmsgbot: reedy synchronized php-1.20wmf7/includes/api/ApiDelete.php

July 13

18:08 logmsgbot: reedy synchronized php-1.20wmf7/includes/GlobalFunctions.php
18:08 logmsgbot: reedy synchronized php-1.20wmf6/includes/GlobalFunctions.php
17:58 logmsgbot: reedy synchronized wmf-config/
17:57 logmsgbot: reedy synchronized php-1.20wmf7/
17:56 logmsgbot: reedy synchronized php-1.20wmf6/
17:32 logmsgbot: reedy synchronized wmf-config/
17:21 AaronSchulz: Running copyFileBackend.php for commons (shards 1-4) (actually started yesterday)
15:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable aftv5 on en_labswikimedia due to out of date schema'
15:21 logmsgbot: reedy synchronized php-1.20wmf7/includes/GlobalFunctions.php
15:20 logmsgbot: reedy synchronized php-1.20wmf6/includes/GlobalFunctions.php
15:20 logmsgbot: reedy synchronized php-1.20wmf7/includes/DefaultSettings.php
15:19 logmsgbot: reedy synchronized php-1.20wmf6/includes/DefaultSettings.php
15:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Setting wgDBerrorLogInUTC = true'
15:07 hasharDeadmau5: unknown column 'af_is_featured' on en_labswikimedia@db25
14:55 mark: Increased all mobile varnish server weights from 10 to 100 to aid chash
14:54 mark: Added cp1041.eqiad.wmnet back into the mobile LVS pool
10:10 mutante: mw1016 - was down, reinstalling with precise
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 13 02:45:59 UTC 2012

July 12

22:15 Aaron|home: Running copyFileBackend.php for commons (shards 0,c-f)
20:58 Aaron|home: Running copyFileBackend.php for commons (shards 0,8-b)
14:22 mutante: apache children on srv193 keep segfaulting since yesterday (test.wp)
14:00 RobHalsell: shutting down srv206 for chris per rt241
11:55 logmsgbot: reedy synchronized wmf-config/
10:40 mark: Depooled mobile varnish server cp1041 for vlan/hostname change and reinstallation with precise
04:38 binasher: started hot backup of db48 to db1048
02:50 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 12 02:50:49 UTC 2012
02:29 maplebed: adjusted the swift rings a bit to move some more traffic to the new hosts. set object partitions to weight 10, account and container 60.
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 12 02:26:57 UTC 2012

July 11

22:52 AaronSchulz: Running copyFileBackend.php for commons (shards 0,4-7)
22:06 AaronSchulz: Running copyFileBackend.php for commons (shards 0,0-3)
21:36 binasher: running hotbackup of db48 -> db49 (otrs / external misc)
21:31 binasher: rebooting db49 for kernel upgrade
21:23 binasher: rebuilding db49 (otrs slave)
21:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:16 logmsgbot: reedy synchronized php-1.20wmf7/extensions/CentralNotice
19:33 cmjohnson1: db10 swapping disk1 out rt3251
19:33 RobH: shutting down srv203 for hardware checking per rt3110
19:12 hashar: finally stopped using my production ssh key on labs.
19:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikiversity to 1.20wmf7
19:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.20wmf7
19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikisource to 1.20wmf7
19:03 cmjohnson1: search35 shutting down to reseat DIMM rt-3260
18:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Switch special, private and closed wikis from 1.20wmf6 to 1.20wmf7
15:16 logmsgbot: robh synchronized wmf-config/mc.php 'moving srv208 to down'
15:14 logmsgbot: robh synchronized wmf-config/mc.php
14:30 cmjohnson1: shutting down search32 to reseat DIMM rt-3076
11:16 mutante: yay @ svn removal
11:13 hashar: /home/wikipedia/conf/httpd cleaned out the svn repository. That directory is now 100% under git tracking yeah!!! Thanks TIm!
11:11 mutante: rebooting srv266
11:05 mutante: purged wikipedie.cz/Experti_na_prirodu URLs with purgeList.php
10:52 mutante: apache-graceful-all
10:48 mutante: sync-apache to push fixed wikipedie.cz redirect
08:36 hashar: synced /h/w/conf/httpd git and svn repositories
08:33 mutante: chmod -R g+w /home/wikipedia/conf/httpd on fenari to fix group write on .git/objects
07:59 mutante: srv266 - package/kernel upgrades
07:35 mutante: powercycled downed owa3, installed kernel upgrades on owa1-3 (but they are also waiting to be repurposed, RT:2511)
04:05 Tim: on hume: running updateCollation.php --dry-run on ptwiki to determine whether there will be any key truncation issues with bug 35632
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 11 02:48:18 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 11 02:25:19 UTC 2012

July 10

23:47 AaronSchulz: Doing shards c-f
23:47 maplebed: put two new swift front ends into rotation ms-fe3 and ms-fe4
23:40 AaronSchulz: Doing shards 8-b
23:34 AaronSchulz: Doing shards 4-7
23:31 AaronSchulz: copyFileBackend.php run rate for above processes is at /home/aaron/NFStoSwiftCopyRate, currently 50
23:26 AaronSchulz: Running copyFileBackend.php for zhwiki public zone shards 0-3 on hume
23:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:12 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'moving parsercache from db40 to pc1'
22:05 maplebed: put swift backends ms-be9-12 in rotation for containers (weight 30) and objects (weight 5)
21:17 maplebed: rebooting new ms-be6-8 to change BIOS setting to boot from disk
20:26 logmsgbot: mlitn Finished syncing Wikimedia installation... :
20:26 RobH: shutting down sq36 for hardware troubleshooting
19:38 LeslieCarr: restarted frozen pdns on ns2
19:28 logmsgbot: mlitn Started syncing Wikimedia installation... :
19:15 cmjohnson1: sq36-requires a hard shutdown-unresponsive to mgmt. rt-3254
19:05 RoanKattouw: chmod g+w /home/wikipedia/common/php-1.20wmf7/cache/l10n/
19:03 RoanKattouw: chown l10nupdate:wikidev /home/wikipedia/common/php-1.20wmf7/cache/l10n/
18:58 hashar: reworked the git repository in /home/wikipedia/conf/httpd , manually synced changes from svn to the git repo
18:12 RobH: srv266 shutdown for chris
18:06 cmjohnson1: srv266 being brought down for an extended period of time to run diagnostic tests
17:25 cmjohnson1: srv266 shutting down for HW troubleshooting rt-2896
16:33 mutante: package upgrades and kernel on niobium
16:28 mutante: powercycling niobium
15:49 mutante: powercycling mw1008,mw1070,mw1073
14:10 logmsgbot: reedy synchronized hastidy
13:26 logmsgbot: tstarling synchronized live-1.5/robots.php
13:17 logmsgbot: tstarling synchronized live-1.5/favicon.php
13:13 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:53 Tim: deploying favicon.php test alias
12:41 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:30 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:11 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:10 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:03 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:02 logmsgbot: tstarling synchronized live-1.5/favicon.php
11:58 logmsgbot: tstarling synchronized live-1.5/favicon.php
10:10 logmsgbot: reedy synchronized images/sul/ 'crushed'
10:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
07:49 logmsgbot: tstarling synchronized live-1.5/favicon.php
07:46 logmsgbot: tstarling synchronized live-1.5/favicon.php
07:41 logmsgbot: tstarling Finished syncing Wikimedia installation... :
06:32 logmsgbot: tstarling Started syncing Wikimedia installation... :
06:14 logmsgbot: tstarling Started syncing Wikimedia installation... :
06:06 Tim: removing php-*/cache/l10n/l10nupdate* and running scap with the new version of scap from I8bcd2817
05:01 Tim: on fenari: running git submodule update --init in /var/lib/l10nupdate/mediawiki/
03:56 Tim: testing new scheme for LU involving not pushing out LU files to all apaches
02:49 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 10 02:49:16 UTC 2012
01:53 brion: test.wikipedia.org fails with 'TrustedXFF: hosts file missing.'
01:44 Tim: on manganese: ran date -s "`date`" to make sure that isn't the cause of the high CPU usage during clone, it wasn't
01:44 Tim: running rebuildLocalisationCache.php for testwiki
01:43 brion: test.wikipedia.org broken again, "No localisation cache found for English."
00:51 brion: test.wikipedia is broken loading Cite.php

July 9

23:53 Tim: deleting and recreating php-1.20wmf7 to test a script
20:51 logmsgbot: reedy synchronized php-1.20wmf7/includes/media/FormatMetadata.php
20:51 logmsgbot: reedy synchronized php-1.20wmf6/includes/media/FormatMetadata.php
20:30 pp-pdf3: upgraded mwlib.epub
20:30 pp-pdf1: upgraded mwlib.epub
20:30 pp-pdf2: upgraded mwlib.epub
20:27 cmjohnson1: db10 swapping disk1 for new disk
19:41 logmsgbot: reedy synchronized php-1.20wmf6/includes/WikiPage.php
19:37 logmsgbot: reedy synchronized php-1.20wmf7/includes/WikiPage.php
18:13 logmsgbot: reedy synchronized php-1.20wmf7/extensions/CodeReview/ 'CR to trunk to fix fatals'
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.20wmf7
15:45 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf7.php
15:43 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
15:01 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
14:54 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
14:47 logmsgbot: reedy synchronized php-1.20wmf7
14:17 Reedy: Copying php-1.20wmf7 from /tmp to /h/w/c on fenari
13:26 Reedy: Running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf3"
13:23 Reedy: Running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf2"
13:13 Reedy: Removed wmf2 and wmf3 from bits docroots
13:11 mutante: powercycling and upgrading a couple more mw* servers
13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:01 mutante: labs.wikimedia.org is now a redirect to labsconsole
12:49 mutante: authdns-update to add labs.wm entry for redirect to labsconsole
08:34 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'adjust $wgCollectionFormats'
08:16 hashar: pallium, updating jenkins build script with gerrit 14666 & gerrit 14667
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 9 02:25:57 UTC 2012

July 8

12:17 apergos: removed HTCPpurger.log.1 and current log ater restart of purger on ms6, /was full. people reporting thumb issues from europe
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 8 02:26:21 UTC 2012

July 7

16:39 logmsgbot: reedy synchronized docroot/bits/DolphinBrowser/js/ 'Remove BOM'
16:10 logmsgbot: reedy synchronized php-1.20wmf6/extensions/cldr/ 'remove bom'
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 7 02:27:22 UTC 2012

July 6

21:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38227 - Please enable WikiLove on Turkish Wikipedia'
21:48 binasher: truncating pagetriage tables on enwiki (per bsitu)
21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38227 - Please enable WikiLove on Turkish Wikipedia'
21:46 Reedy: Created WikiLove tables on trwiki
21:39 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php
21:07 maplebed: removed a bit more load from the swift spinning media container servers by adjusting the ring weights
20:45 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'rev debug logging'
20:22 logmsgbot: aaron synchronized php-1.20wmf6/includes/Revision.php 'es debug logging'
20:20 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'es debug logging'
19:49 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
19:30 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
19:24 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
17:28 maplebed: adjusted swift ring files to move container listings off spinning disks (ms-be1-4)
17:04 logmsgbot: reedy synchronized wmf-config/
16:53 Reedy: Running updateCollation.php in foreachwiki on fenari in screen as reedy
16:45 Reedy: Running updateCollation.php against enwiki on fenari in screen as reedy
16:39 Reedy: Running updateCollation.php against commonswiki on fenari in screen as reedy
16:08 logmsgbot: reedy synchronized images/wiki-en.png
15:54 logmsgbot: reedy synchronized images/sul/meta.png
15:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:21 cmjohnson1: ms-be5 removing disk0 to replace with good disk
14:38 RobHalsell: authdns-update to correct typo in mgmt dns entry
13:22 mark: Inserted new pybal 1.04 package in the precise-wikimedia APT repository, and upgraded all precise LVS servers
11:31 logmsgbot: tstarling Finished syncing Wikimedia installation... :
11:25 logmsgbot: tstarling Started syncing Wikimedia installation... :
11:20 Tim: running scap to delete docroot files per I48502d90 and Ie3afd137
08:58 mutante: dist-upgrading (unused) db10xx servers
07:43 Tim: also trusted-xff.phps
07:39 Tim: removing some untracked junk from /home/wikipedia/common
07:29 mutante: continue to restart and upgrade downed mw10xx servers
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 6 02:26:22 UTC 2012
01:35 binasher: updated squid redirector to cover wiki(quotes|books|versity)
00:13 Ryan_Lane: restarting gerrit
00:09 Ryan_Lane: force running puppet on manganese, it'll restart gerrit

July 5

23:54 Ryan_Lane: force running puppet on formey and manganese, since a config change is involved, it's going to restart
23:45 Ryan_Lane: restarting gerrit on manganese
23:38 Ryan_Lane: upgrading gerrit on formey
23:21 Ryan_Lane: updating database for gerrit
23:09 Ryan_Lane: upgrading gerrit on manganese
23:08 Ryan_Lane: stopping gerrit on manganese and disabling puppet
23:07 Ryan_Lane: stopping gerrit on formey and disabling puppet
22:47 logmsgbot: preilly synchronized wmf-config/mobile.php 'add subdomain check'
22:34 logmsgbot: reedy synchronized wmf-config/
22:17 logmsgbot: reedy synchronized wmf-config/ 'Tidying config for randomrootpage'
21:55 logmsgbot: reedy synchronized wmf-config/ 'Strike 2'
21:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'revert'
21:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Random root page enabled everywhere but wikipedias'
21:47 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:34 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:18 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Bumping AFTv5 lottery percentage to 20% of en.wiki'
21:15 LeslieCarr: powercycling unresponsive mw1116
21:09 maplebed: added new ms-be pmtpa hosts to DNS
20:03 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue with token'
19:14 Jeff_Green: *somebody* purged binary logs on blondel
18:58 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue'
18:54 mark: Upgraded pybal on all precise LVS servers
18:43 mark: Built new pybal_1.03 package and inserted it into the precise-wikimedia APT repository
17:41 mutante: powercycling db1048, mw1136, mw1046
17:32 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'reenabling db40'
17:30 mutante: more powercycling and upgrading: mw1036, mw1134, mw1043 ..
17:26 mutante: powercycling db1009,db1010
17:22 mutante: powercycling db1027,db1028
17:14 mutante: powercycling db1013
16:50 mutante: powercycling downed db1015
16:44 mutante: powercycling and upgrading more mw10xx servers, 1017,1023,1025 ...
16:27 mutante: mw1002, mw1007,mw1009,mw1011 - crashed,powercycling,dist-upgrading+kernel,reboot
16:18 mutante: powercycling mw1002
15:37 mutante: argon back up with new kernel,mysql,grub,.. looks happy afaict
15:33 mutante: argon (limesurvey) fscked, dist-upgrading
15:28 mutante: powercycling argon
15:25 hashar: updated Jenkins configuration on gallium : Updating f407ebe..4b669b9
14:08 cmjohnson1: HUME replacing disk 0
14:07 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'disabling parser cache for now'
11:54 mark: Inserted new pybal_1.02 package into APT distribution precise-wikimedia
11:17 mark: Installed new pybal snapshot build for testing on lvs1005
11:13 mutante: add missing Russian locales on singer, run localegen, run ru.planet update
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 5 02:26:45 UTC 2012

July 4

16:32 mutante: wikidata.org on now - redirect purged from squids
15:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'imports'
15:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable LQT on fiwikimedia'
15:39 Reedy: Killed php-1.20wmf5 localisation cache from mediawiki-installation group
15:37 Reedy: Created LQT tables on fiwikimedia
15:25 mutante: wikidata.org works now, besides old redirect may still be cached on cp* boxes (not purged by purgeList.php via multicast?). http://www.wikidata.org/?notcached
15:06 mutante: sync-common-file extract2.php, apache-graceful-all
15:04 logmsgbot: dzahn synchronized extract2.php
13:23 mutante: apache-graceful-all to add wikidata.org virtual host
13:21 mutante: svn commiting gerrit 9874, sync-apache
13:14 mutante: git pull in /h/w/common/docroot . adding wikidata.org files on fenari. , then "sync-docroot"
12:44 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
12:44 mutante: updating/syncing interwiki cache
09:32 hashar: swift-container-auditor seems to get down from time. Nagios reporting 0 processes at 8:15am and 9:25am UTC (I guess it get restarted automatically by puppet)
07:34 hashar: updating Jenkins copy of integration/jenkins from 0f069c3 to e264d1b. Bring new ant script + update to testswarm fetcher
07:09 Tim: on srv193: ran dpkg --set-selections to revert holds on php5 packages and ran apt-get upgrade
07:07 Tim: on srv193: fixing broken PHP packages causing puppet failure, nothing in the server admin log about them so I assume they were installed by accident
06:21 Tim: deployed Idb6d9a8b and restarting apaches
06:11 Tim: deployed Id7008681 and restarting apaches
05:45 Tim: reniced apache processes to level 0
05:04 Tim: deploying apache nice level change per RT #664
03:59 Tim: on mw1: experimenting with renice methods for RT 664
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 4 02:26:39 UTC 2012

July 3

23:03 logmsgbot: mlitn synchronized php-1.20wmf6/extensions/ArticleFeedbackv5/
22:36 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'weekly update'
22:27 hashar: updating testswarm submitter on gallium
21:01 logmsgbot: mlitn Finished syncing Wikimedia installation... :
20:37 logmsgbot: mlitn Started syncing Wikimedia installation... :
20:18 logmsgbot: asher synchronized wmf-config/db.php 'lowering db32 weight'
19:34 logmsgbot: asher synchronized wmf-config/db.php 'lowering db36 weight'
19:31 logmsgbot: asher synchronized wmf-config/db.php 're-add db36, db32 (low weight), es3 (innodb)'
18:11 RobH: virt1006 mgmt serial not set correctly, fixed
17:51 RobH: investigating stat1001 power issue
17:22 RobH: fluorine offlining to test disks
17:22 RobH: pulling helium offline for disk testing with fluorine disks
17:10 RobH: db1047 disk0 rebuild in progress
17:04 RobH: replacing bad disk in db1047
16:45 logmsgbot: reedy synchronized wmf-config/ 'Various config changes'
16:30 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.7.xsd
16:15 Jeff_Green: silicon gets dist-upgrade & reboot
16:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgCheckSerialized is deaded'
03:53 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable API action=purge on commonswiki'
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jul 3 02:27:02 UTC 2012

July 2

22:33 binasher: rebooted db36 for kernel upgrade
22:25 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db36'
22:02 brion: fun with routers in tampa, wikis down
21:48 maplebed: rebooted emery - it's been unresponsive for 3 days.
18:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:30 hashar: set up ignore file in httpd configuration directory
18:23 logmsgbot: reedy synchronized wmf-config/ 'Enable WikimediaShopLink on enwiki'
18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 273 wikipedias to 1.20wmf6
18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf6
15:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '/etc/wikimedia-realm detection https://gerrit.wikimedia.org/r/13888'
15:18 logmsgbot: hashar synchronized docroot/bits/static-master '(bug 37245) docroot 'static-master' for beta bits'
15:04 mutante: authdns-update to switch jobs.wm redirect to wikimedia-lb to fix SSL cert mismatch (RT-3071)
14:55 mark: Reboot of cr1-sdtpa did not fix the RE packet loss issue... therefore unlikely to be leap second related
14:41 mark: Rebooting cr1-sdtpa
14:37 mark: Shutdown PyBal BGP sessions on cr1-sdtpa
14:34 mark: Shutdown BGP session to 2828 on cr1-sdtpa
13:36 hashar: db12 suffering some 1400sec (and growing) replag. mysqldump in progress on that host.
12:35 mutante: installing upgrades on fenari (linux-firmware linux-libc-dev..)
12:27 mutante: rebooting gallium one more time to install kernel
12:26 mutante: upgrading kernel on gallium
12:23 logmsgbot: hashar synchronized live-1.5/CREDITS
11:31 mark: Now we have packet loss within pmtpa/sdtpa... reverting change
10:57 mark: Problems on one of two pmtpa-eqiad waves; raised OSPF metric to 60 to failover traffic to the other link
10:50 Tim: fixing leap second issue on bastion1 by rebooting it
10:47 Tim: fixed leap second issue on bastion-restricted
09:56 Tim: fixing leap second issue on virt1,virt2,virt3,virt4,virt5
09:52 Tim: fixing leap second issue on aluminium,gallium,manganese
09:47 Tim: fixing leap second issue on formey,grosley,hooper,sanger,sockpuppet
09:43 Tim: on fenari: fixed leap second issue with the mozilla method
09:39 apergos: rebooting gallium, it's pretty unhappy (maybe related to leap second issue)
08:14 logmsgbot: hashar: srv190 srv266 srv281 timeouts on sync-file
08:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Bug 37457 - fix import sources for viwikibooks'
08:11 hashar: Stopped Jenkins on gallium. It is not doing anything anyway. Asked to reboot box RT #3208
02:53 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jul 2 02:53:51 UTC 2012
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 2 02:28:48 UTC 2012
01:48 Tim: kill -CONT on populateRevisionSha1.php processes
00:47 Tim: on nfs1: trying leap second fix suggested at https://bugzilla.mozilla.org/show_bug.cgi?id=769972#c5
00:26 logmsgbot: tstarling synchronized wmf-config/db.php 'reduce db32 read load to zero due to persistent lag'
00:12 Tim: switched enwiki back to r/w
00:12 logmsgbot: tstarling synchronized wmf-config/db.php
00:06 Tim: on hume: stopped all populateRevisionSha1.php processes with kill -STOP
00:03 logmsgbot: reedy synchronized wmf-config/db.php 's1/enwiki into readonly'

July 1

19:12 logmsgbot: reedy synchronized php-1.20wmf6/extensions/WikimediaMaintenance/ 'Update to master for hashar'
17:55 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
17:45 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
17:43 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
17:32 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
17:30 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
16:53 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
16:48 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'logging'
12:54 notpeter: also going to reboot all pmtpa search nodes. not in prod, but are still freaking out from leap second bug.
05:33 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'logging'
04:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jul 1 04:25:25 UTC 2012
04:06 Ryan_Lane: virt1000 is back up, rebooting virt0
04:01 Ryan_Lane: rebooting virt1000
03:16 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 1 03:16:39 UTC 2012
01:43 notpeter: that worked. restarting all remaining search nodes.
01:39 notpeter: problem with lucene persisting through service restart, but not node restart. restarting en pool nodes.
01:20 paravoid: restarting opendj (nfs1/nfs2), load spike, possibly related to leap second
00:51 notpeter: search1004 dead. powercycling.
00:50 notpeter: based on ganglia evidence, lucene seems to have been affected by leap second bug. restartig each instance, one minute wait in between

June 30

16:12 mark: Temporarily added path 6939+ 14907+ to AVOID-PATHs on cr2-knams
02:53 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 30 02:53:46 UTC 2012
02:28 maplebed: corrected LVS pdns_recursor config error causing DNS queries to fail on LVS servers in gerrit r13554 and r13555.
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jun 30 02:27:08 UTC 2012

June 29

19:49 hashar: restarting Jenkins to fix an issue with "parameterized builds" plugin. Updated git plugin as well.
19:35 RobH: dns update via authdns-update for vanadium ip
18:05 Jeff_Green: sync-apache and apache-graceful-all for http://donate.wikimedia.org-->https redirect
16:02 RobH: ms-be1001 and ms-be1002 powering down for ssd installation
15:59 RobH: authdns-update run
15:47 RobH: updating dns
15:16 mutante: dist-upgrading srv280,srv270,srv264
15:11 Jeff_Green: apache-graceful-all for redirect conf change
15:10 Jeff_Green: sync-apache to push out new foundation.conf
14:50 mark: Reinstalled chromium with precise
13:46 hashar: fixed interwiki on http://wikisource.org/ main page by hacking a script in production and refreshing cache
13:46 logmsgbot: hashar synchronized php-1.20wmf6/cache/interwiki.cdb 'Updating interwiki cache for 1.20wmf6'
13:32 logmsgbot: hashar synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
12:38 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
12:38 mutante: dumping interwiki and updating interwiki cache (to fix broken interwiki links, like wikisource.org -> wikipedia.org)
09:31 hashar: Jenkins: deployed gitsqlhaschanged patch ( d04f779 0f069c3 integration/jenkins.git )
07:56 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'send header from CS.php only for non CLI scripts gerrit 13435'
07:08 mutante: upgrading apt packages on brewster
02:51 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 29 02:51:13 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jun 29 02:26:25 UTC 2012

June 28

23:30 logmsgbot: reedy synchronized php-1.20wmf6/includes/resourceloader/ResourceLoader.php
23:15 binasher: completed aft offload_large_feedback migration on enwiki
23:03 logmsgbot: asher synchronized wmf-config/db.php 'returning db36'
21:50 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:39 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:13 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db36'
20:38 binasher: ran aftv5 offload_large_feedback migrations on testwiki and en_labswikimedia
20:14 RobH: dns update for pc1-pc3
19:54 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:08 logmsgbot: kaldari Started syncing Wikimedia installation... :
18:47 hashar: the internal change to CommonSettings.php caused a lack of stylesheet for less than a minute on most wikis. I did test on test.wikipedia.org and beta project, but there must be a logic error somewhere that mess with the prod projects. Revert changes have been sent out in gerrit and merged in master.
18:35 hashar: so the nicely reviewed changes broke the enwiki stylesheets :/ reverted change :-(((
18:34 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
18:33 hashar: srv190 and srv281 got ssh timeout
18:31 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
18:30 hashar: did various tests using eval.php. Most important is $realm -> production. $cluster -> pmtpa. Syncing
18:25 hashar: updating mediawiki-config to grab a12545d edceb4c & eee97ad
18:23 RobHalsell: swapped bad psu out of ms1001-array3, redundant so no downtime
15:40 RobHalsell: pulling the following servers, relocating to payments rack: payments1001-1004, boron, beryllium, lithium
15:34 RobHalsell: dns updated
15:31 RobHalsell: boron appears to be unallocated, pulling IP allocation, rack allocation, moving to payments per 1227
14:40 mutante: svn server is rebooting.brb
14:38 mutante: dist-upgrading formey (svn/gerrit), rebooting soon
14:38 Jeff_Green: manganese rebooted for kernel update
14:37 RobH: allocating yttrium to payments rack per rt 1227
14:24 Jeff_Green: manganese dist-upgrade
14:15 Ryan_Lane: restarting apache on manganese
14:15 Ryan_Lane: restarting gerrit
05:08 Tim: srv266 was flooding the fatal error log, complaining about a missing file. Killed apache and ran sync-common.
05:03 Tim: fixed fatal.log on fenari, socat was writing to a deleted file
02:58 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Thu Jun 28 02:58:42 UTC 2012
02:30 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jun 28 02:30:03 UTC 2012

June 27

23:50 K4-7131: sync'd payments cluster to 592e0a5ba195
23:43 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add rule for mediawiki'
22:37 K4-7131: sync'd payments cluster to 7e9072c2d571c
22:23 binasher: temporarily pulling srv211 from pybal
21:56 RobH: mw1102 has no nic0, rather than troubleshoot it for a long time, reinstall! (rt 3058)
21:01 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix CSRF'
21:00 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'fix CSRF'
20:55 RobH: db1003 back online, replaced mgmt cable and mgmt is working now as well
20:51 LeslieCarr: rebooting srv266 as it is unresponsive
20:44 RobH: db1003 mgmt issue due to bad cable, system booting back up, replacing mgmt cable
20:35 RobH: clean mysql shutdown, db1003 now offline
20:33 RobH: db1003 mgmt is not responsible, I need to remove power and reboot. confirmed iwth asher this is an s3 slave and can do a short downtime without issues
20:31 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'MF fixes and logging'
20:29 logmsgbot: maxsem synchronized php-1.20wmf6/extensions/MobileFrontend/ 'MF fixes and logging'
19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'disable LastModified and LastModified/E3Experiment'
19:54 logmsgbot: reedy synchronized php-1.20wmf6/maintenance/runJobs.php
19:53 logmsgbot: reedy synchronized php-1.20wmf5/maintenance/runJobs.php
19:32 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/specials/SpecialLanguageStats.php
19:19 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/
19:06 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/
19:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: meta back to wmf6, not cause of translate issues
18:53 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed to 1.20wmf6
18:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia wikis to 1.20wmf6
18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiversity to 1.20wmf6
18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.20wmf6
18:47 Jeff_Green: added several mobile hostnames to DNS for RT #2996
18:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf6
18:44 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved metawiki back to 1.20wmf5
18:41 K4-713: synchronized payments cluster to fundraising/1.20 de0256084a
18:33 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special wikis to php-1.20wmf6
18:29 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved en(wikibooks|wikinews|wikiquote|wikisource|wikiversity|wiktionary) to 1.20wmf6
18:05 RobH: cp1017 memory replaced
17:52 RobH: cp1017 is offline due to memory error. replacement memory on site, pulling system for swap
17:46 logmsgbot: preilly synchronized php-1.20wmf6/extensions/ZeroRatedMobileAccess 'update for landing page'
17:46 logmsgbot: asher synchronized wmf-config/mc.php 'disabling wgMemCachedPersistent; lowering wgMemCachedTimeout to 2x client default from 30x default'
17:45 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
17:44 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
17:19 LeslieCarr: restarting apache2 on srv258
17:11 maplebed: powercycled srv270
17:11 mutante: powercycling srv277 (had to, frozen console)
17:06 LeslieCarr: rebooting srv287
17:05 Ryan_Lane: rebooting srv280
17:03 mutante: powercycling srv280
17:01 paravoid: rebooting srv264, swapdeath
16:52 mark: Rebooting srv279, swapdeath
16:52 paravoid: rebooting srv275, swapdeath
16:50 paravoid: rebooting srv258, swapdeath
16:43 logmsgbot: reedy synchronized php-1.20wmf5/extensions/ArticleFeedbackv5/
15:08 Reedy: ExtensionDistributor now works from git on mediawiki.org
15:08 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/
15:06 logmsgbot: reedy synchronized php-1.20wmf5/extensions/ExtensionDistributor/
15:01 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/
14:51 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/ 'ED to trunk'
14:46 mark: Rebooting lvs1005 (after dist-upgrade)
13:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikimediaShopLink
13:24 mark: Added IPv6 LVS service IPs to the LVS_import policy on cr2-eqiad, for testing with lvs1005
13:22 Tim: installing apache2.2-bin-dbgsym on mw1
13:04 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for WikimediaShopLink
13:04 mark: Started PyBal 1.02 snapshot build on lvs1005
12:39 Reedy: WikimediaShopLink is deployed to testwiki/test2wiki
11:59 apergos: kicked morebots
13.56.44 (CEST) <logmsgbot> !log reedy synchronized wmf-config/ 'WikimediaShopLink'
13.54.28 (CEST) <logmsgbot> !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/
13.44.37 (CEST) <logmsgbot> !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/
11.36.23 (CEST) <mutante> !log starting swift-container-auditor on ms-be3
10.29.51 (CEST) <mutante> !log apt-get upgrade on gallium, installs newer jenkins
10.26.41 (CEST) <mutante> !log importing jenkins_1.472_all.deb into lucid-wikimedia using reprepro
08.04.26 (CEST) <logmsgbot> !log tstarling synchronized wmf-config/CommonSettings.php
08.00.45 (CEST) <logmsgbot> !log tstarling synchronized wmf-config/CommonSettings.php
04.48.48 (CEST) <logmsgbot> !log LocalisationUpdate completed (1.20wmf6) at Wed Jun 27 02:48:51 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Wed Jun 27 02:24:59 UTC 2012
01:23 Tim: on manganese: restarting gerrit
01:02 Tim: on manganese: killing all gitweb.cgi processes

June 26

23:49 Tim: on fenari: doing git and 1.19 checkouts for ExtensionDistributor
22:17 JeLuF: Slowly starting to import 100,000 images from the Deutsche Fotothek into Commons using importImages.php on fenari as user jeluf.
14:07 mutante: shutting down unused cp1037-cp1040 per RT-3189
10:48 mark: Moving all API traffic back to API apaches
10:41 mark: Restarted gmetad on nickel
10:36 apergos: powercycling srv261
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jun 26 02:47:56 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Tue Jun 26 02:25:02 UTC 2012
01:05 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix css'
01:04 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'fix css'
01:03 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'fix css'
00:59 LeslieCarr: ignoring cp1037 to cp1040 alarms for now as they are unused
00:45 LeslieCarr: rebooting cp1040
00:45 LeslieCarr: rebooting cp1039
00:43 LeslieCarr: rebooting cp1037

June 25

22:49 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Weekly MF deployment'
22:40 logmsgbot: maxsem synchronized php-1.20wmf6/extensions/MobileFrontend/ 'Weekly MF deployment'
22:36 maplebed: powercycling ms1002 - it's unresponsive to ssh and on the console though it does respond to a ping.
21:31 Jeff_Green: manual apache restart on srv265, srv277
21:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37345 - Request: Enable Ext:Collection on mk.wiki'
21:07 LeslieCarr: rebooting neon
21:01 hashar: Triggered several jobs on Jenkins to run tests on change that did not received their blame stick token
21:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37507 - Babel configuration for tl.wikipedia'
20:58 Jeff_Green: pushing out new redirects.conf adjusted for RT #3138
20:52 logmsgbot: reedy synchronized wmf-config/ 'Various site config bugs'
20:13 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Math/ 'Updating math to master'
20:12 logmsgbot: reedy synchronized php-1.20wmf5/extensions/Math/ 'Updating math to master'
20:07 logmsgbot: reedy synchronized php-1.20wmf6/extensions/WikiEditor/
20:06 logmsgbot: reedy synchronized php-1.20wmf5/extensions/WikiEditor/
19:17 LeslieCarr: rebooting unresponsive gallium
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf6
17:27 logmsgbot: reedy Finished syncing Wikimedia installation... : Take 2
16:43 logmsgbot: reedy Started syncing Wikimedia installation... : Take 2
16:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable EducationProgram on enwiki per request'
16:30 logmsgbot: reedy synchronized php-1.20wmf6/extensions/EducationProgram/ 'sync education programf iles'
16:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Scapping to rebuild message cache for 1.20wmf6
16:08 logmsgbot: reedy Started syncing Wikimedia installation... : Scapping to rebuild message cache for 1.20wmf6
16:03 logmsgbot: reedy synchronized php-1.20wmf6 'Syncing php-1.20wmf6'
15:43 Reedy: Copying php-1.20wmf6 from /tmp to NFS /home on fenari
13:52 Reedy: Killed php-1.20wmf4/cache/l10n from mediawiki-installation hosts
12:26 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37699) Chage logo on uzwiki'
09:38 mutante: so the several redirects for education->outreach requested to work by today look good now. RT-3138
09:30 mutante: apache-graceful-all to push out needed redirects for education
09:23 mutante: looking good. running sync-apache
09:20 mutante: creating dsh group "testwikipedia" with just srv193, creating sync-apache-test to just sync there...testing sync
02:49 Tim: configured mediawiki-commits to discard mails from gerrit pending resolution of the "implicit destination" issue
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jun 25 02:24:46 UTC 2012

June 24

18:21 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jun 24 02:25:23 UTC 2012

June 23

13:28 apergos: powrcycling srv288, swap death etc, some message to mgmt console but only the timestamp so couldn't see the issue, also couldn't get past the login prompt
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 23 02:24:11 UTC 2012

June 22

23:11 LeslieCarr: restarted apache on srv278
22:23 binasher: stopping mysql on es3, reseeding slave via innodb hotbackup of es1004
18:57 logmsgbot: preilly synchronized docroot/bits
18:36 LeslieCarr: removing 28790 bounce messages from exim queue on mchenry
16:50 Ryan_Lane: added a database account on db9/10 for read-only access to the gerrit database
12:06 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37700) update stewardwiki logo & favicon'
11:44 mutante: installing upgrades and kernel on pdf1, can reboot? (also needs puppetizing and precise reinstall)
10:46 mutante: installing security upgrades and kernel on bast1001 (still needs reboot, but dont break user sessions)
10:42 mutante: fenari upgrade - this included replace wikimedia-lvs-realserver 0.04 (using .../wikimedia-lvs-realserver_0.08
10:41 mutante: installing security upgrades on fenari
10:31 mutante: installing security upgrades on formey (gerrit)
08:49 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'gerrit 12569 Load transcode conf on -e /etc/wikimedia-transcoding (wmflabs change)'
08:40 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'gerrit 12568 Disable wgNoticeInfrastructure on beta cluster'
08:30 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'gerrit 12566 labs use the same wgCentralDBname on all wiki'
07:59 apergos: powercycled lvs1001, not pingable, nothing good from mgmt console, etc.
04:11 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 22 02:24:15 UTC 2012
01:00 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor 'VisualEditor updates'
00:37 LeslieCarr: restarting exim4 on mchenry with split_spool_directory = true
00:30 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Updated swift user config.'

June 21

23:11 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
23:07 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/VisualEditor.php
22:49 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
22:26 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
21:28 notpeter: restarting all lucene instances to direct logs to oxygen
21:16 binasher: deploying new mobile redirector to esams text squids
21:00 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor 'VisualEditor bugfixes'
20:34 logmsgbot: reedy synchronized wmf-config/
20:29 binasher: deployed new squid mobile redirector, now covers additional projects
20:27 logmsgbot: reedy synchronized wmf-config/
20:10 logmsgbot: hashar: on gallium, cloning mediawiki/extensions.git to /var/lib/jenkins/jobs/MediaWiki-Extensions-Fetching/workspace
19:25 mark: Restarted 3 queue runners as exim -qff &
19:15 logmsgbot: catrope Finished syncing Wikimedia installation... : VisualEditor updates
18:57 logmsgbot: catrope Started syncing Wikimedia installation... : VisualEditor updates
18:50 mark: Started 5 exim queue runners on mchenry with exim -qqff &
18:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Point $wgVisualEditorParsoidURL to cadmium'
18:36 logmsgbot: catrope synchronized php-1.20wmf5/includes/OutputPage.php 'Core patches for VisualEditor deploy'
18:36 logmsgbot: catrope synchronized php-1.20wmf5/resources/Resources.php 'Core patches for VisualEditor deploy'
18:36 logmsgbot: catrope synchronized php-1.20wmf5/resources/mediawiki.page/mediawiki.page.watch.ajax.js 'Core patches for VisualEditor deploy'
17:51 notpeter: restarting puppet on brewster
15:25 notpeter: stopping puppet on brewster
14:11 paravoid: powercycling srv272, unreachable due to load spike
05:30 binasher: clearing mobile varnish cache - my friend can't expand some article categories on his iphone after rebooting and clearing cache
04:41 logmsgbot: catrope synchronized php-1.20wmf4/extensions/LastModified/modules/lastmodified.js
04:40 logmsgbot: catrope synchronized php-1.20wmf5/extensions/LastModified/modules/lastmodified.js
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Thu Jun 21 02:25:40 UTC 2012
00:10 binasher: stopped puppet on cp1020 until tomorrow - testing new build of the squid mobile redirector on one server until tomorrow

June 20

23:29 logmsgbot: kaldari synchronized php-1.20wmf5/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'updating clicktracking for LastModified and E3Experiments exts'
21:52 Reedy: pointed /usr/local/apache/common/php at /usr/local/apache/common/php-1.20wmf5 on mediawiki-installation
21:49 LeslieCarr: see RT3170 for more details on above change and mchenry pain
21:48 LeslieCarr: freezing many bounce messages on mchenry (all older than 2400 minutes)
21:04 LeslieCarr: replaced srv268 with srv245 in memcached list
21:04 logmsgbot: lcarr synchronized wmf-config/mc.php 'removed broken srv268'
21:03 paravoid: powercycling srv268; unreachable due to load spike
18:31 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
18:23 LeslieCarr: reloading mr1-pmtpa for sw upgrade (fixing a cpu bug)
18:18 logmsgbot: reedy synchronized wikiversions.dat
18:17 logmsgbot: reedy synchronized wikiversions.cdb
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of pedias to 1.20wmf5
18:03 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'test with disable caching on'
18:02 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'test with disable caching on'
17:59 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'testing with disable caching off'
17:58 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'testing with disable caching off'
17:42 logmsgbot: preilly synchronized docroot/bits
17:11 logmsgbot: preilly synchronized wmf-config/mobile.php 'add Grameenphone Bangladesh'
17:08 logmsgbot: preilly synchronized wmf-config/mobile.php 'add telenor'
14:00 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
13:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on enwiki *gulp*'
13:41 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/ 'Push out master EP'
13:37 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/ 'Push out master EP'
13:19 Reedy: Created EducationProgram database tables on enwiki
12:40 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touching InitialiseSettings.php to refresh cache'
12:39 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 37740) raise account throttle for an edit marathon'
09:43 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37457) viwikibooks can import from fr/it wikibooks'
09:40 logmsgbot: hashar synchronized wmf-config/mobile.php '$wgMobileResourceVersion does not exist anymore'
09:25 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37327) Configure chr.wikipedia site logo'
04:35 Tim: on nickel: there were data sources for both "Apaches 8 CPU" and "Application servers", these were getting the same cluster name from the remote gmonds, and so different threads in gmetad were trying to write to the same summary files. Fixed temporarily, will fix in puppet shortly
04:26 Tim: on nickel: ran gmetad with -d3, it spews errors when trying to write to the faulty summary info files
04:20 Tim: on nickel: restarting gmetad
04:19 Tim: on srv258: started gmond
04:12 Tim: experimentally stopping gmond on srv258 to check for effects on oscillating appserver stats
03:23 Tim: on fenari, queueing refreshLinks jobs for some 2.8M commons image description pages that use location templates
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Wed Jun 20 02:48:41 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Wed Jun 20 02:26:26 UTC 2012
02:24 Tim: started socat for /var/log/mw/fatal.log on fenari
01:23 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php

June 19

23:37 paravoid: temporarily adding wikimedia.org, wikipedia.org etc. to sodium's /etc/exim4/defer_domains
23:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'bug 37611, plan B'
22:55 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'bug 37611'
21:07 Jeff_Green: deployed a hacked up exim conf on sodium to block a mail ddos, puppet disabled there too
20:37 logmsgbot: mlitn synchronized php-1.20wmf5/extensions/ArticleFeedbackv5 'desc'
19:45 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
19:28 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
19:13 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Enable AFTv4 on testwiki'
18:06 maplebed: failed out ms-be5 after failed ssd test
14:50 RobH: updating dns
14:40 RobH: db14 is out of rotation, shutting down to make room for new es servers in rack
14:03 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Syncing misc changes
11:03 Ryan_Lane: adding IPs for virt6-8
10:11 logmsgbot: nikerabbit Finished syncing Wikimedia installation... : Updating TranslationNotifications
10:05 hashar: TranslationNotifications extension updated by Nikerabbit!
09:56 logmsgbot: nikerabbit Started syncing Wikimedia installation... : Updating TranslationNotifications
09:41 hashar: updating TranslationNotifications extension with NikeRabbit
09:04 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Merge I8572d5f4 to fix workflowstates in Translate'
07:27 apergos: reboot snapshot1, package and kernel updates
07:14 apergos: reboot snapshot2, package and kernel updates
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Tue Jun 19 02:47:42 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Tue Jun 19 02:24:46 UTC 2012
01:34 logmsgbot: tstarling Finished syncing Wikimedia installation... :
00:53 logmsgbot: tstarling Started syncing Wikimedia installation... :
00:43 Tim: put DolphinBrowser files in docroot/bits (from preilly's Ie3fefec6) and now running scap

June 18

22:31 K4-713: updated production civicrm to r1814
22:26 logmsgbot: aaron synchronized php-1.20wmf5/extensions/FlaggedRevs 'deployed e53310f548cf3f3e4f1ddfa10f5efd0eff06eeec'
22:17 maplebed: rebooting es1002 to look at the raid setup
20:43 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'update to remove bad code'
20:43 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'update to remove bad code'
19:11 hashar: updating several Jenkins plugins
19:06 RobH: updating dns
18:52 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/
18:51 logmsgbot: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/
18:23 logmsgbot: reedy synchronized wikiversions.dat
18:15 logmsgbot: reedy synchronized wikiversions.cdb 'sync using sync-file'
18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf5
16:30 mutante: installing package upgrades on sodium
16:29 mutante: restarting lighttpd on sodium - redirecting mediawiki-cvs list page
16:05 binasher: rebooting es1001
15:59 mutante: there have been no archives, so that should be it. there may be another issue in BZ 37690 but should be unchanged by renaming
15:58 mutante: copied full config/users/passes from mediawiki-cvs to mediawiki-commits, merged redirects, added old list name to acceptable_aliases in recipient filters
15:52 mutante: making the mailing list switch. mediawiki-cvs -> mediawiki-commits
15:49 Ryan_Lane: assigned service IPs for labs-ns0/labs-ns1
15:25 binasher: rebooting es1002 and es1003
15:11 Ryan_Lane: added virt1000 as a secondary ldap server for labsconsole
15:08 Ryan_Lane: testing gerrit config with multiple ldap servers
14:52 hashar: hume is out of disk space again. Probably the wmf branches taking toooo much space
14:52 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37662) change wgUploadNavigationUrl @ dawiki'
14:21 mutante: creating new list MediaWiki-commits, not in use yet, but will replace outdated -cvs list soon
14:18 apergos: reboot snapshot3, package and kerne updates
14:13 apergos: rebooting snapshot4, kernel and other updates
09:31 Ryan_Lane: added virt1000 to dns, using titanium misc server
08:17 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php ' (bug 37672) Use odf on collection for ml projects '
06:03 apergos: powercycling db1047
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Mon Jun 18 02:47:49 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jun 18 02:25:09 UTC 2012

June 17

02:45 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sun Jun 17 02:45:15 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jun 17 02:23:01 UTC 2012

June 16

02:49 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sat Jun 16 02:49:55 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 16 02:24:56 UTC 2012
00:34 paravoid: esams SSL should be back up
00:25 paravoid: SSL ipv6 access logs disabled; force running puppet and rm'ing access.logs on esams
00:08 paravoid: esams SSL outage, working on it

June 15

20:56 LeslieCarr: attaching asw-c1-pmtpa to asw-d-pmtpa ring
20:41 RobHalsell: updating dns for mc1-mc16 mgmt
20:07 RobH: mobing asw and msw-d3-sdtpa from single to dual power again, got sidetracked
19:59 RobH: mobing asw and msw-d3-sdtpa from single to dual power
19:24 RobH: updating dns for ms-be12 mgmt
19:18 logmsgbot: aaron synchronized php-1.20wmf5 'deployed 2755f255e45b53a083207d69c3e2d9fca62a3a1c'
19:15 paravoid: virt0: modify pdns.conf to listen on the old IP; temporarily disable puppet
19:15 paravoid: adding pre-renumbering virt0's IP back on eth1; doing policy routing to work out multihoming
18:54 RobH: updating dns for educacao redirect
18:53 LeslieCarr: deactivated rpf-filter on cr1-sdtpa and cr2-pmtpa temporarily for virt0
18:53 Ryan_Lane: doing a git pull for OpenStackManager on virt0
18:33 RobH: morebots, dont leave me again!
15:50 RobH: updating dns for new cisco machines
14:34 hashar: hume: 5.0G 5.0G 68K 100% /usr/local/apache
14:33 hashar: hume is out of disk space
14:32 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 34866) Change wgLanguageCode of several wikis to be renamed'
14:21 logmsgbot: hashar synchronized phpunit.xml
14:09 logmsgbot: hashar synchronized tests
13:39 Ryan_Lane: adding labs-ns0 and labs-ns1 dns entries
11:46 mark: csw1-esams.wikimedia.org line card 2 in trouble, power cycled it
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Fri Jun 15 02:48:46 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 15 02:26:17 UTC 2012
00:00 binasher: rebooting / upgrading kernel on es1003 first

June 14

23:58 binasher: stopping mysql on es1003 and disabled notifications. going to convert to innodb via hotbackup of es1004 for testing
23:23 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning LastModified on for en.wiki'
23:10 logmsgbot: kaldari Finished syncing Wikimedia installation... :
23:00 logmsgbot: kaldari Started syncing Wikimedia installation... :
22:36 logmsgbot: kaldari synchronized php-1.20wmf5/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'sycing E3Experiment js for wmf5'
22:34 logmsgbot: kaldari Started syncing Wikimedia installation... :
22:07 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:57 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:16 RobH: updating dns for new mgmt ips and move of scs
20:37 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:33 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:29 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update to MoodBar
20:07 logmsgbot: bsitu Started syncing Wikimedia installation... : Update to MoodBar
19:51 logmsgbot: py synchronized wmf-config/db.php 're-add db43 to s6 pool after kern upgrade'
19:28 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update to PageTriage
19:10 notpeter: rebooting db43 for kernel upgrading
19:08 logmsgbot: asher synchronized wmf-config/db.php 'returning db22'
18:58 logmsgbot: bsitu Started syncing Wikimedia installation... : Update to PageTriage
18:52 RobH: unracking and decommissioning db21 and db23
18:46 RobH: db22 relocated, powering up
18:43 RobH: db22 relocating
18:41 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db22 for hw move'
18:39 pgehres: re-enabled donation queue consumption after all updates
18:36 notpeter: pushing new dns zone file (minor change)
18:33 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
18:33 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
18:32 notpeter: new master log and pos for s6 MASTER_LOG_FILE='db47-bin.000230', MASTER_LOG_POS=876357616
18:27 logmsgbot: py synchronized wmf-config/db.php 'completed master switch for s6'
18:24 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
18:23 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
18:23 logmsgbot: py synchronized wmf-config/db.php 'switching master for s6 to db50'
18:11 Jeff_Green: erzurumi dist-upgrade & reboot [up 633 days]
18:02 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'update for ie7'
18:01 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'update for ie7'
18:00 Jeff_Green: aluminium/db1008 dist-upgrade & reboot
17:57 pgehres: disabled queue consumption on aluminum for dist-upgrade
17:03 mark: Copied udp-filter package from lucid-wikimedia to precise-wikimedia (but do as I say and rebuild, not as I do...)
16:40 binasher: running pagetriage_page schemea changes on enwiki and testwiki via osc (https://gerrit.wikimedia.org/r/#/c/11014/1/sql/PageTriagePagePatch.sql)
16:22 Jeff_Green: hume dist-upgrade & reboot
16:08 Jeff_Green: loudon dist-upgrade & reboot
16:04 mark: Manually disabled GRO on amslvs3/4 eth0
16:03 logmsgbot: reedy synchronized wmf-config/ 'Enable EP on test2wiki'
16:01 Ryan_Lane: restarting nova-compute on virt5
15:58 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
15:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisationcache for EP
15:49 Jeff_Green: grosley dist-upgrade & reboot
15:40 Jeff_Green: silicon dist-upgrade and reboot
15:29 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisationcache for EP
15:28 mark: Starting dist-upgrade of manutius to Precise
15:27 Ryan_Lane: to satisfy mark's pedantry that's lucid-wikimedia
15:27 mark: Ryan needs coffee
15:26 Ryan_Lane: specifically wikimedia-lucid repo
15:26 Ryan_Lane: added adminbot 1.2 to repo
15:00 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
14:30 mark: Reinstalled stat1 with Ubuntu Precise
11:21 pp-pdf1: updated mwlib.rl to 0.12.12
11:21 pp-pdf2: updated mwlib.rl to 0.12.12
11:21 pp-pdf3: updated mwlib.rl to 0.12.12
02:46 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Thu Jun 14 02:46:00 UTC 2012
02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Thu Jun 14 02:23:26 UTC 2012
00:01 logmsgbot_: tstarling synchronized php-1.20wmf4/includes/DefaultSettings.php

June 13

23:39 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Deploying MF fix'
23:37 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Deploying MF fix'
21:49 logmsgbot_: reedy synchronized wmf-config/ 'More changes from gerrit'
21:35 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bringing in numerous changes merged via gerrit'
21:32 logmsgbot_: lcarr synchronized wmf-config/mc.php 'replacing broken srv203 with working srv250'
21:31 LeslieCarr: replacing srv203 with srv250 in memcache rotation since srv203 is broken
21:20 logmsgbot_: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'deployed 82742bccf3b5f2da0d5df05630eb31978afbbce1'
19:57 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php
19:51 Jeff_Green: payments cluster dist-upgrades & reboots
19:43 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Added debug log.'
19:37 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php
19:29 binasher: drop table enwiki.trackbacks
19:28 binasher: converted enwiki.interwiki to innodb
19:27 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php 'temporary logging code.'
19:26 binasher: drop table enwiki.exlogging
18:35 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non wikipedia to 1.20wmf5
18:33 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote to 1.20wmf5
18:31 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.20wmf5
18:30 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity to 1.20wmf5
18:22 logmsgbot_: reedy synchronized php-1.20wmf5/extensions/Vector/
18:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special.dblist wikis to 1.20wmf5
18:14 logmsgbot_: aaron synchronized php-1.20wmf5/extensions/FlaggedRevs 'deployed 537bb248bb93948844f195014227512f169a439b'
18:06 Jeff_Green: db1025 dist-upgrade & reboot
17:37 logmsgbot_: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'changes for zero needed for carrier testing'
17:36 logmsgbot_: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'changes for zero needed for carrier testing'
17:35 Ryan_Lane: restarting opendj on virt0 again...
17:29 Ryan_Lane: restarting opendj again on virt0
17:24 Ryan_Lane: restarting gerrit
17:23 LeslieCarr: rebooting stat1 for wipe and reinstall into precise
16:59 Ryan_Lane: restarting opendj on virt0
16:49 Ryan_Lane: restarting mysql on virt0 with correct bind address
16:46 LeslieCarr: changing virt0's ip address and vlan
16:38 cmjohnson1: shutting down search32 to replace main board
16:24 cmjohnson1: sq48 powercycled
16:19 mutante: shut down sq33
16:18 cmjohnson1: performing hard reset on sq33
16:12 mutante: adding gerrit@wikimedia.org to accepted nonmembers of mediawiki-cvs list
15:29 mark: Unstuck torrus
14:46 Ryan_Lane: lowering ttl for virt0
14:17 Jeff_Green: storage3 dist-upgrade and reboot
12:31 mutante: backing up wikitech dir locally on linode instance
09:24 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
09:08 hashar: finished deploying my wmflabs related change. mediawiki-config is now at commit c0baf3e
09:07 logmsgbot_: hashar synchronized wmf-config
09:06 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-wmflabs.php
09:05 logmsgbot_: hashar synchronized wmf-config/throttle.php
09:05 logmsgbot_: hashar synchronized wmf-config/mobile-wmflabs.php
09:05 logmsgbot_: hashar synchronized wmf-config/mobile.php
08:47 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37006 - fawiki: add Book namespace + aliases'
08:45 hashar: reverted '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource' --> used the wrong configuration setting.
08:37 mutante: installing samba-common-bin, smbclient package upgrades on tridge
08:33 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
08:28 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
08:24 hashar: deploying several changes made to mediawiki-config gerrit changes 11034 11035 9131 11036 11037 9132 9136 and 9237
02:52 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Wed Jun 13 02:52:38 UTC 2012
02:27 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Wed Jun 13 02:27:37 UTC 2012

June 12

23:58 maplebed: started swift container listing loop to compare purge timing when listings are fresh
23:16 logmsgbot_: preilly synchronized php-1.20wmf5/extensions/MobileFrontend/ 'try again'
23:15 logmsgbot_: preilly synchronized php-1.20wmf4/extensions/MobileFrontend/ 'try again'
22:58 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Updating MobileFrontend'
22:53 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess'
22:52 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Updating MobileFrontend'
22:50 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess'
22:49 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Updating MobileFrontend'
20:28 RoanKattouw: Correction: the /usr/local/apache filesystem is full on hume, the root fs is not
20:27 RoanKattouw: hume has a full disk
20:23 RoanKattouw: Fixed ownership of php-1.20wmf{4,5}/cache/l10n , should be l10nupdate:wikidev . The wmf4 copy had wrong ownership causing rebuildLocalisationCache.php to fail for shell users (e.g. from scap)
19:40 logmsgbot_: mlitn Finished syncing Wikimedia installation... :
19:13 logmsgbot_: mlitn Started syncing Wikimedia installation... :
18:09 logmsgbot_: py synchronized wmf-config/db.php 're-adding db25 to pool after kern upgares'
18:02 notpeter: halting owa3 for repairs
16:44 logmsgbot_: py synchronized wmf-config/db.php 'removing db25 from pools for kern upgares'
16:27 logmsgbot_: py synchronized wmf-config/db.php 're-adding dbs 33, 34, 36, 50, 55, 56 to pools after kern upgares'
15:55 RobH: virt1001 and virt1002 rebooting, disregard
15:48 cmjohnson1_: shutting down search32 to run a diagnostic test
15:45 binasher: migrating enwiki.bv2009_edits (?) to innodb
15:41 binasher: migrating enwiki.moodbar_feedback to innodb
15:39 binasher: migrating enwiki.aft_article_filter_count to innodb
15:33 logmsgbot_: py synchronized wmf-config/db.php 'removing dbs 33, 34, 36, 50, 55, 56 from pools for kern upgares'
15:31 notpeter: doing another round of DB kernel upgrades
15:25 logmsgbot_: py synchronized wmf-config/db.php
15:14 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'reenabling mysql pcache'
15:00 binasher: rebooting db40
14:52 binasher: set innodb_max_dirty_pages_pct = 0 on db40 in prep for shutdown
14:48 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'disabling mysql parsercache (db40) in order to perform maintenance'
14:45 notpeter: putting kern-upgraded DBs back into pools
14:44 binasher: resumed replication on es3, es1002 after cluster23 sync completed
13:41 Jeff_Green: added awight to fr-tech@wikimedia.org email alias
12:06 mutante: gerrit create-project --name=mediawiki/extensions/UniversalLanguageSelector --parent=mediawiki/extensions
11:28 mutante: powercycling downed srv232 (also cause for check_all_memcached crit)
11:08 mutante: powercycled mw1042 to check for hardware issues and fscked. appears to be just unused (though down since ~3d like mw1071 per nagios)
10:37 mutante: test to show linking from !log via SAL to RT: RT:3100 (before/without template)
10:31 hashar: incubatorwiki.translate_messageindex on db39 uses MyISAM engine. See RT #3100
10:25 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 , take 2 - Install Translate extension on be.wikimedia.org'
10:24 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 , take 2 - Install Translate extension on be.wikimedia.org'
10:23 hashar: Compared translate% tables schema on bewikimedia with incubatorwiki. diff prove they are the same so the schema changes made early are successful.
10:17 hashar: bewikimedia (db39) : dropped tables translate_tmf , translate_tms and translate_tmt I have incorrectly added
09:59 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'revert translate extension on be.wikimedia.org, need DB update'
09:58 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 - Install Translate extension on be.wikimedia.org'
06:24 apergos: db1047 looks like the aft_article_filter_count is missing a few rows compared to the master (after replication caught up), presumably this is a side effect of the repair, have pinged binasher for help, leaving everything running and hope it's tolerable error for a day
02:34 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Tue Jun 12 02:34:49 UTC 2012
02:25 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Tue Jun 12 02:25:21 UTC 2012
01:35 binasher: passes the dba mantel to notpeter
01:22 notpeter: removing one slave from each db shard to upgrade/restart
00:24 binasher: shutdown mysql on es3. stopped slaving on es1002, rsyncing cluster23 tables to es3
00:09 binasher: pointed es3 to MASTER_LOG_FILE='es1-bin.000788', MASTER_LOG_POS=453509865
00:05 binasher: es3:~# rm -rf /usr/local/mysql*

June 11

23:54 logmsgbot_: asher synchronized wmf-config/db.php 'fully commenting out es3'
23:52 logmsgbot_: asher synchronized wmf-config/db.php 'making es1 the master for blobs cluster 23'
23:51 binasher: es1 is the new master, now switching mw conf
23:48 binasher: preparing to switch es master to es1
23:31 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 27706 - enable RSS extension on uawikimedia'
23:12 logmsgbot_: reedy Finished syncing Wikimedia installation... :
21:49 Reedy: Applied PageTriage schema updates to testwiki and enwiki
20:54 logmsgbot_: reedy Started syncing Wikimedia installation... :
20:07 logmsgbot_: reedy synchronized php-1.20wmf5/
19:20 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf5
18:54 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf5 also
18:52 logmsgbot_: reedy synchronized php-1.20wmf5/ 'Now we have some more space...'
18:50 logmsgbot_: reedy synchronized php-1.20wmf3/cache/l10n/ 'Kill l10ncache for php-1.20wmf3 as its not needed'
18:45 logmsgbot_: reedy synchronized php-1.20wmf2/cache/l10n/ 'Kill l10ncache for php-1.20wmf2 as its not needed'
18:44 logmsgbot_: reedy synchronized php-1.20wmf5/ 'Scap is taking an age, just ensure deployment files are in sync'
18:42 binasher: resuming coversion of es1004 to innodb, using compact row format after testing dynamic and compressed
18:03 logmsgbot_: reedy Started syncing Wikimedia installation... : Consistency
17:56 Ryan_Lane: enabling TitleBlacklist on labsconsole
17:28 notpeter: moving /usr/local/apache to /a/apche with symbolic link on searchidx1001 as a temp measure until it can be reimaged
16:58 logmsgbot_: reedy Started syncing Wikimedia installation... : Running scap to ensure consistency
16:53 logmsgbot_: reedy synchronized php-1.20wmf5/cache/l10n/
16:28 mutante: installing security upgrades on sodium
16:19 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
16:17 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.20wmf5.php
16:02 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
15:55 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
15:54 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
15:46 mutante: hume /usr/local/apache is out of disk (just 5GB but more branches now). (LVM vg "tank" lv "tank-apache" ) but no free extents. could take from /archive but unsure about shrinking the xfs.
15:35 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
15:30 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf5
15:28 logmsgbot_: reedy synchronized php-1.20wmf5/
15:20 Reedy: running sync-dir php-1.20wmf5
15:13 Reedy: Copying checkout of 1.20wmf5 onto NFS
14:53 mutante: running puppet on stat1. installs plotting packages
09:56 apergos: shut down mysqld on db1047, reparing tables
06:54 pp-pdf2: upgraded mwlib to 0.13.8
06:54 pp-pdf3: upgraded mwlib to 0.13.8
06:54 pp-pdf1: upgraded mwlib to 0.13.8
02:24 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Mon Jun 11 02:24:06 UTC 2012

June 10

02:22 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sun Jun 10 02:22:33 UTC 2012

June 9

07:45 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sat Jun 9 07:45:17 UTC 2012
02:35 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sat Jun 9 02:35:29 UTC 2012
02:25 Reedy: Running LU manually
02:14 Reedy: Cleared a bit of space on fenari by deleting checkouts from /tmp
02:00 logmsgbot_: LocalisationUpdate failed: git pull of core failed
01:51 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:50 logmsgbot_: reedy synchronized wikimedia.dblist
01:49 Reedy: fenari has a full /
01:49 logmsgbot_: reedy synchronized all.dblist

June 8

20:32 Reedy: Updated php to point to php-1.20wmf4 rather than php-1.20wmf3
20:09 logmsgbot_: reedy synchronized wikimedia.dblist
20:08 logmsgbot_: reedy synchronized all.dblist
20:04 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 2 wikimedia wikis to 1.20wmf4
19:57 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuilding localisation cache for message updates
19:39 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for message updates
18:47 logmsgbot_: reedy synchronized php-1.20wmf4/languages/messages/ 'Pushing out updated files upon siebrands request'
16:30 cmjohnson1: shutting down search32 to swap DIMM around
10:19 notpeter: ganglia down, restarting apache on nickel.
09:32 notpeter: stopping indexing on searchidx1001 to re-copy to searchidx2
08:45 notpeter: reimaging searchidx2 with correct partitioning
06:50 Tim: on cp1001: disabled HTCP plugin in gmond for testing, seems to work so I will disable it properly
03:55 Tim: disabled LastModified extension due to overload on cp1005
03:51 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
03:18 Tim: restarting squid on cp1005, maybe out of FDs or something, cachemgr shows exactly 1000 open connections to 10.2.1.1
03:08 Tim: stopped gmond on cp1001 with kill -STOP for memory leak debugging
03:02 Tim: on cp1002: killed gmond again, it was leaking memory again, already up to 27GB in the few minutes since I restarted it
03:01 Tim: on fenari: copied *.text and *.upload from /home/wikipedia/conf/squid/generated/clusters to /etc/dsh/group
02:56 Tim: cp1001: same as on cp1002, restarted gmond
02:53 Tim: on cp1002: killed gmond, which was using 100% CPU and 23GB RSS. Restarting squid which had died
02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Fri Jun 8 02:23:09 UTC 2012
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Fri Jun 8 02:14:55 UTC 2012

June 7

23:32 Tim: deploying varnish configuration change https://gerrit.wikimedia.org/r/#/c/10672/ on cp1041, cp1042, cp1043, cp1044
20:46 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/LastModified 'syncing LastModified extension'
20:32 logmsgbot_: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on LastModified and E3Experiemnts for en.wiki'
19:54 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'syncing js file for E3Experiments'
18:58 cmjohnson1_: shutting down search32 for testing
16:52 Ryan_Lane: added ldap automount entries for /public/datasets and /public/keys
02:19 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Thu Jun 7 02:19:52 UTC 2012
02:11 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Thu Jun 7 02:11:49 UTC 2012

June 6

22:29 logmsgbot_: aaron synchronized wmf-config/swift.php 'deployed 7dc77e431310580da0dbd368b8b290a293e3ee21'
19:55 cmjohnson1: shutting down search32 to reseat DIMM B2
19:18 cmjohnson1: shutting down storage3
18:05 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaining wikis to 1.20wmf4
17:02 Jeff_Green: mailman 'site' password changed per RT 3039
11:12 mark: Added AAAA record to mobile
10:38 mark: Wikipedia is IPv6-enabled.
10:37 mark: Added AAAA records to all non-mobile wiki projects
10:19 mark: Added AAAA record to bits.wikimedia.org
10:02 Ryan_Lane: repooling ssl3001
10:00 mark: Added AAAA record to upload.wikimedia.org
09:51 Ryan_Lane: depooling ssl3001
08:41 mark: Converted bits.wikimedia.org into a direct geodns record, removed the old bits -> bits-geo CNAME
07:53 mark: Converted geoiplookup.wikimedia.org into a separate, IPv4-only geodns record
07:37 pp-pdf1: installed tmpreaper cronjob for /home/pp/mathcache directory
07:37 pp-pdf2: installed tmpreaper cronjob for /home/pp/mathcache directory
07:37 pp-pdf3: installed tmpreaper cronjob for /home/pp/mathcache directory
02:35 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Wed Jun 6 02:35:47 UTC 2012
02:11 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Wed Jun 6 02:10:58 UTC 2012

June 5

22:37 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Undo temporary woff whitelisting'
22:34 logmsgbot_: aaron synchronized php-1.20wmf4/includes/filerepo/file/LocalFile.php 'deployed 4791e3d25aebe9643a7cea91f2eb49e6b54593c5'
22:33 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Temporarily allow uploading woff files on slwikisource'
22:13 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/PageTriage/modules/ext.pageTriage.models/ext.pageTriage.article.js 'updating default PageTriage filters'
20:56 logmsgbot_: mmullie Finished syncing Wikimedia installation... :
20:16 logmsgbot_: mmullie Started syncing Wikimedia installation... :
19:49 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Fix notice in MobileFrontend config'
19:45 pp-pdf2: updated mwlib to 0.13.7-1-g827780b
19:45 pp-pdf1: updated mwlib to 0.13.7-1-g827780b
19:45 pp-pdf3: updated mwlib to 0.13.7-1-g827780b
19:45 pp-pdf2: updated mwlib to 0.13.7-1-g827780b
18:45 notpeter: starting innobackupex dump from blondel to bellin
18:44 notpeter: starting indexing on new searchidx2
18:39 mark: Added static routes 2002::/16 and 2001::/32 for 6to4 and teredo on the Tampa routers; these are redistributed in OSPF to eqiad
18:29 notpeter: restart indexing on searchidex1001
18:20 mark: Replaced static LVS IPv6 routes with correct next-hops on cr1-eqiad and cr2-eqiad
18:09 mark: Redistributing static routes in OSPF on cr1-eqiad and cr2-eqiad
18:04 paravoid: rebooting capella to make sure things work after a reboot
17:53 mark: Redistributed statics in OSPF3 on csw1-esams
17:14 logmsgbot_: aaron synchronized php-1.20wmf4/includes/logging/LogEventsList.php 'deployed d9f146ac42f2884e76390d6bc979eb10032adf7f'
17:09 mark: Added uRPF exception for 6to4 traffic on all routers
17:05 jeremyb: (UTC) 23:42:14 <binasher> !log re-enabled es4 monitoring. its currently our only es server without any tables marked as crashed / needing recovery, myisam recovery has been absent for all systems since the ms servers were migrated off of in nov 2011. (Sum of human knowledge * Renyi entropy = ES)
16:52 mark: Pooled ssl1001
15:44 logmsgbot_: asher synchronized wmf-config/db.php 'returning es2 to service'
15:25 paravoid: rebooting lvs1004 and reinstalling with precise
15:17 binasher: rebooting es2 for kernel + mysql upgrade
15:16 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es2 for kernel+mysql upgrades'
14:56 paravoid: rebooting amslvs3 & amslvs4 to reinstall with precise
14:20 paravoid: rebooting lvs1006 to reinstall with precise
13:51 cmjohnson1: shutting down bellin to replace main board
13:49 notpeter: reimaging db1042
13:40 paravoid: rebooting lvs1005 to reinstall with precise
12:59 paravoid: rebooting lvs2 to reinstall with precise
12:47 Ryan_Lane: changing capella's subnet in DNS
12:10 Ryan_Lane: rebuilding capella as precise
10:01 logmsgbot_: asher synchronized wmf-config/db.php 'putting es1 in production'
09:53 notpeter: cancel that, it's mid-cron. will do later
09:52 notpeter: stopping indexing on searchidx1001 to rsync to searchidx2
09:35 binasher: rebooting es1 for kernel+mysql upgrade. dont need to pull from db.php because it was never correctly added or queried?
09:14 mark: Built PyBal 1.01 for precise, and included it in the precise-wikimedia APT repository
08:45 binasher: restarted mysql on es1004 with default innodb file format as barracuda
08:31 notpeter: reimaging searchidx2
02:36 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Tue Jun 5 02:36:44 UTC 2012
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Tue Jun 5 02:14:17 UTC 2012

June 4

23:23 logmsgbot_: asher synchronized wmf-config/db.php 'returning es4'
22:49 binasher: started an experiment on es1004 - altering all es tables from myisam to innodb one at a time with file_per_table enabled
22:39 binasher: stopping mysql on es4. all tables marked as having repair fails are in cluster22, resyncing just those from es1002
21:52 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es4 again'
21:43 logmsgbot_: asher synchronized wmf-config/db.php 'returning es4 to service'
21:16 Ryan_Lane: restarting nginx on all ssl boxes again
21:07 Ryan_Lane: force running puppet on all ssl hosts again
21:04 Ryan_Lane: repooling ssl1, ssl1001, ssl3001
21:03 Ryan_Lane: restarting nginx on all ssl hosts
20:23 binasher: rebooted es4
20:18 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es4 for post-crash upgrade'
19:55 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/PageTriage/PageTriage.hooks.php 'syncing PageTriage.hooks.php'
19:32 Ryan_Lane: force running puppet on ssl servers
18:22 Reedy: Nuked php-1.20wmf4 on mw64 then ran sync-common. Seems to have dealt with the permission errors
18:11 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf4
16:48 binasher: upgraded kernel on db1047 / analytics
16:09 Ryan_Lane: restarting ircecho on manganese
15:21 paravoid: reinstalling lvs1 with precise
15:13 mark: Added new IPv6 LVS prefixes to all routers for uRPF filters; BGP import filters still need adjusting for dual-family sessions
15:08 cmjohnson1: physically power cycling lvs1
15:02 Ryan_Lane: depooling ssl1001 and ssl3001
14:55 Ryan_Lane: disabling puppet on all ssl hosts
13:27 mark: Changed upload.esams.wikimedia.org CNAME to upload-lb.esams, effectively disabling the IPv6 selective answer script
12:23 mark: Upgrading wikimedia-lvs-realserver to version 0.08 across the cluster (by Puppet)
12:18 Ryan_Lane: depooling ssl1
11:32 mark: Copied wikimedia-lvs-realserver 0.08 from APT distribution precise-wikimedia to lucid-wikimedia
02:38 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon Jun 4 02:38:15 UTC 2012
02:14 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Mon Jun 4 02:14:45 UTC 2012

June 3

15:45 paravoid: aborting lvs1 install, partition map is not ready; putting it back to production as-is
15:31 paravoid: reinstalling lvs1 with precise
15:10 RobH: torrus failed to refresh via puppet (failed refresh takes too long) so manually running the refresh/rebuild command as puppet copied the updates to the system
14:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37237 - Change Wikisource namespace for Tamil wikisource'
14:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 37211 - Set $wgUseCombinedLoginLink = false'
14:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37294 - Add English Wikibooks as import source at Vietnamese Wikibooks'
14:07 logmsgbot: midom synchronized wmf-config/db.php
13:10 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '59753a9 (allow bureaucrats on frwiki to add+remove accountcreator group'
13:09 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production) -- was not correctly deployed earlier'
13:02 logmsgbot: midom synchronized wmf-config/db.php
10:18 hashar: mw64: rsync: write failed on "/apache/common-local/wmf-config/CommonSettings.php": No space left on device (28)
10:17 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production)'
09:16 notpeter: pushing new zone files. only minor changes
02:35 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun Jun 3 02:35:31 UTC 2012
02:13 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sun Jun 3 02:13:39 UTC 2012

June 2

15:37 hashar: We ran out of beer, see bug 37307
15:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily enable $wmgReduceStartupExpiry on testwiki for Berlin tutorial'
14:12 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
14:10 logmsgbot: hashar synchronized wmf-config/ext-wmflabs.php
14:10 hashar: deploying some nasty configuration changes in wmf-config
14:10 logmsgbot: hashar synchronized wmf-config/ext-pmtpa.php
14:04 logmsgbot: reedy synchronized wmf-config/proofreadpage.php 'Default proofreadpage-showheaders to 1 on enwikisource and svnwikisource'
13:46 mutante: rebuilding archives for fd-advisorygroup mailing list
13:21 RobHalsell: updated quotas on labstore1 for publicdata-proect
08:57 logmsgbot: reedy synchronized wmf-config/ 'https://gerrit.wikimedia.org/r/#/c/9717/'
02:36 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat Jun 2 02:36:20 UTC 2012
02:14 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sat Jun 2 02:14:25 UTC 2012

June 1

20:54 logmsgbot: reedy synchronized php-1.20wmf4/includes/specials/SpecialRecentchanges.php
20:53 logmsgbot: reedy synchronized php-1.20wmf4/includes/SpecialPage.php
20:28 logmsgbot: reedy synchronized php-1.20wmf4/extensions/ExtensionDistributor/
20:27 logmsgbot: reedy synchronized php-1.20wmf3/extensions/ExtensionDistributor/
19:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36805 - Enable NewUserMessage extension on mrwiki and mrwikisource'
19:34 logmsgbot: reedy synchronized wmf-config/ 'Fix some typos'
19:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36965 - Please setup Collection extension on Telugu Wikipedia'
19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37027 - Install Collection extension in Hebrew Wiktionary'
19:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enabling collection on tawikis'
19:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enabling collection on tawikis'
17:56 logmsgbot: reedy synchronized wmf-config/
16:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Testing
16:23 logmsgbot: reedy Started syncing Wikimedia installation... : Testing
15:33 Reedy: pointing php to php-1.20wmf3
13:22 logmsgbot: maxsem synchronized php-1.20wmf3/extensions/MobileFrontend/ 'Deploying MF fixes'
13:21 logmsgbot: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Deploying MF fixes'
12:19 Reedy: Purged www.mediawiki.org/xml/export-0.7.xsd
12:17 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.7.xsd 'Push out updated version of export-0.7.xsd'
06:45 apergos: reboot dataset2, kernel update and security updates
02:33 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Fri Jun 1 02:33:36 UTC 2012
02:11 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Fri Jun 1 02:11:08 UTC 2012

May 31

23:40 logmsgbot: kaldari Finished syncing Wikimedia installation... : scapping for new LastModified and E3Experiments extensions
23:14 logmsgbot: kaldari Started syncing Wikimedia installation... : scapping for new LastModified and E3Experiments extensions
23:09 logmsgbot: aaron synchronized multiversion/ 'Updating multiversion code to head.'
22:25 logmsgbot: kaldari Started syncing Wikimedia installation... :
22:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable randomrootpage again'
22:04 logmsgbot: aaron synchronized wmf-config/swift.php 'Purge from squid all thumbs in Swift on purge.'
21:50 logmsgbot: reedy synchronized php-1.20wmf4/includes/specials/SpecialLog.php
21:50 logmsgbot: reedy synchronized php-1.20wmf4/includes/logging/LogEventsList.php
20:46 pp-pdf2: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
20:46 pp-pdf3: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
20:46 pp-pdf1: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
20:38 logmsgbot: lcarr synchronized wmf-config/mc.php
20:00 LeslieCarr: powering down mw32 for maintenance
19:59 LeslieCarr: powering down mw30 for maintenance
19:57 LeslieCarr: powering off mw31
17:32 LeslieCarr: rebooting mw1135 for kernel upgrade
17:24 cmjohnson1: running memtet on mw64
17:14 LeslieCarr: rebooting unresponsive mw1143
17:13 LeslieCarr: rebooting unresponsive mw1135
17:10 LeslieCarr: rebooted mw1091 for kernel upgrade
17:09 LeslieCarr: rebooted ms1004 for kernel upgrade
17:08 LeslieCarr: rebooted mw1102 because it thinks it has no eth0
17:05 LeslieCarr: rebooted mw1091 due to being unresponsive
17:01 LeslieCarr: rebooted ms1004 due to it being unresponsive
17:01 LeslieCarr: rebooted ms1004
14:28 binasher: pulling srv199 from lvs again for further experimentation
14:03 binasher: returning srv199 to lvs (dialed back to slow / no longer running php 5.4)
13:40 maplebed: upgrading and rebooting eqiad es hosts due to 210 day kernel bug thingy.
13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bring in a few shell requests'
12:46 logmsgbot: reedy synchronized php-1.20wmf4/languages/Language.php '(bug 36839) Use mb_check_encoding() if available'
12:45 logmsgbot: reedy synchronized php-1.20wmf3/languages/Language.php '(bug 36839) Use mb_check_encoding() if available'
10:47 binasher: temporarily pulled srv199 from lvs for php testing
10:44 Reedy: reedy synchronized php-1.20wmf4/extensions/UploadWizard/ 'Push trunk UW to cluster'
10:43 Reedy: reedy synchronized php-1.20wmf3/extensions/UploadWizard/ 'Push trunk UW to cluster'
02:32 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Thu May 31 02:32:48 UTC 2012
02:17 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings to disable PageTriage on test2'
02:10 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Thu May 31 02:10:43 UTC 2012
00:26 logmsgbot: awjrichards synchronizing Wikimedia installation... :

May 30

23:01 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
22:22 logmsgbot: awjrichards synchronizing Wikimedia installation... : Weekly MobileFrontend deployment and picking up ZeroRatedMobileAccess i18n changes
21:50 logmsgbot: kaldari synchronized php-1.20wmf4/extensions/CentralNotice 'deploying 98db6a177df977a699576da9688588c77bf81b04'
21:30 K4-713: Synchronized payments cluster to DonationInterface 43a457e56d
21:16 LeslieCarr: rebooted db1044 (unresponsive server)
21:10 LeslieCarr: rebooted db1031 (unresponsive server)
21:08 LeslieCarr: rebooted db1029 (unresponsive server)
21:07 LeslieCarr: restarted db1026 (unresponsive server)
21:03 LeslieCarr: restarted db1012 (unresponsive server)
20:57 LeslieCarr: restarted networking on cp1036
19:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable randomrootpage on wikibooks and wikisources'
19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 27 more of the misc wikis to 1.20wmf4
19:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Some more of the misc wikis to 1.20wmf4
19:11 LeslieCarr: rebooting neon
18:55 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All special wikis to wmf4
18:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisource, wikiversity, wiktionary to wmf4
18:45 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikibooks, wikinews, wikiquote to wmf4
18:29 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Commonswiki to wmf4
18:16 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved EN, non-wikipedia, non-special, sites to wmf4
18:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Fix capitalisation'
18:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable random root page on testwiki'
17:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add enabling code for randomrootpage'
17:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add setting for random root page'
17:47 LeslieCarr: cleared mobile varnish cache
17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repository
17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repo
17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repo
17:40 logmsgbot: aaron synchronized php-1.20wmf4/extensions/PageTriage 'Switched to wmf4 extension branch to get 0be1787634613a36439b760d6d5f0639724f8a7b'
16:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'subpages for frwikibooks'
12:00 mutante: restarting pdns on ns2
11:41 mutante: running authdns-update to push analytics1011 to 1022 entries
06:05 logmsgbot: hashar synchronized docroot/mediawiki/xml 'bug 37111 deploying export-0.7.xsd'
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Wed May 30 02:37:00 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 30 02:23:47 UTC 2012

May 29

21:09 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabled new thumb purge hook on remaining wikis'
20:49 LeslieCarr: renaming analytics1001.eqiad.wmnet to analytics1001.wikimedia.org
20:25 pp-pdf2: restarted services
20:25 pp-pdf3: restarted services
20:25 pp-pdf1: restarted services
20:25 pp-pdf2: cleaned /tmp and sandbox/cache/
20:25 pp-pdf1: cleaned /tmp and sandbox/cache/
20:25 pp-pdf3: cleaned /tmp and sandbox/cache/
20:15 Thehelpfulone: "Site requests" was renamed to "Site configuration" under the Wikimedia product in Bugzilla, don't know who did it though
20:12 LeslieCarr: reloading analytics1001
19:50 pp-pdf2: restarted all services
19:50 pp-pdf3: restarted all services
19:50 pp-pdf1: restarted all services
19:50 pp-pdf3: add libtidy.so
19:50 pp-pdf1: add libtidy.so
19:50 pp-pdf2: add libtidy.so
19:49 pp-pdf3: install mwlib.epub
19:49 pp-pdf2: install mwlib.epub
19:49 pp-pdf1: install mwlib.epub
19:49 pp-pdf1: update simplejson to 2.5.2
19:49 pp-pdf3: update simplejson to 2.5.2
19:49 pp-pdf2: update simplejson to 2.5.2
19:49 pp-pdf1: update mwlib.rl to 0.12.11
19:49 pp-pdf2: update mwlib.rl to 0.12.11
19:49 pp-pdf3: update mwlib.rl to 0.12.11
19:48 pp-pdf1: update pip to 1.1
19:48 pp-pdf3: update pip to 1.1
19:48 pp-pdf2: update pip to 1.1
18:59 LeslieCarr: flushed mobile varnish caches after push
16:54 maplebed: kicking pdns on dobson to try and make it happy again.
16:40 notpeter: decom of all srv lower than 190
16:27 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/#/c/9204/ - use protocol-relative url for nostalgiawiki wgSiteNotice'
16:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'cleanup wgNoticeBanner_Harvard2011 https://gerrit.wikimedia.org/r/#/c/9205/'
16:17 hashar: /usr/local/apache/common-local is 4G where as / is 7G on srv187. Looks like deploying wmf2 + wmf3 + wmf4 will require partitions to be resized.
16:10 hashar: srv187 and srv188 are out of disk space
16:10 logmsgbot: hashar synchronized search-redirect.php 'https://gerrit.wikimedia.org/r/9206 - cleanup search-redirect.php'
14:37 cmjohnson1: removing disk4 on virt1 for replacement
12:18 mutante: killing / restarting morebots
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Tue May 29 02:37:44 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 29 02:24:21 UTC 2012

May 28

23:15 logmsgbot: reedy synchronized php-1.20wmf4/extensions/UploadWizard/ 'Master UW per Kaldari'
18:25 logmsgbot: reedy synchronized php-1.20wmf4/extensions/Translate/tag/PageTranslationHooks.php 'Fix Catchable fatal error'
18:08 logmsgbot: reedy synchronized php-1.20wmf4/LocalSettings.php
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf4
16:13 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild l10n for php-1.20wmf4
15:44 logmsgbot: reedy synchronizing Wikimedia installation... : test2wiki to 1.20wmf4 to build localisation cache
14:39 logmsgbot: reedy synchronizing Wikimedia installation... : Does running scap on its own with no wikis on that version build l10n for it? I suspect not...
14:22 logmsgbot: reedy synchronized php-1.20wmf4/ 'Staging php-1.20wmf4'
14:15 Reedy: sync-dir'ing php-1.20wmf4
14:12 Reedy: Copying php-1.20wmf4 from /tmp to /h/w/c on Fenari
13:12 apergos: doing security updates for a batch of mws in eqiad
11:22 apergos: updated kernel etc on mw1133, reboot
11:08 apergos: powercycled mw1133
09:05 apergos: rebooted snapshot4, 3 for security updates
08:43 apergos: rebooted snapshot1002, security updates (will do the same for 1003, 1004 shortly)
08:37 apergos: rebooted snapshot1001, security updates
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon May 28 02:24:05 UTC 2012

May 27

22:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37134 - s:cs: site settings'
14:45 logmsgbot: asher synchronized wmf-config/db.php 'returning db51 to prod as an s4 slave'
14:34 maplebed: dns and puppet changes for s4 master rotation done.
14:18 binasher: rebooted db51, reslaved
14:09 maplebed: new s4 master position post-rotation is master_log_file="db31-bin.000334", master_log_pos=583315125
14:02 logmsgbot: ben synchronized wmf-config/db.php 's4 master switch complete; db31 is the new master. turning off read only on s4'
13:25 Tim: on db31 set global read_only=1
13:17 logmsgbot: tstarling synchronized wmf-config/db.php 's4 read-only and taking out db51'
04:27 binasher: kaulen - temporarily disabled swap and set oom_adj score to 15 for apache
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun May 27 02:24:13 UTC 2012

May 26

19:54 apergos: restarting apache2 on kaulen
19:34 Nemo_bis: bugzilla down again
14:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Settting wgBlacklistSettings'
11:05 apergos: stopping and restarting apache on kaulen blah blah blah
09:12 Reedy: Bugzilla is down, Kaulen looks to be in swap death again
06:46 apergos: powercycling kaulen
05:53 hashar: kaulen dead :-[
05:47 hashar: Bugzilla on Kaulen being super slow again
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat May 26 02:22:37 UTC 2012
00:40 K4-713: synchronized payments cluster to DonationInterface 04809f1cf0d

May 25

22:57 maplebed: powercycled kaulen on the mgmt interface
19:09 maplebed: disabled the outdated /etc/init.d/gmond on spence. use ganglia-monitor instead.
18:57 RobH: bugzilla appears back online
18:57 RobH: kaulen is rebooted, it may have had a runaway process or a memory leak, not sure yet, but it was locked up from access
18:55 RobH: kaulen serial console unresponsive, rebooting
17:42 paravoid: rebooting gurvin & yvon with new kernel
17:25 paravoid: resetting gurvin, load spiking at 370+, SSH unreachable, 214 days of uptime
15:34 mark: Power cycled kaulen
15:23 hashar: kaulen (bugzilla) unreacheable :-(
13:56 RobH: palladium disk replaced
13:52 cmjohnson1: replacing ps2 on mw1017
13:50 RobH: palladium has a bad disk, goign to replace it
13:37 RobH: updating drac on search18, shouldnt cause system reboot.
10:42 apergos: restarted apache on kaulen, was seeing page.cgi segfaults in dmesg and he logs, huge cpu wait spikes (why?)

May 24

23:34 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
23:32 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Updating technical feedback email address for mobile feedback'
23:17 logmsgbot: awjrichards synchronizing Wikimedia installation... : Picking up changes to hide feedback form to prevent spamming of mobile feedback page - f6ed8ba
23:12 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Enabling 'technical feedback' link on mobile feedback form to disable feedback form'
23:00 binasher: stopped replication on es1002 in order to rsync cluster23 to es1003
21:53 logmsgbot: aaron synchronized php-1.20wmf3/extensions/UploadWizard 'deployed 144b58854e38d910210ccd23402225e5b1d2d62d'
21:52 RoanKattouw: Restarted morebots

May 23

21:46 binasher: shutting down mysql on db12 in able to restart with binlogging disabled
21:45 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db60'
21:34 ottomata1: upgraded udp-filter to 0.2.4 on oxygen, emery, and locke (with maplebed's help)
18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last 286 wikis over to 1.20wmf3
17:34 maplebed: deployed change to varnish configs for preilly; adding more carriers
17:13 logmsgbot: ben synchronized wmf-config/CommonSettings.php 'changing the URL for the mobile feedback page to the Project namespace'
17:12 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'RL hack'
17:00 logmsgbot: reedy synchronized wmf-config/ 'Tidying unpushed changes'
15:35 RobH: ns1 died on update, restarting pdns
15:34 RobH: updated dns for analytics mgmt
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 23 02:45:44 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 23 02:23:33 UTC 2012
01:10 K4-713: synchronized payments cluster to DonationInterface 4be175e43f
00:16 Ryan_Lane: flushed the varnish cache for mobile again
00:13 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'
00:11 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'
00:06 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'try again'
00:00 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'

May 22

23:49 Ryan_Lane: flushed the varnish cache for mobile again
23:31 Ryan_Lane: flushed the varnish cache for mobile
23:21 logmsgbot: preilly synchronizing Wikimedia installation... : MobileFrontend Weekly Deployment
21:34 RobH: updating dns for mgmt of new servers in eqiad
20:52 logmsgbot: reedy synchronized php-1.20wmf3/extensions/MoodBar/ 'updating to master'
20:24 notpeter: powering up db1003
20:05 notpeter: starting xtrabackup dump from db1004 to db1020 for new eqiad s4 slave
19:55 notpeter: starting xtrabackup dump from db1033 to db1001 for new eqiad s1 slave
19:31 notpeter: reimaging db1001 and db1020
19:18 RobH: dns update for new servers mgmt ips
19:12 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding message files for interwiki extension
18:44 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/file/LocalFile.php 'deployed 826f82eaccdf2a017a8ddb27829156f7c474db84'
18:44 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
18:16 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php 'deployed 178a8597e32122feeb593219452f26864639d9ad'
17:56 maplebed: done with deploy to swift to make mediawiki write thumbnails for all wikis
17:47 logmsgbot: aaron synchronized wmf-config/swift.php 'Switched all wikis to new Swift thumb copy hook.'
17:44 maplebed: starting deploy to make mediawiki write thumbnails to swift for all wikis
17:35 logmsgbot: reedy synchronized php-1.20wmf3/extensions/TranslationNotifications/ 'Pushing new version of translationnotification out'
17:32 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'pushing interwiki loading code'
17:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'pushing interwiki variables out'
17:30 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Interwiki/ 'pushing interwiki code to cluster'
03:57 hashar: GlusterFS receiving 30Mbytes/sec of input traffic. Killing labs again :-D
02:50 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Tue May 22 02:50:09 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 22 02:26:50 UTC 2012
01:40 K4-713: updated payments cluster to Donation Interface 67b40c9307b
00:17 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/file/LocalFile.php 'deployed dfa7120f1bcd2c172096caf0ca65a06119e592c3'

May 21

22:31 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove header fail logging'
22:30 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php
22:20 logmsgbot: preilly synchronized wmf-config/PrivateSettings.php
22:19 logmsgbot: preilly synchronized wmf-config/CommonSettings.php
21:52 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Don't do content length checking if it's a head request'
21:47 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Don't do content length checking if it's a head request'
20:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable PageTriage on enwiki'
20:47 logmsgbot: catrope synchronized php-1.20wmf3/extensions/PageTriage 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
20:47 logmsgbot: catrope synchronized php-1.20wmf3/extensions/ArticleFeedbackv5 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
20:46 logmsgbot: catrope synchronized php-1.20wmf3/includes/resourceloader 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
20:46 logmsgbot: catrope synchronized php-1.20wmf2/extensions/PageTriage 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
20:46 logmsgbot: catrope synchronized php-1.20wmf2/extensions/ArticleFeedbackv5 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
20:45 logmsgbot: catrope synchronized php-1.20wmf2/includes/resourceloader 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf3
18:04 Jeff_Green: dist-upgrade and reboot loudon
17:48 andrewbogott_: ran authdns-update on dobson to pick up virt1002-1008 changes
17:13 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/LocalRepo.php 'deployed dfa7120f1bcd2c172096caf0ca65a06119e592c3'
16:22 mutante: analytics1001 to 1010 installed and up in puppet
12:45 mark: Started ircecho on manganese
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Mon May 21 02:46:37 UTC 2012
02:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'There's no helping metawiki now!'
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon May 21 02:23:51 UTC 2012

May 20

19:29 logmsgbot: aaron synchronized wmf-config/swift.php 'more profiling'
17:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Syncing I6b0e91cd/bug 36931: tweaking account creation whitelist for ptwiki event'
02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sun May 20 02:43:08 UTC 2012
02:21 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun May 20 02:21:20 UTC 2012

May 19

21:07 cmjohnson1: shutting down storage3 to replace RAID controller card
18:37 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling PageTriage extension on enwiki per request from Kaldari, due to bug 36968'
02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sat May 19 02:43:56 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat May 19 02:22:21 UTC 2012

May 18

23:27 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on commonswiki.'
20:18 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Tighten debugging'
20:12 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Widen debugging'
20:06 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/FileBackendStore.php 'deployed 0624af8f2e9666fbe0820c0caca6d7ea3c6eeb7b'
20:02 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 're-enable debugging (fatal disabled)'
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Put wikibooks back on 1.20wmf3
19:27 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Disable debugging and content length checking for now'
19:26 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'better debugging'
19:24 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'better debugging'
19:21 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'RE-enable header fail stuff with debug logs'
19:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add debug log group for headerfail'
19:00 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Trying older version'
18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Push wikibooks back to 1.20wmf2 due to collection being broken
18:31 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection
18:28 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection '1.20wmf2 collection for testing'
18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.20wmf3
18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.20wmf2
18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable collection on test2wiki'
18:02 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection/Collection.session.php
18:00 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection/Collection.templates.php 'Testing partial revert'
15:45 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Applying various changes made this afternoon: 718fb59..811dbd8'
14:25 hashar: setup a Jenkins job to lint PHP files in operations/mediawiki-config.git:/wmf-config/
14:00 mutante: authdns-update - pushing fix for reverse lookup in eqiad subnets
12:04 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Syncing https://gerrit.wikimedia.org/r/7931 & https://gerrit.wikimedia.org/r/7934 : minor pmtpa/wmflabs switches'
10:39 logmsgbot: hashar synchronized wmf-config/wgConf.php 'https://gerrit.wikimedia.org/r/#/c/7933/ change cluster name "beta" to "wmflabs"'
08:59 logmsgbot: nikerabbit synchronized php-1.20wmf3/extensions/Translate/specials/SpecialAggregateGroups.php 'Temp fix for bug 36944'
03:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable UploadWizard on donatewiki and foundationwiki'
02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 18 02:43:38 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Fri May 18 02:22:11 UTC 2012
01:42 Tim: on cp1004: set net.ipv4.tcp_tw_recycle=0 and net.ipv4.tcp_tw_reuse=1
01:39 binasher: filejournal migration complete
00:50 binasher: migrating fliejournal to innodb on all wikis
00:22 Tim: on cp1005: set tcp_tw_recycle=0

May 17

22:18 binasher: migrated centralauth.wikiset to innodb
22:01 binasher: migrating centralauth.spoofuser to innodb via osc (13.5mil rows)
22:00 binasher: migrated centralauth.global_group to innodb
21:53 maplebed: reverted mobile change from this morning - testing completed.
21:42 binasher: es1004 is replicating again
21:39 binasher: resumed replication to es1002
21:21 logmsgbot: catrope synchronized php-1.20wmf3/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa'
21:20 logmsgbot: catrope synchronized php-1.20wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa'
21:09 logmsgbot: aaron synchronized wmf-config/swift.php '+commonswiki'
20:58 logmsgbot: aaron synchronized wmf-config/swift.php 'revert change to itwiki'
20:57 Jeff_Green: several package updates on payments* and silicon
20:56 logmsgbot: aaron synchronized wmf-config/swift.php '+itwiki'
20:40 logmsgbot: aaron synchronized wmf-config/swift.php
20:34 maplebed: deployed change to swift and mediawiki for MW to write thumbnails to swift instead of rewrite.py with aaron
20:32 maplebed: deployed parallel thumbnail purging for test, test2, and mediawiki with aaron
20:25 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabled thumb copy hook for testwikis and mw.org'
20:14 binasher: completed securepoll_votes.vote_ip and all ipv6 schema migration
20:10 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
20:09 logmsgbot: aaron synchronized wmf-config/swift.php
20:03 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabling new purge hook on testwikis again.'
20:03 binasher: running securepoll_votes.vote_ip schema migration on s1
20:01 binasher: running securepoll_votes.vote_ip schema migration on all s2 dbs
19:19 binasher: running securepoll_votes.vote_ip schema migration on all s4 + s3 dbs
19:17 binasher: running securepoll_votes.vote_ip schema migration on all s5 dbs
19:16 binasher: running securepoll_votes.vote_ip schema migration on all s6 dbs
19:02 binasher: running securepoll_votes.vote_ip schema migration on all s7 dbs
18:49 binasher: syncing cluster23 tables from es1002 to es1004
18:46 binasher: stopped replication on es1002
18:44 notpeter: restarting puppet on brewster
18:39 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
18:13 logmsgbot: aaron synchronized wmf-config/swift.php
18:08 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo 'deployed 103efda39dd57bc22898bd0e69932982c1cfd588'
18:00 Jeff_Green: shutting down grosley for disk and RAM upgrades
17:42 notpeter: temporarily turning off puppet on brewster for preseed hackz
17:20 maplebed: flushing the mobile cache post-deploy
17:17 maplebed: deploying config change to mobile - more zero IP addresses. gerrit r7867
15:31 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
15:30 mutante: sync-common-file interwiki.cdb
15:30 mutante: creating fresh interwiki.cdb from dumpInterwiki.php
15:30 Jeff_Green: adding DNS records to wikimedia.org for RT #2960
14:22 mutante: adding gerrit project analytics/udplog parent analytics
13:44 cmjohnson1: shutting down bellin for troubleshooting
09:04 hashar: Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown.
08:59 hashar: Broken the cluster by having an invalid global set
08:58 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
08:47 logmsgbot: hashar synchronizing Wikimedia installation... :
08:44 hashar: running scap to apply https://gerrit.wikimedia.org/r/7702
08:41 hashar: Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination
08:15 hashar: WMFLabs seems to have recovered now
06:50 hashar: WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore
03:41 logmsgbot: asher synchronized wmf-config/db.php 'returning db12 and db46'
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Thu May 17 02:48:02 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Thu May 17 02:22:02 UTC 2012
02:18 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable SpecialCite everywhere'
01:40 Tim: on cp1004: reverted after TIME_WAIT client connections reached 38k with no sign of a plateau
01:37 Tim: on cp1004: trying tcp_tw_reuse=1 instead of tcp_tw_recycle
01:00 Tim: reverted after client-side TIME_WAIT connections rose rapidly from 367 to 9000
00:59 Tim: experimentally setting net.ipv4.tcp_tw_recycle=0 on cp1004

May 16

23:50 logmsgbot: aaron synchronized php-1.20wmf3/includes/upload/UploadBase.php 'deployed 4b0a61227fce37202da2b62b7dc2474bd227873f'
22:47 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use $wgSiteStatsAsyncFactor=1.'
22:32 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on testwikis.'
21:45 maplebed: reverted this morning's mobile push - tests completed
21:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36913 - Enable Collection on kkwiki'
21:09 logmsgbot: reedy synchronized php-1.20wmf3/cache/interwiki.cdb 'Updating interwiki cache'
21:09 logmsgbot: reedy synchronized php-1.20wmf2/cache/interwiki.cdb 'Updating interwiki cache'
20:05 binasher: ran ipv6 migrations on globalblocks
20:05 binasher: converted centralauth.globalblocks from myisam to innodb
19:57 binasher: rebooting db12 for kernel upgrade
19:51 binasher: stopping mysql on db12
19:50 binasher: recentchanges.rc_ip migration completed
19:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last bits of tidying up
19:44 binasher: rebooted db46
19:38 binasher: shutting down mysql on db46, preparing to reboot for kernel upgrade
19:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
19:28 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db59'
19:25 binasher: running recentchanges.rc_ip (ipv6) schema migration on enwiki master (5.2mil rows) via os��c - batten down the hatches!
19:17 binasher: running recentchanges.rc_ip (ipv6) schema migration on s2 dbs via os��c
19:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
19:14 Reedy: manually ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 "sudo -u mwdeploy rsync -a 10.0.5.8::common/*.dblist /usr/local/apache/common-local" because sync-dblist is woefully out of date..
19:13 notpeter: restarting ganglia on nickel
19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 12 more misc/wikimedia wikis to 1.20wmf3
18:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All closed wikis to 1.20wmf3
18:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All special wikis to 1.20wmf3
18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikimedia wikis to 1.20wmf3
18:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisource to 1.20wmf3
18:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote to 1.20wmf3
18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiversity to 1.20wmf3
18:45 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktibooks to 1.20wmf3
18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf3
18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikinews to 1.20wmf3
18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedia en projects to 1.20wmf3
18:27 binasher: running recentchanges.rc_ip (ipv6) schema migration on s3 dbs via os��c (s4 already completed during prior testing)
18:25 mutante: synced wikiversions.* files from NFS to spence local to prevent death of check_job_queue monitoring
18:21 binasher: running recentchanges.rc_ip (ipv6) schema migration on s5 dbs via os��c
18:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf3, again
18:17 logmsgbot: aaron synchronized php-1.20wmf3/includes/ImagePage.php 'deployed 86e2372772e618c5d1238ae480d9f632789bbe50'
18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf2
18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf3
18:10 binasher: running recentchanges.rc_ip (ipv6) schema migration on all s6 dbs via os��c
18:03 binasher: running recentchanges.rc_ip (ipv6) schema migration on all s7 dbs via os��c
17:43 binasher: ipblocks migration completed for all wikis
17:38 binasher: running ipblocks schema migration on all s2 dbs via os��c
17:35 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend 'Picking up fix for fatal in api in MobileFrontend at 9936e7a'
17:34 logmsgbot: awjrichards synchronized php-1.20wmf3/extensions/MobileFrontend 'Picking up fix for fatal in api in MobileFrontend at 9936e7a'
17:16 maplebed: deploying change to swift to make which containers write thumbs configurable
17:11 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'zero and mobile changes'
17:10 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'zero and mobile changes'
17:08 RobH: aluminum back online
17:00 binasher: running ipblocks schema migration on all s3 (819) dbs via os��c
16:59 binasher: running ipblocks schema migration on all s4 dbs via os��c
16:58 binasher: running ipblocks schema migration on s5/dewiki via osc
16:57 RobH: aluminum shut down for hard disk additions
16:56 binasher: running ipblocks schema migration on all s6 dbs via osc
16:51 RobH: udpating dns for osm web servers
16:50 binasher: running ipblocks schema migration on all s7 dbs via osc
16:49 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
16:41 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php 'deployed 634c3be2bba6a46e28aa997d7ab388ebf90b36a6'
16:31 maplebed: clearing the mobile varnish cache
16:29 maplebed: deploying gerrit change 7798 to the mobile varnish servers
07:41 logmsgbot: raindrift synchronized php-1.20wmf3/extensions/PageTriage/api/ApiPageTriageTemplate.php 'fixing exception bug that makes lots of logspam'
07:41 logmsgbot: raindrift synchronized php-1.20wmf2/extensions/PageTriage/api/ApiPageTriageTemplate.php 'fixing exception bug that makes lots of logspam'
06:20 Ryan_Lane: restarted lucene on search1015
05:58 Tim: setting net.ipv4.tcp_tw_recycle=1 on cp1005 seems to have fixed it, doing it on cp1004 as well now
05:52 Tim: on cp1005 setting tcp_tw_recycle=1
05:29 Tim: experimentally started squid on cp1004
04:05 hashar: updating a few plugins on Jenkins (host: gallium )
03:34 Ryan_Lane: stopped the squid process on cp1004 and stopped puppet to avoid it being restarted. it's having issues and I can't debug it right now.
03:22 Ryan_Lane: repooling squid frontend on cp1004
03:14 Ryan_Lane: depooling cp1004 and stopping the squid backend service to let some connections close
02:43 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 16 02:43:51 UTC 2012
01:10 logmsgbot: reedy synchronized php-1.20wmf3/extensions/RandomRootPage 'dark deploy randomrootpage extension (I'll enable it later)'

May 15

23:53 logmsgbot: aaron synchronized php-1.20wmf3/includes/Block.php 'deployed 7694faf68f975ea9c4888d575b33dabb84e90083'
23:42 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
23:27 ssmollett: upgraded ganglia-monitor and gmetad from 3.1.2-2.1 to 3.3.5-2
23:26 logmsgbot: awjrichards synchronizing Wikimedia installation... :
23:24 K4-713: upgraded minfraud version on the payments account
23:22 K4-713: updated and synchronized the payments cluster to DonationInterface d997e7ea1c
23:06 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
22:50 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Disable CentralAuth logging to file'
22:50 logmsgbot: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend to 0880467
22:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable CentralAuth logging to file'
22:13 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
21:09 logmsgbot: raindrift synchronizing Wikimedia installation... : PageTriage update
21:05 logmsgbot: raindrift synchronizing Wikimedia installation... : PageTriage update
19:40 logmsgbot: aaron synchronized wmf-config/swift.php 'disabled new hook for now.'
19:35 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Add wfDebugLog call'
19:31 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Add wfDebugLog call'
19:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php '$wgDebugLogGroups[updateTranstagOnNullRevisions] = udp://10.0.5.8:8420/updateTranstagOnNullRevisions'
19:26 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Only conditionally disable updateTranstagOnNullRevisions hook. Debugging to come'
17:45 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
17:41 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
17:35 logmsgbot: aaron synchronized wmf-config/swift.php 'Use new thumb purge hook for testwikis'
16:54 RobHalsell: updated apache config for wiki-pedia.org, seems the bot doesnt spam that anymore =[
16:36 mutante: srv app servers max. uptime with older kernel down to ~120 days after another bunch of upgrades
16:34 RobHalsell: updating dns for wiki-pedia.org
12:20 hashar: deployment-prep replaced most occurrences of /mnt/upload to /mnt/upload6
10:37 apergos: on db39 dropped triggers pt_osc_elwiki_recentchanges ins, del, upd, they were preventing all elwiki edits except bot edits with the complaint Table 'elwiki._recentchanges_new' doesn't exist ... binasher, doublecheck me please?
09:24 mutante: srv278 - still has issues as in reopnened RT #24 - upgrading kernel anyways
03:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'update wgUploadNavigationUrl on all cs wikis'
02:35 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 15 02:35:53 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Tue May 15 02:23:47 UTC 2012
01:09 logmsgbot: asher synchronized wmf-config/db.php 'returning db31 as an s4 slave'
01:05 logmsgbot: aaron synchronized php-1.20wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed f20e752630575f8384083f0ad0401e250c8babf5'
01:00 binasher: shutting down mysql on db31, then rebooting
00:59 logmsgbot: asher synchronized wmf-config/db.php 'pulling db31 from s4 for kernel upgrade'
00:58 binasher: new s4 master position - MASTER_LOG_FILE='db51-bin.000114', MASTER_LOG_POS=1772578
00:57 logmsgbot: asher synchronized wmf-config/db.php 'new s4 master'
00:55 logmsgbot: asher synchronized wmf-config/db.php 'setting s4 to read-only, switching master to db51'
00:54 binasher: preparing to rotate s4 master from db31 to db51
00:48 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bring in numerous shell requests from gerrit'
00:48 binasher: rebooting db51 for kernel upgrade, prior to promoting to s4 master
00:47 logmsgbot: asher synchronized wmf-config/db.php 'pulling db51 from s4 for kernel upgrade'
00:01 binasher: just completed an online schema change for commonswiki.recentchanges in prod. woo!

May 14

22:02 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
21:08 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Live hack out updateTranstagOnNullRevisions'
20:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf3
19:38 logmsgbot: reedy synchronizing Wikimedia installation... : Running scap to make sure everything is ok...
19:36 logmsgbot: reedy synchronized php-1.20wmf3/cache/l10n/ 'Resync localisation cache'
19:26 logmsgbot: reedy synchronized live-1.5/ 'Push live-1.5 new symlinks'
19:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf3
19:16 logmsgbot: reedy synchronized php-1.20wmf3/cache/trusted-xff.cdb
19:14 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf3.php
19:13 logmsgbot: reedy synchronized php-1.20wmf3/extensions/ 'Push extensions out properly'
19:11 binasher: resyncing cluster22 from es1002 to es1004
19:02 logmsgbot: reedy synchronized php-1.20wmf3/LocalSettings.php 'Use newer version'
19:01 logmsgbot: reedy synchronized php-1.20wmf2/LocalSettings.php 'Use newer version'
18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf3
18:57 logmsgbot: reedy synchronized php-1.20wmf3/cache/l10n/ 'Syncing localisation cache files'
18:49 Ryan_Lane: added OATHAuth to components list for MediaWiki Extensions product in bugzilla
18:43 Ryan_Lane: switching sessions back to memcached for labsconsole
18:42 Ryan_Lane: adding OATHAuth to labsconsole
18:40 Ryan_Lane: completed upgrade to 1.20wmf2 on labsconsole
18:30 Ryan_Lane: upgrading labsconsole to 1.20wmf2
18:26 logmsgbot: reedy synchronized php-1.20wmf3 'Initial pushing of php-1.20wmf3 files to apaches'
18:12 Reedy: Killing old php-1.20wmf1 directories from apaches to save full disks
13:48 mutante: copying outdated wikiversions.dat/.cdb files from /home to /usr/local on spence, which fixes check_job_queue (thanks jeremyb)
13:07 mutante: opening a bz bug for check_job_queue issue related to CommonSettings.php BZ:36835
07:43 mutante: still upgrading/rebooting a couple srv (API) application servers with long uptime
06:22 apergos: restarted lucene search on search1016 it had stopped doing anything useful (see ganglia graphs, also nothinig wtitten to logs)
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Mon May 14 02:22:09 UTC 2012

May 13

02:24 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sun May 13 02:24:51 UTC 2012

May 12

02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sat May 12 02:22:18 UTC 2012

May 11

22:10 logmsgbot: preilly synchronized wmf-config 'add wikimedia to zero image disable list fix header'
21:52 logmsgbot: preilly synchronized wmf-config 'add wikimedia to zero image disable list'
19:49 Reedy: ran apache-graceful-all
19:42 RobH: apache restarted by puppet run on srv286
19:31 RobH: shutting down srv286 and srv286 for power rebalancing
19:23 RobH: srv260 and srv261 back in business
19:10 RobH: srv261 & srv261 shutting down for power rebalancing within the rack
18:33 notpeter: shutting down search 13-20 for hd upgrades
18:05 maplebed: swift: deleting the unsharded version of all sharded containers
18:03 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UploadWizard/ 'Deploy 4b5df1a1151ac80e309d396102e5e2a8d0c27ccb'
17:46 maplebed: deleted wikipedia-de-local-thumb container from swift. the sharded version is currently being used.
15:33 mutante: adding DNS entries for analytics hosts in new vlan 1121 (10.64.21.0/24), hosts starting at .101 to match names analytics1001 = .101 and ++
15:03 mutante: mw62 -unless somebody was on that right now it died. mgmt also just Create Instance Error
14:06 mutante: kernel upgrading / rebooting srv servers where uptime > 200 d order by uptime desc limit 1
13:12 mutante: installing package upgrades on pdf1-3 (and installed requested indic fonts via new puppet role class)
11:39 mutante: starting ms-be swift-container-auditors every once in a while
11:35 mutante: stat1 - installed new kernel, but waiting to reboot. schedule with aotto
11:24 mutante: upgrading packages/kernel on hooper, rebooting (Blog,Etherpad,Racktables)
09:21 mutante: ekrem was close running out of disk again. logrotated apache logs, changed config to: size 512M,rotate 3
08:58 mutante: package upgrades on ekrem (IRC server, WAP, Apple dict...)
08:51 mutante: rebooting marmontel (blog)
08:48 mutante: upgrading apache/mysql/kernel on marmontel (blog)
02:20 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 11 02:20:39 UTC 2012
02:00 RoanKattouw: Started Apache back up on srv200, done debugging
01:58 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UserDailyContribs/UserDailyContribs.hooks.php 'Deploy 3c45831ffe1817f3dc18f06644db46b1b74173e7'
01:17 RoanKattouw: Stopping Apache on srv200 so I can use it as my guinea pig for segfault debugging
00:56 logmsgbot: tstarling synchronized php-1.20wmf2/includes/User.php 'header log'
00:49 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
00:48 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
00:40 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
00:30 Tim: restarted socat on fenari so that fatal.log is reopened
00:29 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'removed logging hack tweaks.'
00:29 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'logging hack tweaks.'
00:27 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'removed some temp logging'
00:27 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
00:16 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
00:00 binasher: pulling cp1044 from lvs for testing

May 10

23:38 logmsgbot: reedy synchronized php-1.20wmf2/extensions/LiquidThreads/classes/Hooks.php 'Updating to master'
22:38 logmsgbot: catrope synchronized php-1.20wmf1/.git 'Make Special:Version show the correct commit now that I have fixed the weird repo state'
22:37 logmsgbot: catrope synchronized php-1.20wmf2/.git 'Make Special:Version show the correct commit now that I have fixed the weird repo state'
22:36 RoanKattouw: Cleaned up weird git repo states on fenari in php-1.20wmf1 and php-1.20wmf2
22:04 maplebed: swift: deleting the unsharded wikipedia-de thumb container contents (the sharded version is currently serving traffic)
19:51 notpeter: rebooting db29 for do a test install of precise
19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36420 - Wikipedia namespace alias for sr.wp'
19:02 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
18:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36694 - Set wgSitename on srwikisource'
18:43 LeslieCarr: restarting mobile varnish
18:33 LeslieCarr: reloaded and purged cache of mobile varnish
18:03 notpeter: starting innobackupex from db10 to blondel
17:39 notpeter: pushing out new zone files. only minor changes
16:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'showupdatemarker on enwiki tooooo'
03:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable show update markers on dewiki'
02:14 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Thu May 10 02:14:03 UTC 2012
01:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/7133/'
00:11 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UploadWizard 'Deploy b45437b6e09018dacfc78c8e4fa822a917858b2d / 62631485ba36f973c0d4a850ef494a8f84c4c86b'
00:11 logmsgbot: catrope synchronized php-1.20wmf1/extensions/UploadWizard 'Deploy b45437b6e09018dacfc78c8e4fa822a917858b2d / 62631485ba36f973c0d4a850ef494a8f84c4c86b'
00:06 logmsgbot: preilly synchronized wmf-config 'remove MF passwords'
00:01 logmsgbot: preilly synchronized wmf-config 'remove MF passwords'

May 9

23:33 notpeter: taking down search20 to do precise test-install
23:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Translate on outreachwiki'
23:25 Reedy: Created Translate tables on outreachwiki
22:49 Reedy: ExtensionDistributor fixed
22:32 Reedy: Debugging ExtensionDistributor being broken. Likely to show more debug output on mw.org if you attempt to use it (though, it wouldn't give you what you wanted anyway)
22:15 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
21:53 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed b9ac85cbf304a65d900cda00fafe53bf82d7a227'
21:53 logmsgbot: aaron synchronized php-1.20wmf2/includes/SiteStats.php 'deployed b9ac85cbf304a65d900cda00fafe53bf82d7a227'
20:52 LeslieCarr: done
20:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bump memory limit to 128MB'
19:39 Ryan_Lane: updating OpenStackManager on virt0 to master again
19:16 Ryan_Lane: updating OpenStackManager on virt0 to master
18:54 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed fa1a8d5119e1174f7458eb9516287f4867c46484'
18:50 RobH: dns update for db61 and db62
18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 295 other wikipedias over to 1.20wmf2
18:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki to 1.20wmf2
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: ruwiki to 1.20wmf2
18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwiki to 1.20wmf2
18:11 notpeter: turning db30 back on
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.20wmf2
17:51 cmjohnson1: to shutting down storage3
16:58 LeslieCarr: restarted mobile varnish instances
16:58 LeslieCarr: flushed mobile varnish cache
16:54 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Make sure Swift backend will have journaling too.'
16:31 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Removed backend config conditional now that everything was switched over.'
14:06 mutante: started container-auditor on ms-be1
09:24 mutante: started container-auditor on ms-be3 and 4
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed May 9 02:37:02 UTC 2012
02:19 Reedy: Running cleanupUploadStash.php over all wikis
02:13 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 9 02:13:10 UTC 2012
01:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36506 - Site logo for Tsonga Wikipedia -- ts.wikipedia.org'
01:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36522 - Upload link should lead to UploadWizard instead of commons:Special:Upload'
01:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36663 - Please allow bureaucrats to add and remove autoreviewer status on pt.wiki'
01:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgShowUpdatedMarker enabled on anything that isn't enwiki or dewiki'
01:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36533 - Set sitename to Telugu Wiktionary'
01:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36595 - Please enable Extention:NewUserMessage on ml.wikipedia'
01:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36595 - Please enable Extention:NewUserMessage on ml.wikipedia'
01:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36571 - Please lock wikimania2011 wiki'
01:21 logmsgbot: reedy synchronized closed.dblist 'Closing wikimania2011wiki'
00:11 maplebed: started process to delete objects that don't exist in the container listings on all swift backends

May 8

23:44 K4-713: synchronized payments cluster to r115155, DonationInterface ccfbb304
23:34 LeslieCarr: purged varnish mobile cache
23:25 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobilefrontend resource version again'
23:25 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ 'd43f5f19ff3599f16200d247b6838cfb04ef1473'
23:25 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ 'd43f5f19ff3599f16200d247b6838cfb04ef1473'
23:22 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobilefrontend resource version'
23:11 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
23:11 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ '2b1e8573fdbcab0feb3a2481167b68fb96abf663'
23:10 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ '2b1e8573fdbcab0feb3a2481167b68fb96abf663'
22:53 RoanKattouw: Actually fixed it now with chmod -R g+w /h/w/conf/httpd
22:47 RoanKattouw: Fixed permissions in /h/w/conf/httpd by running find -group wikidev -not -perm 020 -exec chmod g+w \{\} \;
22:38 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/stylesheets/sections.css 'Live hack to live test broken interface on ICS devices on very large articles'
22:34 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile url transformation on testwiki'
22:13 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'bumping MobileFrontend resource version number'
22:13 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ 'd828a8196d8bc877afdbd1559e8e6d639b51cef7'
22:12 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ 'd828a8196d8bc877afdbd1559e8e6d639b51cef7'
21:53 binasher: rebooting db1018 one more time
21:47 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed 43aa35016b03935b27d439afe9a6b3f1aad1aa8b'
21:45 Ryan_Lane: adding adminbot to the repo
21:32 binasher: rebooting eqiad core db slaves for kernel upgrade
21:29 logmsgbot: aaron synchronized wmf-config/swift.php 'Added new thumbnail purge/import hooks handlers that use the swift backend class; unused atm.'
21:23 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Added swift backend config; unused atm.'
21:15 logmsgbot: asher synchronized wmf-config/db.php 'returning db45 to service'
21:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
21:13 maplebed: delpoyed container sharding for thumbnails to swift for 'dewiki', 'fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki' (in addition to existing sharding for commons and enwiki)
21:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move wikimania2013wiki to php-1.20wmf2
21:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
21:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
21:10 binasher: shutting down mysql across all eqiad core db slaves
20:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'logo for wikimania2013wiki'
20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'remove w'
20:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on wikimania2013wiki'
20:56 logmsgbot: aaron synchronized wmf-config/swift.php 'Switching purge hook to use new sharding scheme.'
20:54 Reedy: Created translate related tables for wikimania2013wiki
20:31 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AntiBot/
20:30 logmsgbot: reedy synchronized php-1.20wmf2/extensions/AntiBot/
20:14 maplebed: creating sharded containers for swift for 'dewiki','fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki'
19:54 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Moved remaining wikis over to new backend config'
19:34 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/
19:12 LeslieCarr: flushed mobile varnish cache
19:11 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/
19:10 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/
18:37 LeslieCarr: reenabled services on fpc5 of cr1-eqiad
18:16 cmjohnson1: updating md1000 controller card firmware on storage3
18:14 LeslieCarr: turned off fpc5 on cr1-eqiad to swap
18:05 LeslieCarr: powering on fpc 5 on cr1-eqiad
18:03 LeslieCarr: powering off fpc5 on cr1-eqiad in order for RobH to physically reseat the card
17:48 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable TranslationNotifications on meta, incubator and wikimania2012'
17:44 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable TranslationNotifications on mediawikiwiki'
17:42 LeslieCarr: switching all masterships over to cr2-eqiad in preparation to reseat cr1 linecard
17:25 LeslieCarr: flushed the mobile cache
17:24 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding localisation cache files before TranslationNotification deploy
17:18 logmsgbot: reedy synchronized php-1.20wmf2/extensions/MobileFrontend/ 'Pushing out head'
17:16 logmsgbot: reedy synchronized php-1.20wmf1/extensions/MobileFrontend/ 'Pushing out head'
17:14 RobH: asw-c1-eqiad connected to both cr1 and cr2
15:16 cmjohnson1: shutting down storage3 to replace raid card
12:40 pp-pdf1: updated mwlib to 0.13.7
12:39 pp-pdf2: updated mwlib to 0.13.7
12:36 pp-pdf3: updated mwlib to 0.13.7
11:59 mutante: merging CSS fix for broken mobile site table layout
02:18 RoanKattouw: Removed and recloned /var/lib/l10nupdate/mediawiki/extensions , it was in a weird state because magic extension submodules work now but my hacky workaround for them not working was still in place
02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
01:12 logmsgbot: tstarling synchronized php-1.20wmf2/includes/api/ApiMain.php
01:10 logmsgbot: tstarling synchronized php-1.20wmf1/includes/api/ApiMain.php
00:44 binasher: rebooted db1034
00:42 logmsgbot: tstarling synchronized php-1.20wmf2/includes/Exception.php
00:42 logmsgbot: tstarling synchronized php-1.20wmf2/includes/DefaultSettings.php
00:37 logmsgbot: tstarling synchronized php-1.20wmf1/includes/Exception.php
00:36 logmsgbot: tstarling synchronized php-1.20wmf1/includes/DefaultSettings.php
00:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Switched enwiki to new backend config.'

May 7

23:52 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobilefrontend resource version #'
23:44 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/
23:43 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PageTriage/
23:35 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/includes/PageTriageUtil.php
23:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgShowExceptionDetails to true for testwiki and test2wiki'
23:25 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07, take 3
23:15 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PageTriage/includes/PageTriageUtil.php
23:15 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/includes/PageTriageUtil.php
23:00 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'wgShowExceptionDetails = false'
22:57 Ryan_Lane: restarting glusterd processes on virt1-5
22:56 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobile resource version'
22:54 Ryan_Lane: upgrading glusterfs on virt1-5
22:49 Ryan_Lane: upgrading glusterfs on labstore1-4
22:48 binasher: running an osc against plwiktionary.recentchanges on master
22:40 paravoid: deleting 14k tmp files from spence's /home/nagios
22:35 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07
22:34 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07
22:24 RoanKattouw: chmod 775 /usr/local/apache/common-local/php-1.20wmf2/extensions/PageTriage with dsh as root
22:19 logmsgbot: raindrift synchronized php-1.20wmf1/resources/startup.js 'touch'
22:18 binasher: rebooting nfs2 to new kernel
22:16 logmsgbot: raindrift synchronized wmf-config/InitialiseSettings.php 'enabling PageTriage on enwp'
22:14 logmsgbot: raindrift synchronized php-1.20wmf2/extensions/PageTriage 'Syncing PageTriage to enwp, a la carte'
22:14 logmsgbot: raindrift synchronized php-1.20wmf1/extensions/PageTriage 'Syncing PageTriage to enwp, a la carte'
21:59 mutante: was still upgrading/rebooting amssq* and knsq* hosts on the side (slow,b/c upload squids). expect temp. nagios squid reports tomorrow as well. out for now.
21:44 binasher: moved default resolution for upload from eqiad to pmtpa
21:29 cmjohnson1: shutting down storage3 for troubleshooting
20:37 binasher: attempting a live online schema change for zuwikitionary.recentchanges on the prod master
20:22 LeslieCarr: (above) restarted nagios-wm on spence
20:20 LeslieCarr: restarted irc bot
20:15 binasher: rebooting db45
20:11 binasher: rebooting db1019
18:46 logmsgbot: reedy synchronized php-1.20wmf1/extensions/Collection/Collection.session.php 'head'
18:45 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection/Collection.session.php 'head'
18:25 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php
18:24 logmsgbot: reedy synchronized php-1.20wmf1/extensions/GlobalBlocking/GlobalBlocking.class.php
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf2
16:16 cmjohnson1: shutting down storage3 to reseat RAID card
15:58 cmjohnson1: Going to power cycling storage3 several times to troubleshoot hardware issue
15:15 RobH: updating firmware on storgae3
14:20 Jeff_Green: stopped cron jobs on storage3 because of RAID failure
12:49 mutante: pushing out virtual host for wikimania2013 wiki. sync / apache-graceful/all
11:18 mutante: continuing with upgrades/reboots in amssq* on the side during the day
11:09 mutante: squids - sq* done. all latest kernel and 0 pending upgrades.
09:27 mutante: rebooting bits varnish sq68-70 one by one..
08:00 mutante: upgrading/rebooting the last couple sq* servers
07:20 binasher: power cycled db45 (crashed dewiki slave)
07:05 logmsgbot: asher synchronized wmf-config/db.php 'db45 is down'
02:25 Tim: on locke: introduced 1/100 sampling for banner impressions, changed filename to bannerImpressions-sampled100.log
02:12 Tim: on locke: moved fundraising logs back where they were
02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
01:38 Tim: on locke: compressing bannerImpressions.log
01:35 Tim: on locke: moved bannerImpressions.log to archive and restarted udp2log
01:26 Tim: on locke: moved fundraising logs from /a/squid/fundraising/logs to /a/squid so that they will be processed by logrotate

May 6

07:03 apergos: manually rotates udplogs on locke, copying destined_for_storage3 off to hume:/archive/emergencyfromlocke/ (jeff, this note's for you in particular)
06:36 apergos: bringing up storage3 with neither /a nor /archive mounted, saw "The disk drive for /archive is not ready yet or not present" etc on boot, waited a long time, finally skipped them
06:12 apergos: and powercycling the box instead. grrrr
06:05 apergos: rebooting storage3: we have messages like May 6 05:45:12 storage3 kernel: [465081.410025] Filesystem "dm-0": xfs_log_force: error 5 returned. in the log, and the raid is unaccessible, megacli doesn't run either
02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

May 5

09:37 mutante: squids - upgrading in the sq5x range (upload)
08:53 apergos: disabling modcompress temporarily for lightty on dataset2 (live hack), let's see what that does as far as it dying. could be issue similar to http://redmine.lighttpd.net/issues/2391
06:45 mutante: squids - upgrading sq44,48 (upload)
05:23 mutante: squids - finishing a couple reboots in the sq7x range
03:04 binasher: rebooting db1006 as well
03:04 binasher: rebooting db1038, kernel uptime scheduler chaos
02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
00:21 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php

May 4

23:46 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php
23:45 logmsgbot: reedy synchronized php-1.20wmf1/extensions/GlobalBlocking/GlobalBlocking.class.php
22:35 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/backend/FSFileBackend.php 'deployed a807624'
22:34 LeslieCarr: clearing varnish cache and reloading varnish on mobile
21:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
21:13 logmsgbot: reedy ran sync-common-all
20:18 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Fix typo (cswikquote vs cswikiquote)'
20:06 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 writable'
20:05 binasher: performing mysql replication steps for s2 master switch to db52
20:04 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 read-only, db52 (still ro) as master, db13 removed'
19:49 logmsgbot: asher synchronized wmf-config/db.php 'setting db52 weight to 0 in prep for making new s2 master'
19:32 binasher: powering off db24
18:08 LeslieCarr: reloaded mobile varnish caches and purged them
18:02 Ryan_Lane: gerrit upgrade is done
17:55 Ryan_Lane: starting gerrit
17:32 Ryan_Lane: installing gerrit package on manganese
17:28 Ryan_Lane: adding gerrit 2.3 package to the repo
17:25 Ryan_Lane: shutting down gerrit so that everything can be backed up
16:45 apergos: lighty on dataset2 is running under gdb in screen session as root, if it dies please leave that alone (or look at it if you want to investigate)
16:26 notpeter: turning off db30 (former s2 db, still on hardy, will ask asher what to do with it) to test noise in DC
15:50 mutante: rebooting sq67 (bits)
15:42 mutante: going through sq7x servers (text), full upgrades
15:32 notpeter: removing srv281 from rending pool until we figure out what's going on with it
15:23 notpeter: putting srv224 back into pybal pool
15:09 notpeter: removing srv224 from pybal pool for repartitioning
14:56 notpeter: putting srv223 back into pybal pool
14:50 mutante: going through sq6x (text), full upgrades
14:08 notpeter: removing srv223 from pybal pool for repartitioning
14:02 notpeter: putting srv222 back into pybal pool
13:50 notpeter: removing srv222 from pybal pool for repartitioning
13:43 notpeter: putting srv221 back into pybal pool
13:30 notpeter: removing srv221 from pybal pool for repartitioning
13:16 mutante: going through sq80 to sq86 (upload), full upgrade & reboot
12:56 mutante: maximum uptime in the sq* group down to 171 days, so we have like a month now for the rest. stopping upgrades for the moment being.
12:54 notpeter: starting script to move /usr/local/apache to /a partition on all remaing non-imagescaler apaches
12:47 mutante: (just) new kernels & reboot - sq45,sq49 (upload)
12:30 mark: Sending ALL non-european upload traffic to eqiad
12:23 mutante: (just) new kernels & reboot - sq63 to sq66 (209 days up)
12:06 mutante: dist-upgrade & kernel & reboot - sq42,sq43 - rebooting upload squids one by one
11:48 mutante: powercycling srv266 one more time, but now creating RT for it, once already showed CPU issue before it was reinstalled recently
11:13 apergos: restarted lighty on dataset2 ... about ... half an hour ago. stupid case sensitivity
10:02 apergos: tossed knsq1 through 7 from squid_knams dsh nodegroups file, prolly lots more cleanup where that came from
09:34 mutante: dist-upgrade/kernel/reboot: sq37, sq41. rebooting upload squid sq41
08:49 mutante: dist-upgrade & new kernel & reboot: sq33, sq36
07:47 mutante: preemptive rebooting of sq* servers identified as having > 200 days of uptime
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 4 02:22:42 UTC 2012
02:13 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Fri May 4 02:13:58 UTC 2012
00:20 logmsgbot: raindrift synchronizing Wikimedia installation... :
00:18 logmsgbot: raindrift synchronizing Wikimedia installation... : Syncing the PageTriage extension, but only enabling on testwiki
00:08 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Adding 'fr' to language codes for mobile feedback'
00:06 maplebed: moved ms1-3 from the production cluster to the test cluster

May 3

23:29 LeslieCarr: restarting networking on sq55
23:29 logmsgbot: tstarling synchronizing Wikimedia installation... :
23:27 LeslieCarr: restarting networking on sq54
23:24 LeslieCarr: restarting networking on sq53
23:21 LeslieCarr: restarting networking on sq52
23:16 LeslieCarr: restarting networking on sq51
21:30 notpeter: removing srv220 from pybal pool for repartitioning
21:29 LeslieCarr: switching asw-a4-sdtpa from single uplink to lag
21:19 notpeter: putting srv219 back into pybal pool
21:14 logmsgbot: asher synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23'
21:09 logmsgbot: asher synchronized wmf-config/db.php 'reverting cluster23 change'
21:05 logmsgbot: asher synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23'
21:02 binasher: about to move ES writes to cluster23
20:47 notpeter: removing srv219 from pybal pool for repartitioning
20:37 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection/Collection.templates.php
20:37 logmsgbot: reedy synchronized php-1.20wmf1/extensions/Collection/Collection.templates.php
19:50 binasher: restarted profiling collector post parser.php livehack and stats.db removal
19:45 notpeter: starting script to move /usr/local/apache to /a partition on all non-imagescaler, non-jobrunner apaches
19:42 logmsgbot: aaron synchronized php-1.20wmf2/includes/parser/Parser.php 'live-hack out template profiling...again.'
19:40 logmsgbot: aaron synchronized php-1.20wmf1/includes/parser/Parser.php 'live-hack out template profiling...again.'
19:31 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert $wgDefaultUserOptions[enotifwatchlistpages] = 1'
19:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 36316 - Set Add pages I edit to my watchlist to true by default for new users'
19:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php '$wgDefaultUserOptions[enotifwatchlistpages] = 1'
19:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable show update markers for some more of the larger wikis'
19:00 paravoid: powercycling all of sq51-sq62, hanged due to 209 days uptime
18:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36092 - Activation of flood flag on vec.wikipedia.org'
18:43 paravoid: powercycling sq59; inaccessible via either SSH or serial due to load
18:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36183 - Fix namespace alias on Hindi Wikipedia'
18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36171 - Imports from Wikibooks'
18:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
18:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
18:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
18:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36386 - cswikiquote user group changes'
18:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36480 - Create namespace Comments: in Greek Wikinews'
17:44 RobH: db1029 ssd test items removed, can go back to normal service via asher
17:43 notpeter: returning mw58 to pool
17:34 RobH: shutting down db1029 for ssd card testing removal per rt 2766
17:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36320 - Set $wgShowUpdatedMarker back to true on ptwiki'
17:18 notpeter: removing mw58 from pool for more testin'
17:16 LeslieCarr: reloaded and purged varnish cache for mobile in eqiad
17:03 notpeter: mwm59 out of apache pool. using it for some testing
16:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36359 - Add namespace 102 to $wgContentNamespaces on ptwiki Bug 36360 - Add namespace 102 to $wgNamespacesToBeSearchedDefault on ptwiki'
16:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 36460 - Enable chunked uploads as opt-in user preference'
16:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 31406 - Set $wgUseMathJax = true on Wikimedia wikis'
15:12 notpeter: chris is taking down search1-12 to replace with new search nodes
15:05 mutante: powercycling srv266
13:49 mark: Built new wikimedia-base 1.00 package, stripped of most stuff now handled by Puppet, and inserted it into the lucid-wikimedia and precise-wikimedia APT repositories
10:33 mutante: starting container-auditor on ms-be3
08:42 logmsgbot: ariel synchronized php-1.20wmf2/LocalSettings.php 'job runners don't have /home mounted'
08:16 Nemo_bis: siebrand: job queue stuck, on en.wiki jumped from o to 37k in the last ~36h
04:52 jeremyb: fixed complaints of beta simplewiki appearing in #cvn-simplewikis on freenode on the labs side. details
04:00 logmsgbot: tstarling synchronizing Wikimedia installation... :
02:47 logmsgbot: tstarling synchronizing Wikimedia installation... :
02:38 Tim: fixed scap, was failing on the remote side due to mwversionsinuse exiting with status 1 due to /home/wikipedia/common not existing on apaches
02:21 logmsgbot: tstarling synchronizing Wikimedia installation... :
02:21 Tim: aborted scap and re-ran with fanout=5 instead of 30, since nfs1 CPU was maxed out
02:14 logmsgbot: tstarling synchronizing Wikimedia installation... :
02:04 logmsgbot: aaron synchronized multiversion/activeMWVersions 'deployed r115116'
02:00 logmsgbot: LocalisationUpdate failed (php-1.20wmf2) at Thu May 3 02:00:13 UTC 2012
02:00 logmsgbot: LocalisationUpdate failed (php-1.20wmf1) at Thu May 3 02:00:12 UTC 2012

May 2

23:56 logmsgbot: aaron synchronized multiversion/ 'deployed svn HEAD'
23:41 maplebed: started swift old-object-deleter on ms-be3
23:28 maplebed: update - roan takes the blame
23:28 logmsgbot: raindrift synchronized wmf-config/InitialiseSettings.php 'Aborting todays PageTriage deployment'
23:22 maplebed: swift is recovered; ~20 minutes of impaired service. cause unknown, but the swiftcleaner looks likely.
23:18 RoanKattouw_away: Scap tried to push two new source trees to php-1.20wmf1-* and php-1.20wmf2-* , causing full disks. Cleaning up now
23:13 LeslieCarr: restarting nagios bot
22:59 logmsgbot: raindrift synchronizing Wikimedia installation... :
22:49 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'contact us change'
22:48 logmsgbot: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/ 'contact us change'
21:43 logmsgbot: asher synchronized wmf-config/db.php 's2: pulling db30, raising weights on new hosts'
21:02 ^demon: finished database maintenance on db9.reviewdb
20:24 hashar: hashar: updated TestSwarm to distribute tests to Firefox 12 users.
20:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Re-pushing for srv219 and srv220
20:07 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
20:04 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special, wikimedia, wikiquote, wikiversity, and wiktionary wikis to 1.20wmf2
19:59 logmsgbot: asher synchronized wmf-config/db.php 'adding dbs 52,53,57 to s2 at lower weights'
19:55 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
19:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.20wmf2
19:40 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf2
19:36 preilly: fix for PHP Warning: in_array() expects parameter 2 to be array, string given in /usr/local/apache/common-local/php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php on line 156
19:36 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/skins/SkinMobile.php 'fix php notice for in_array'
19:35 logmsgbot: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php 'fix php notice for in_array'
19:34 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikibooks to 1.20wmf2
19:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwikibooks to 1.20wmf2
19:21 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved sourceswiki to 1.20wmf2
19:20 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
19:11 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikisource sites to 1.20wmf2
19:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikinews sites to 1.20wmf2
19:00 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'only remove images for DIGI'
18:51 logmsgbot: asher synchronized wmf-config/db.php 'added ES cluster23 to templateOverridesByCluster but not activating'
18:48 binasher: creating a blobs_cluster23 ES shard table for all active projects
18:31 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikisource to 1.20wmf2
18:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf2
18:24 RobH: updating dns
18:20 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf2
18:09 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'only remove images for DIGI'
17:57 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
17:56 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
17:46 K4-713: updated production civicrm to r1726
17:36 logmsgbot: aaron synchronized php-1.20wmf2/includes/specials/SpecialContributions.php 'Deployed 799998c3a160ef6dd3b926b7d6fec223682b788c'
17:30 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
17:28 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
17:14 logmsgbot: catrope synchronized php-1.20wmf2/skins/vector/ 'Deploying 7260cc5fe4071e03241378ba1a48bc0b6f188948'
16:51 RoanKattouw: Changing docroot/bits/skins-1.19 and other 1.19 symlinks to point to the 1.20wmf1 tree instead. This is needed because we're still getting requests for magnify-clip.png at the 1.19 URL from cached HTML
16:16 notpeter: starting innobackupex from db1040 to db1022 for new s6 snapshot slave
15:31 notpeter: no nagios bot, kicking nagios on spence
15:04 RobH: shutting down mw64 for hw test per rt 1890
15:03 RobH: bellin crashed, unresponsive to ssh or serial console
14:43 mark: Built varnish for precise as 3.0.2-2wm5 and imported it into APT repository precise-wikimedia
11:52 mark: Started distribution upgrade of server stafford from Lucid to Precise
10:41 mutante: refreshLinks.php - started it once again in a screen on hume, just for s1. last cron failed with "mwscript command not found"?? well now it is there again and running
10:09 mark___: Started distribution upgrade of server sockpuppet from Lucid to Precise
09:20 mutante: upgrading bugzilla to 4.0.6
08:43 mutante: kaulen: installing various upgrades (apache,mysql,cron,php-wikidiff2,...)
08:40 logmsgbot: hashar synchronized php-1.20wmf2/includes/GitInfo.php 'Fix Special:Version for 1.20wmf2 (commit ae12df0 , bug 36361 )'
08:20 hashar: cherry-picked ae12df0 commit to 1.20wmf2 since there are mobilefrontend commits pending.
02:35 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 2 02:35:51 UTC 2012
02:32 K4-713: updated production civicrm to r1723
02:13 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed May 2 02:13:30 UTC 2012
01:01 notpeter: starting innobackupex from db57 to db53 for new s2 slave for the one zillionth time

May 1

22:28 logmsgbot_: asher synchronized wmf-config/db.php 'returning db45'
22:23 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db45, last coredb on prior fb mysql build'
22:17 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable doublepage on test2wiki'
22:11 binasher: upgraded percona-toolkit on coredbs to 2.1.1 - now with the potential to run online schema changes on tables without single column unique keys!!
21:39 binasher: created an ops db on all core mysql shards
21:00 notpeter: reinstalling db53. this time with correct raid!
20:40 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Fixing mailto links on mobilefrontend feedback form to properly populate subject lines'
19:32 LeslieCarr: reverting vrrp mastership of row a to cr2-eqiad
19:29 LeslieCarr: switching vrrp mastership of row a to cr1-eqiad
18:32 logmsgbot_: awjrichards synchronized wmf-config/InitialiseSettings.php 'Make testwiki use mobile domain for URLs'
18:28 LeslieCarr: making routing change, higher risk
17:51 Ryan_Lane: make that virt0
17:51 Ryan_Lane: switching the session cache back to filesystem on virt1, since it isn't working properly with memcache
17:29 maplebed: kicking nagios to check a change to fix the mobile LVS alert
17:25 logmsgbot_: nikerabbit synchronized php-1.20wmf2/extensions/TranslationNotifications/ 'Deploying TranslationNotifications code'
17:08 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf2
16:27 notpeter: starting innobackupex from db1034 to db53 for new s2 slave
16:27 notpeter: starting innobackupex from db57 to db52 for new s2 slave
16:03 notpeter: rebuilding db52 and db53 as s2 slaves
15:47 logmsgbot_: asher synchronized wmf-config/db.php 's1: raising db59,60 weights, pulling db52/53 for reuse'
09:23 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'hewiki account creation high throttle limits'
04:04 Tim: on all apaches, running "chmod -R a+rX /usr/local/apache/common-local/" to clean up after killed rsyncs which left files unreadable
02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf2) at Tue May 1 02:23:29 UTC 2012
02:21 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileFeedback.php
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue May 1 02:14:06 UTC 2012
02:06 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileOptions.php
01:51 Ryan_Lane: bringing up all labs instances with a 60 second lag
01:40 Ryan_Lane: rebooting virt0
01:35 Ryan_Lane: rebooting virt3
01:33 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/HtmlFormatter.php
01:26 Ryan_Lane: rebooting virt5
01:18 Ryan_Lane: rebooting virt4
01:03 Ryan_Lane: rebooting virt2
00:51 LeslieCarr: restarted swift-container-auditor on ms-be5
00:38 logmsgbot_: tstarling synchronizing Wikimedia installation... :
00:26 Tim: removed large syslogs from mw60 and ran sync-common
00:18 Tim: on mw60 there was an actual directory at /usr/local/apache/common/php where a symlink should have been. fixed

April 30

23:58 logmsgbot_: aaron synchronized php
23:44 RoanKattouw: Started Apache back up on mw60
23:39 RoanKattouw: Running scap-1 on the Apaches with dsh
23:38 RoanKattouw: Moved /home/catrope/php-1.19 to /home/wikipedia/lazy-backups/php-1.19
23:38 Reedy: mediawiki.org to 1.20wmf2
23:37 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mw.org to 1.20wmf2
23:35 RoanKattouw: Strike that, instead moving /home/w/common/php-1.19 to /home/catrope/php-1.19
23:34 RoanKattouw: Removing /home/w/common/php-1.19 , NFS might freak out a bit
23:31 RoanKattouw: Removed php-1.19 from mw60 , synced it, and restarted Apache
23:28 RoanKattouw: Synced docroot and purged varnish for static-1.20wmf2, bits seems to be working for 1.20wmf2 now
23:27 RoanKattouw: mw60 has full disk, stopping Apache for now
22:50 Ryan_Lane: rebooting virt5
22:42 Ryan_Lane: rebooting virt3
22:35 Ryan_Lane: rebooting virt4
22:28 Ryan_Lane: rebooting virt1
22:23 Ryan_Lane: bringing down all instances (yay gluster)
21:12 pgehres: re-enabled Jenkins jobs on Aluminium after db1008 reboot
21:11 pgehres: CiviCRM back to normal after db1008 reboot
21:07 Jeff_Green: db1008 gets kernel update and reboot
21:00 pgehres: put CiviCRM on Aluminium in maintenance mode for db1008 reboot
20:59 logmsgbot_: reedy synchronized php-1.20wmf2/resources/startup.js 'touch'
20:57 pgehres: disabled all Jenkins jobs on Aluminium in prep for db1008 reboot
20:50 Jeff_Green: db1025 and storage3 get new kernels and reboot
20:28 notpeter: restarting, once again, innobackupex from db1034 to db57 for new s2 slave after fenari crash killed my screen
20:24 Reedy: Running ddsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 '/usr/bin/scap-1' in the hope it syncs all the files that would be nice to be on the app servers
20:18 logmsgbot_: reedy synchronized php-1.20wmf2/cache/ 'Synching whole cache directory'
19:59 notpeter: restarting nagios to get rid of some old checks
19:57 Jeff_Green: payments cluster gets kernel updates and reboots
19:55 logmsgbot_: reedy synchronizing Wikimedia installation... : Rebuiild l10n for 1.20wmf2
19:49 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.20wmf2.php 'Syncing file'
19:49 logmsgbot_: reedy synchronized php-1.20wmf2/LocalSettings.php 'Pushing LocalSettings.php'
19:48 paravoid: upgraded & rebooted ssl3001, ssl3002, ssl3003
19:45 logmsgbot_: reedy synchronizing Wikimedia installation... : Pushing out new symlinks etc, moving test2wiki to 1.20wmf2
19:30 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 live hack revisions'
19:28 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf1 live hack revisions'
19:26 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 for deployment'
19:18 Reedy: Syncing php-1.20wmf2 files from NFS to apaches. Likely to upset NFS (or the uplink for the switch nfs is on) for a little while...
19:14 paravoid: rebooting ssl1004
19:06 paravoid: rebooting ssl1003
19:00 paravoid: rebooting ssl1002
18:59 notpeter: starting innobackupex from db1034 to db57 for new s2 slave
18:50 paravoid: rebooting ssl1001
18:42 Jeff_Green: grosley gets new kernel + reboot
18:35 Jeff_Green: aluminium gets kernel update, yayyyyyyy!
18:34 paravoid: pooled back ssl1; depooling ssl3 and rebooting
18:29 binasher: rebooting mw45 for kernel upgrade
18:27 Jeff_Green: power cycling aluminium which faceplanted
18:22 binasher: rebooting mw45
18:21 notpeter: rebuilding db57 again, this time with more correct raid level!
18:19 logmsgbot_: asher synchronized wmf-config/db.php 'adding db59,60 to s1 with low weights'
18:16 paravoid: depooled & rebooting ssl1
18:09 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Sanity run after script changes.
18:00 logmsgbot_: aaron synchronized multiversion
17:58 logmsgbot_: reedy synchronized php-1.20wmf1/includes/MagicWord.php 'https://gerrit.wikimedia.org/r/6135'
17:44 logmsgbot_: aaron synchronized wikiversions.cdb
17:43 AaronSchulz: updating multiversion code
08:34 mutante: reinstalling srv266
08:08 mutante: upgraded mw1,mw2,mw35
07:59 mutante: reinstalling srv206
07:50 mutante: upgrading mw36
07:37 apergos: powercycling srv266, had this message on mgmt console: Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted
07:22 mutante: installing upgrades on srv212
07:19 apergos: reinstalled srv284, seems to be up now
07:17 mutante: powercycled mw8
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Mon Apr 30 02:13:59 UTC 2012

April 29

20:13 apergos: srv206 won't run puppet, see syslog, clearing out the yaml file didn't help, since it's not urgent I'm leaving it for tomorrow
19:51 Ryan_Lane: depooling ssl3004
19:51 Ryan_Lane: removed the ipv6 addresses from maerlant and added them to ssl3001, then restarted nginx
19:50 Ryan_Lane: repooling ssl3001
19:46 apergos: powercycled mw60, same reason as the rest
19:12 apergos: power cycled mw48 and mw52 (hung just like the others)
18:05 apergos: sll3002 and 3003 were rebooted and are the entire ssl esams pool right now
18:02 apergos: ok the ssl300x situation: ssl3001 is now disabled in the pybal conf file on fenari; it is picking up the ipv6and4labs tmplate and I don't know if that's right, anyways nginx doesn't want to bind to one of those addresses. ssl3004 isn't reachable or pingable even via mgmt but at leasy lvs sees it's gone
16:34 apergos: powercycling the ssl300x.esams hosts. 212 days of uptime... (and 3001 had gone out to lunch)
12:34 mutante: and finally mw1, so just leaving mw1102 and mw60 for having other issues for a while (->Nagios)
12:22 mutante: check_all_memcached recovered, but still same treatment for mw10 and 11 (8 and 15h ago)
12:15 mutante: powercycling mw32,mw33,mw44,mw46 one by one, they were all frozen and went down between like 17 and 24 hours ago approx.
12:07 mutante: powercycling mw30
02:56 paravoid: rebooting ssl2 (has 214 days uptime)
02:47 paravoid: powercycled ssl3
02:13 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sun Apr 29 02:13:58 UTC 2012

April 28

22:53 Reedy: Job queue logs on gdash seem to have stopped on the 26th...
22:29 logmsgbot_: reedy synchronized php-1.20wmf1/includes/EditPage.php 'https://gerrit.wikimedia.org/r/6088'
21:52 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
21:51 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
21:12 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
21:10 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
21:09 logmsgbot_: reedy synchronized common/php-1.20wmf1/extensions/cldr/LanguageNames.body.php 'more debugging'
20:51 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php 'Add debugging'
20:49 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Add debuglog group for language code not being a string'
19:04 logmsgbot_: reedy synchronized php-1.20wmf1/includes/ExternalEdit.php 'https://gerrit.wikimedia.org/r/6077'
19:03 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api/ApiParse.php 'https://gerrit.wikimedia.org/r/6076'
02:24 Ryan_Lane: rebooting all mediawiki boxes that have uptimes affected by the bug are being rebooted at 8 minute intervals
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 28 02:14:14 UTC 2012
01:33 paravoid: powecycled mw29
01:21 paravoid: powercycled mw38
00:17 notpeter: db12 is sooooo sloooooow, starting innobackupex from db1017 to db60 for new s1 slave

April 27

22:15 paravoid: upgraded ssl4 to nginx 0.7.65-5wmf1 and added it back to the pool
21:45 paravoid: rebooting ssl4 after upgrading (incl. a kernel update)
20:00 notpeter: starting innobackupex from db1040 to db1022 for new eqiad s6 snapshot slave, again
19:59 notpeter: starting innobackupex from db12 to db60 for new s1 slave, again
19:58 notpeter: starting innobackupex from db1017 to db59 for new s1 slave, again
19:49 paravoid: de-pooling ssl4
19:30 mutante: test - added new gerrit interwiki prefix for SAL/wikitech - gerrit:6002
19:14 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Fix rights for afttest and afttest-hide groups'
18:25 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Cleanup enotif related settings'
18:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnotifWatchlist to true for all wikis. Leaving wgShowUpdatedMarker set to false for all the big wikis'
16:50 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Simplify enotif code'
16:45 notpeter: starting innobackupex from db1040 to db1022 for new eqiad s6 snapshot slave
16:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgEnotifWatchlist defaulting to true. Big wikis explicitly set to false'
12:25 mutante: fixing integration.mw testswarm and applying fixed erb template by hashar
04:35 Tim: added an account for myself on observium
04:22 logmsgbot_: tstarling synchronized wmf-config/mc.php 'increased wgMemCachedTimeout from 500ms to 3000ms for bug 35900'
02:13 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 27 02:13:51 UTC 2012
00:12 Ryan_Lane: upgrading gluster on all instances
00:09 Ryan_Lane: upgrading gluster on labstore1-4

April 26

23:46 logmsgbot_: asher synchronized wmf-config/db.php 'raising db58 weight'
23:09 Reedy: Recreated resources directory symlinks in bits docroot
21:21 LeslieCarr: started deletion script on ms-be4
19:20 notpeter: restarting puppet on db59
19:18 Ryan_Lane: made LiquidThreads disabled by default on labsconsole, now users must add the special string to a page to enable it there.
19:18 Ryan_Lane: enabled NewUserMessage on labsconsole
19:06 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Add group permissions settings for AFTv5'
18:33 logmsgbot_: catrope synchronizing Wikimedia installation... : Deploy AFTv5 updates
17:17 LeslieCarr: reloaded varnish on mobile caches
14:19 notpeter: cleaned log space on search1017 and search1018 and started lucene
14:04 notpeter: stopping lucene on search1017 and 1018 to take that out of the equation
13:57 mutante: installing some (security) upgrades on fenari (apt,cron,samba,...)
13:54 notpeter: restartin lucene on search1017 and search1018
13:27 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayamon tewiki bug 33480'
13:23 logmsgbot_: nikerabbit synchronized php-1.20wmf1/extensions/Narayam/ 'Updating Narayam'
13:03 notpeter: (re)starting innobackupex from db1017 to db59 for new s1 slave
12:56 mark: Created precise-wikimedia APT distribution
08:27 mark: Power cycled mw40
06:57 binasher: restart pybal on amlvs1 with bgp disabled
06:57 binasher: restarted pybal on amlvs2 with bgp enabled
06:47 binasher: restarting pybal on amslvs2
06:26 binasher: shifting all traffic out of esams
02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Thu Apr 26 02:14:03 UTC 2012
01:42 Ryan_Lane: starting mysql on db46
01:40 Tim: on professor: restarted udpprofile collector
01:37 Ryan_Lane: powercycling db46
01:33 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db46, host down'
00:44 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php

April 25

22:14 LeslieCarr: restarted swift-container-auditor on ms-be3
21:55 RobH: pushing dns update for scs-c1-eqiad and ps1-c#-eqiad
21:22 LeslieCarr: reloading varnish on mobile caches cp1041 cp1042 cp1043 cp1044
21:21 LeslieCarr: clearing mobile varnish cache
19:38 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'Attempted fatal fix'
19:33 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/Math/ 'Deploying 4c9e7dbe761c798ce15d7e2acef829a1582c058b'
19:14 notpeter: starting innobackupex from db12 to db59 for new s1 slave, per mr. feldman's directions
18:56 notpeter: starting innobackupex from db1017 to db60 for new s1 slave
18:49 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/FeaturedFeeds/SpecialFeedItem.php 'Deployed 4fb14a7b2ca9be715b820a9847d999f21c7d2cfc'
18:36 logmsgbot_: aaron synchronized php-1.20wmf1/img_auth.php 'Deployed f7e49bd71bd8356751242c5ce1cbae076a27cf7a'
18:10 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moving all remaining wikis to php-1.20wmf1
17:07 LeslieCarr: reloaded mobile varnish configs
17:06 LeslieCarr: purging mobile cache
16:40 LeslieCarr: starting delete script on ms-be3
16:14 RobH: done moving mgmt connections and serial connections in s8-eqiad for now
16:05 RobH: reshuffling cables in eqiad for serial and mgmt connections in a8, this may affect all eqiad mgmt and serial connections for the next 5 minutes
15:29 hashar: hashar: gallium: MySQL had issues most probably because of the mysql configuration snippets. https://gerrit.wikimedia.org/r/5796 might solve that.
14:03 mutante: gallium - don't start puppet unless the erb template fix for mysql has been merged
13:52 mutante: gallium stopped puppet, moved log_slow_queries config, re-setting up mysql again
13:41 mutante: gallium/testswarm - back up after mysql upgrade and issue starting the service
13:36 mutante: gallium - dpkg-reconfigure mysql-server-5.1, mysql does not start right
13:27 mutante: running apt-get upgrade on gallium
12:29 mark: Sending US, Brazil, Indian traffic to upload.eqiad
11:39 mutante: running authdns-update to add analytics100x and labsdb100x mgmt names
05:35 paravoid: powercycled lvs6, was dead and not responding to serial
03:43 logmsgbot_: asher synchronized wmf-config/db.php 'adding db58 to s7 as a new slave with a low weight'
03:24 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db58'
03:23 logmsgbot_: asher synchronized wmf-config/db.php 'adding db58 to s7 as a new slave with a low weight'
02:28 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Wed Apr 25 02:28:47 UTC 2012
02:14 logmsgbot_: LocalisationUpdate completed (1.19) at Wed Apr 25 02:14:46 UTC 2012
00:02 binasher: profiling collector was pegged at 100% cpu and graphs were turned to swiss cheese due to a bad stats call in 1.20, now fixed

April 24

23:59 binasher: powering off db16
23:55 binasher: streaming hot backup of db1041 to db58 (building a new s7 slave)
23:48 logmsgbot_: aaron synchronized php-1.19/includes/Setup.php 'Hacked out session request stats.'
23:46 logmsgbot_: aaron synchronized php-1.20wmf1/includes/Setup.php 'Deployed 42fcd43299246ecd1b265fcfcdd01a60319cf378'
23:19 AaronSchulz: Running 'mwscriptwikiset maintenance/populateRevisionSha1.php all.dblist' on hume
22:43 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Enabled file change journal on wikis using the new backend config.'
22:20 AaronSchulz: Tables added
22:18 binasher: rebooting db16 with updated kernel. it's probably still hopeless (dimm errors)
22:18 AaronSchulz: Creating the filejournal table on all wikis
21:59 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched commonswiki to the new backend config format.'
21:48 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db16, memory errors'
20:13 apergos: re-enabled replication via cron on ms7, it should catch up within an hour or so
20:10 binasher: reimaged db58 with fixed raid setup, imaging db59
19:51 notpeter: starting innobackupex from db1034 to db57 for new s2 slave
19:50 Ryan_Lane: repooling ssl3001
19:28 Ryan_Lane: depooling ssl3001
18:18 LeslieCarr: deploying to frontend
17:48 notpeter: deploying new squid conf to cp1001 frontend. is just a udp2log port change.
17:19 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Using newer backend for shared repos for testwiki, test2wiki, and mediawikiwiki.'
16:55 logmsgbot_: nikerabbit synchronized wmf-config/CommonSettings.php 'Translate extension configuration changes'
11:54 apergos: after much cursing and kicking zfs, a manual snapshot replication is running in screen as root on ms7 to ms8, expect it to take at least a day
11:44 mark: Sending all non-european upload traffic back to pmtpa to prepare for eqiad varnish storage rework
08:56 mutante: updated blog theme per guillaume (April commits)
08:05 apergos: temporarily disabled automatic zfs replication from ms7 -> ms8, cleared out space on ms8, catching up by hand
04:00 Ryan_Lane: powercycling ssl1
02:47 logmsgbot_: aaron synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Fixed notice spam.'
02:45 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/MobileFrontend/MobileFrontend.body.php 'Fixed notice spam.'
02:37 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Restructed filerepo a config a bit; nothing changed yet.'
02:28 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue Apr 24 02:28:47 UTC 2012
02:15 logmsgbot_: LocalisationUpdate completed (1.19) at Tue Apr 24 02:15:00 UTC 2012
00:15 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/stylesheets/common.css '0be2dc1288361c51f91533f1f77e78d9279b86e0'
00:13 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/common.css 'r115019'

April 23

23:35 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bumpging MobileFrontend resource version'
23:07 logmsgbot_: awjrichards synchronizing Wikimedia installation... :
23:02 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Add code for new URL scheme based on version_compare() logic'
22:51 logmsgbot_: awjrichards synchronizing Wikimedia installation... : MobileFrontend updates per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#23_April.2C_2012
22:33 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobile frontend resource version'
21:49 logmsgbot_: catrope synchronized php-1.20wmf1/resources/mediawiki/mediawiki.js 'Deploy 6e55a770b26b17b8fc9b5b4fe943dcc2867df4f3'
21:27 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'Deploy 93d470b'
20:41 mutante: neon - upgraded libssl, started icinga after adding monitor group
20:32 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/FileRepo.php 'Disabled write checks in the cleanDir() function.'
20:31 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/FileRepo.php 'Disabled write checks in the quickImport/quickPurge functions.'
19:43 logmsgbot_: catrope synchronized php-1.20wmf1/includes/specials/SpecialListgrouprights.php 'Deploy 047543b6805a268c8d689a7a1ce12ec545ef79a9'
18:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
18:43 logmsgbot_: reedy synchronized flaggedrevs.dblist 'Seems I never added ukwiki to the dblist... Oh well'
18:32 logmsgbot_: aaron synchronized wikiversions.dat
18:32 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf1
18:28 logmsgbot_: aaron synchronized php-1.20wmf1/includes/specials/SpecialContributions.php 'Deployed 72969cf8c9a403430c8c93fc20ab3118328c4d9c'
17:06 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use the newer backend config.'
14:33 notpeter: stopping puppet on cp1041 as well
14:17 notpeter: temp stopping puppet on cp1042-1044
13:09 mutante: powercycling frozen mw25, looks like mw21 above but no console output to paste here
13:07 mutante: fix puppet run on spence by removing searchidx1 resources from db9 (was in weird state being in site but also decommissioned)
11:23 mutante: mw21 powercycling mw21 - it died with this http://etherpad.wikimedia.org/mw21
10:55 mutante: force-reload ircecho on manganese to make gerrit-wm rejoin #mediawiki
10:48 hashar: banned CIA bots from #mediawiki IRC channel. It started spamming us with notifications from KDE and mandriva projects. See http://permalink.gmane.org/gmane.science.linguistics.wikipedia.technical/60905
10:30 mutante: searchidx1 was in site.pp and decom.pp at the same time. breaks puppet runs on spence. cannot override local resource. removing from site
10:27 mutante: killed a couple morebots processes on wikitech and it came back by itself :p

April 21

02:29 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 21 02:29:40 UTC 2012
02:15 logmsgbot_: LocalisationUpdate completed (1.19) at Sat Apr 21 02:15:20 UTC 2012

April 20

22:03 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched test2wiki to use the new LocalRepo config style.'
22:01 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched testwiki to use the new LocalRepo config style.'
21:52 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Added NFS backends for local/shared repos; they are not used yet.'
21:12 LeslieCarr: starting swift delete script on ms-be2
20:02 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/file/LocalFile.php 'deployed c77fbd394cda701758ad4523113f567bff7ede66'
19:45 apergos: powercycled mw4, it was unresponsive to pings and via mgmt
18:48 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449'
18:48 logmsgbot_: catrope synchronized php-1.19/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449'
18:07 notpeter: restarting nginx on ssl1002 and ssl1004 as they are not back up
18:01 logmsgbot_: awjrichards synchronizing Wikimedia installation... :
17:31 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Remoev wgArticleFeedbackv5OversightEmails override that was messing things up'
17:15 notpeter: stopping puppet on locke and emery. just to be safe...
17:11 RoanKattouw: Fixed ownership of /h/w/common/php-1.20wmf1/cache/l10n , should be owned by l10nupdate but was owned by reedy
17:01 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36124 - Deploy ProofreadPage extension on test2'
17:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Giving test2wiki moar namespaces'
16:11 mutante: add missing memcached servicegroup to nagios, restarted
15:10 mutante: apache error log on stafford has ruby exceptions re: phusion_passenger
15:01 mark: Converted OSPF directly connected redistributed routes from type 2 to type 1
14:51 mutante: starting swift-container-auditor on ms-be1
14:30 mark: Disabled down-pref of Tampa AS2828 routes
13:14 logmsgbot_: demon synchronized php-1.20wmf1/maintenance/backupTextPass.inc 'Pushing out Idb58ce27 for Ariel/Chris for dumps'
13:10 mark: Sending India upload traffic to upload-lb.eqiad
12:40 mark: Disabled iptables firewalls on internal prod swift cluster servers as it's dropping packets
12:22 mutante: restarted pdns on ns2
11:19 mark: Sending US upload traffic to eqiad as well
10:27 mark: Sending Brazil upload traffic to eqiad
08:39 hashar: Gave up running l10nupdate script it has some file permissions issues. Opened bug 36119 and bug 36120
08:36 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 08:36:53 UTC 2012
08:27 logmsgbot_: LocalisationUpdate completed (1.19) at Fri Apr 20 08:27:36 UTC 2012
08:13 hashar: rerunning l10nupdate for bug 34938
08:02 hashar: running l10nupdate for bug 34938
06:27 pgehres: re-eanabled PayPal on donatewiki and wmfwiki and resumed queue consumer on Aluminium
05:32 LeslieCarr: flushing mobile varnish cache
04:56 pgehres: disabled paypal on donatewiki and disabled queue consumer for duration of PayPal outage
02:33 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 02:33:02 UTC 2012
02:23 logmsgbot_: LocalisationUpdate completed (1.19) at Fri Apr 20 02:23:57 UTC 2012
01:47 logmsgbot_: awjrichards synchronizing Wikimedia installation... : r114983 on wikis still running 1.19

April 19

23:33 binasher: powercycled es1004
21:08 Jeff_Green: changed nagios contactgroup fundraising from tfinc/awrichards --> jgreen
21:03 RoanKattouw: Scap is broken in some weird way, it just stops running after the scap1-skins step. Doesn't run scap-1 (which does the actual sync), doesn't log "sync done", doesn't update graphite
21:01 logmsgbot_: catrope synchronizing Wikimedia installation... : Running scap again, AFTv5 is acting up
19:34 logmsgbot_: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
19:29 RoanKattouw: Running scap to deploy AFTv5 updates, and running AFTv5 schema changes on enwiki at the same time
18:50 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Set wmgArticleFeedbackv5OversightEmails for enwiki'
18:25 notpeter: nothing obvious in logs on db1005, starting mysql
18:15 notpeter: rebooting db1005. it's dead, jim.
17:52 RoanKattouw: Running schema changes for AFTv5 on testwiki
17:51 Jeff_Green: discovered nfs1 had ~1K redundant iptables rules, removed extras and reloaded
17:42 Jeff_Green: discovered sanger had ~7K redundant iptables rules, removed extras and reloaded
13:56 mutante: adding refreshLinks cron jobs to hume per RT-2355 (via puppet). if there should be any performance issues, schedule can be changed like <cluster>@<hour> in mediawiki.pp (and/or remove mediawiki::refreshlinks from hume and clear out the jobs of user mwdeploy)
08:35 mutante: emery - "udp2log_age" says some squid logfiles have not been written to in 6 hours, but from the filenames looks like this isnt a reason to worry, right
07:49 mutante: stat1 - this also needs udp2log stuff fixed. currently Could not find class misc::udp2log::udp-filter
07:47 mutante: gilman - what's up with it? closes SSH, does not like mgmt pass, was running jenkins but broken
07:43 mutante: owa[1-3] They dont have real puppet freshness issues, it's rather firewalling and the snmp traps
02:30 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Thu Apr 19 02:30:33 UTC 2012
02:21 logmsgbot_: LocalisationUpdate completed (1.19) at Thu Apr 19 02:21:31 UTC 2012

April 18

22:55 LeslieCarr: updating exim4.conf on mchenry to not allow old ranges
21:03 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
20:47 logmsgbot_: catrope synchronized php-1.20wmf1/resources/startup.js 'touch'
20:46 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/SyntaxHighlight_GeSHi/ 'Deploying GeSHi fix https://gerrit.wikimedia.org/r/#change,4949'
20:04 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: specieswiki and foundationwiki to 1.20wmf1
19:56 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/LiquidThreads/classes/Hooks.php 'Avoid fatals on invalid title in API'
19:51 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All *wiki wikis to 1.20wmf1
19:25 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote and wikiversity projects to 1.20wmf1
19:22 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikibooks to 1.20wmf1
19:18 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikinewses to 1.20wmf1
19:07 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisources to 1.20wmf1
19:07 logmsgbot_: catrope synchronized wmf-config/mc.php 'Swap out 10.0.2.251 (down) with 10.0.11.24 (spare). This is the last spare, there are now NO SPARES LEFT in mc.php'
19:00 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf1
18:57 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/LiquidThreads/classes/Dispatch.php 'Added type hint for better fatals'
18:44 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikiversity to 1.20wmf1
18:43 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikiquote to 1.20wmf1
18:41 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikibooks to 1.20wmf1
18:40 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf1
18:39 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwiktionary to 1.20wmf1
18:32 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikisource to 1.20wmf1
17:20 logmsgbot_: catrope synchronized docroot/bits/ 'Remove static-1.00 again'
16:57 logmsgbot_: catrope synchronized docroot/bits 'Add docroot/bits/static-1.00 for testing'
16:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wmfUseRevSha1Columns to true for enwiki'
13:30 mutante: applied a patch to etherpad that allows admins to delete pads
12:53 mutante: restarting/fixing etherpad issue
11:08 mark: Sending European bits traffic back to esams
02:30 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Wed Apr 18 02:30:50 UTC 2012
02:21 logmsgbot_: LocalisationUpdate completed (1.19) at Wed Apr 18 02:21:49 UTC 2012
02:13 logmsgbot_: catrope synchronized php-1.20wmf1/README 'Dummy sync to capture which hosts time out on sync-file'
00:52 K4-713: updated production civi to r1631
00:41 Ryan_Lane: adding interface for per-project sudo on OpenStackManager

April 17

23:36 K4-713: updated production civi to r1628
23:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Fixes for cswiktionary changes per Danny B'
22:49 RoanKattouw: That was bug 34885 of course
22:43 logmsgbot_: catrope synchronized php-1.19/extensions/WikiEditor/ 'Deploy fix for bug 348885'
22:38 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Deploy fix for bug 348885'
22:05 K4-713: updated prod civi to r1625
21:51 logmsgbot_: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero needed for carrier testing'
21:42 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'use $wmgUseMathJax'
21:41 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'use $wmgUseMathJax'
21:38 K4-713: queue consumer re-enabled
21:35 K4-713: updated prod civi to r1623
21:32 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php
21:29 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/templates/ApplicationTemplate.php 'ec7c5cc'
21:28 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114947'
21:24 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgUseMathJax on mediawikiwiki'
20:33 logmsgbot_: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.flagging.php
20:26 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/VisualEditor/ 'Deploy VisualEditor beta warning'
19:52 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bump mobile resource version'
19:52 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/
19:51 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/
19:50 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobile frontend resource version'
19:01 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php
18:55 logmsgbot_: reedy synchronized php-1.19/includes/api/ApiQueryBlocks.php 'r114941'
18:53 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
18:47 binasher: returning sq68
18:36 binasher: pulling sq68 from pybal for a bit
18:29 RoanKattouw: Did a graceful restart of all job runners using dsh about 15 mins ago
18:29 RoanKattouw: Restarted morebots
07:44 apergos: morebots test
07:44 apergos: restarted varnish service manually a bit a go on sq67 and sq70, the cron job didn't seem to have gone off. restarted morebots too while I was at it
03:37 Jeff_Green: dist-upgrade arsenic
03:29 LeslieCarr: restarting varnish on arsenic again
03:12 maplebed: started a script to delete old objects on ms-be1 for swift truncated object cleaning
02:53 Jeff_Green: dist-upgrade on strontium
02:43 LeslieCarr: restarted varnish on arsenic
02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue Apr 17 02:26:40 UTC 2012
02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Tue Apr 17 02:17:24 UTC 2012
01:44 LeslieCarr: restarting varnish on niobium
00:52 LeslieCarr: reloading amslvs4
00:27 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo 'deployed 552ff0f482f3e65e9795fe304dd810e9ae1b03fb'

April 16

23:31 logmsgbot_: catrope synchronizing Wikimedia installation... : Now with a touch of the specific WikiEditor.i18n.php file
23:11 logmsgbot_: catrope synchronizing Wikimedia installation... : Hopefully fixing the WikiEditor messages this time, now with MessagesEn.php touch
23:07 logmsgbot_: catrope synchronizing Wikimedia installation... : Hopefully fixing the WikiEditor messages this time
22:58 logmsgbot_: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend to r114934
22:49 logmsgbot_: catrope synchronizing Wikimedia installation... : Need to run scap for this WikiEditor change, contains i18n changes
22:39 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Deploy WikiEditor revert'
20:53 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Actually deploy the recent WikiEditor fixes'
18:58 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Commons Wiki to 1.20wmf1
18:47 logmsgbot_: reedy synchronized php-1.20wmf1/resources/mediawiki/mediawiki.js
18:46 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/WikiEditor
18:37 mutante: manually added iptables nat rules on nfs2
18:13 notpeter: upgrade of udp2log on nfs1/2 complete. should be operating normally now.
17:41 mutante: LDAP on nfs2 warnings - opendj was _just_ started there when puppet was fixed with an unrelated issue
17:38 mutante: restarting opendj on nfs2 because it refused connections
17:08 logmsgbot_: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ 'zero and mobile changes'
16:07 notpeter: upgrading and restarting udp2log on nfs1/2
15:04 mutante: puppet fresh on nfs[12] after removing nonexistent misc::mediawiki-logger class
14:46 mark: Shutdown db24 for memory testing by Chris
13:27 mark: Sending European bits traffic back to pmtpa
12:24 mark: Sending European bits traffic back to esams
12:06 mark: Testing sess_leak_fix2 patch with a snapshot varnish build on cp3001
11:56 Reedy: Ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -- "cd /usr/local/apache/common && sudo -u mwdeploy ln -s php php-1.18" to create symlink for php-1.18
11:51 Reedy: Killing php-1.18 again
11:48 mutante: sq34 - System halted! Error: Internal Storage Slot, powered down, -> RT
11:45 logmsgbot_: reedy synchronized php-1.18/ 'Symlink php-1.18 back to php (our current main running version) as lots of requests on bits are for 1.18 resources'
11:44 mutante: sq34 was broken and died when connecting to mgmt, powercycling
11:37 mutante: nfs1 - Could not find class misc::mediawiki-logger for nfs1
10:57 Krinkle: bits.wikimedia.org back up, mark fixed it.
10:33 Krinkle: bits.wikimedia.org serving Error 503 Service Unavailable on all load.php requests for mediawiki.org and nl.wikipedia.org, maybe more
09:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnableJavaScriptTest to true for test2wiki'
02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Mon Apr 16 02:26:58 UTC 2012
02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Mon Apr 16 02:17:57 UTC 2012

April 15

17:35 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api '/me whistles'
17:20 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api
02:25 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sun Apr 15 02:25:58 UTC 2012
02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Sun Apr 15 02:17:19 UTC 2012

April 14

18:14 mark: Shifting european bits traffic back from esams to pmtpa, session leak is still there
17:08 mark: Shifting european bits traffic back from pmtpa to esams
15:31 mark: Reverted varnish to 3.0.2-2wm4 on cp3001; the race condition patch did not fix the problem
14:56 mark: Sending European bits traffic to pmtpa for testing
13:52 mark: Backported varnish bug #897 patch to varnish 3.0.2, testing a snapshot build on cp3001
11:37 mark: Raised session_max to 300000 (runtime) on cp3001/cp3002
05:58 K4-713: re-enabled the queue consumer on aluminium
02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 14 02:26:55 UTC 2012
02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Sat Apr 14 02:17:34 UTC 2012
02:16 K4-713: updated prod civi to r1616
01:36 K4-713: turned off queue consumption on prod civicrm
01:36 K4-713: updated production civicrm to r1614

April 13

20:53 mark: Rebooting cp3002
20:37 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114889'
17:54 Jeff_Green: created new repo operations/debs/wikimedia-search-qa to stay within package naming conventions
17:31 notpeter: upgrading udplog on locke to 1.8-2 and restarting, etc
17:27 Jeff_Green: created new operations/debs/search-qa repo for packaging search qa scripts
17:17 notpeter: restarting udp2log on emery
12:53 notpeter: restopping puppet on locke/emery
12:09 mark: Deploying varnish 3.0.2-2wm4 and enabling persistent storage on all even numbered eqiad upload varnish hosts
11:46 mark: Imported varnish 3.0.2-2wm4 into the Wikimedia APT repository
02:48 logmsgbot: tstarling synchronizing Wikimedia installation... :
02:39 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Fri Apr 13 02:39:01 UTC 2012
02:20 logmsgbot: LocalisationUpdate completed (1.19) at Fri Apr 13 02:20:35 UTC 2012
01:40 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fix robots file'
01:18 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ 'zero and mobile changes'
01:06 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix html formatter'
00:56 logmsgbot: tstarling synchronizing Wikimedia installation... :
00:08 Ryan_Lane: rebooting ssl1004

April 12

23:39 logmsgbot: tstarling synchronizing Wikimedia installation... :
23:08 logmsgbot: preilly synchronizing Wikimedia installation... : zero rated mobile access changes and mobile frontend updates
21:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34923 - namespace required for PORTAL'
19:46 notpeter: stopping puppet on locke and emery
18:41 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 updates
18:22 Reedy: Ran namespaceDupes against bewiki
18:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34024 - Install ArticleFeedback on es.wikinews'
18:15 logmsgbot: reedy synchronized php-1.19/resources/startup.js 'touch'
18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34024 - Install ArticleFeedback on es.wikinews'
18:11 Reedy: Created AFT tables on eswikinews
17:54 RoanKattouw: Running schema updates for ArticleFeedbackv5 on enwiki
17:46 RoanKattouw: Deploying ArticleFeedbackv5 updates to testwiki and rebuilding localization cache
16:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Allow bnwiki crats to grant/remove import'
16:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35258 - Allow bureaucrats to remove sysop rights on fr.wikipedia'
16:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix imports for wm2012'
16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35917 - allow transwiki imports on wikimania2012'
16:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35666 - Renaming Namespace Wikisource:Author in gu.wikisource'
16:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35694 - Add enotif on page changes in watchlist (guwiki and source)'
16:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35818 - Change of Armenian Wikipedia namespace'
16:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35905 - Change namespaces configuration - pl.wikipedia'
16:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35261 - Add block permissions in rollback on Lusophone Wikipedia'
16:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35823 - Wikijunior and cookbook namespaces for the Vietnamese Wikibooks'
16:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35659 - Set logo for sl.wikiversity'
16:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35853 - Set a non-empty default value for wmgArticleFeedbackBlacklistCategories on WMF wikis'
15:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35878 - Enable e-mail notifications for watchlist (EnotifWatchlist) on tawiki'
15:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35852 - Add a category to $wgArticleFeedbackBlacklistCategories for Portuguese Wikipedia to remove AFT from disambiguation pages'
15:10 mutante: gallium - after files have been deleted/moved, puppet back to normal operation (and new clone directory in Apache)
13:23 mutante: killed puppets on gallium
12:33 mark: repooled ssl1002
12:27 mutante: powercycling frozen ssl1002
12:22 mark: Manually depooled down ssl1002 in pybal
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Thu Apr 12 02:24:29 UTC 2012
02:15 logmsgbot: LocalisationUpdate completed (1.19) at Thu Apr 12 02:15:54 UTC 2012

April 11

22:37 maplebed: deployed more log filters to emery: gerrit/r4758
21:35 LeslieCarr: restarted nrpe on db10
21:33 LeslieCarr: db1004 puppet is fubar
21:33 LeslieCarr: restarted puppet on db30
21:33 LeslieCarr: restarted puppet on mw1110
19:41 notpeter: reimaging bellin and blondel
19:28 logmsgbot: reedy synchronized live-1.5/ 'fix resources symlinks'
19:23 logmsgbot: reedy synchronized live-1.5/ 'fix resources symlinks'
16:54 notpeter: enabling notifications for eqiad lucene vips
16:31 mark: Sending Canadian upload traffic to the eqiad varnish upload cluster
15:59 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 4 to eqiad. for realz this time!'
15:45 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 1 and prefix pool to eqiad. for realz this time!'
15:31 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 2 to eqiad. for realz this time!'
15:15 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 3 to eqiad. for realz this time!'
14:40 notpeter: restarting indexer on searchidx2
13:48 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AbuseFilter/special/SpecialAbuseLog.php
13:35 mutante: applied patch-RT-2804.diff to bugzilla per RT:2804 re: XMLRPC content-type verification
12:07 mutante: moved another list: museum-l -> glam (http://lists.wikimedia.org/pipermail/glam/2012-April/000000.html)
11:58 mark: Setup cp1036 with the persistent storage backend
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed Apr 11 02:26:28 UTC 2012
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Wed Apr 11 02:17:55 UTC 2012
00:11 LeslieCarr: nagios down

April 10

23:50 RoanKattouw: Removed srv187-189 from /etc/dsh/group/job-runners , their jobrunner class has been commented out in puppet since October
23:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'bug 35869 - Add strategywiki as an import source on testwiki'
22:53 RoanKattouw: Trying a graceful restart of the job runner on mw1 by sending SIGHUP to the jobs-loop.sh process
22:53 logmsgbot: catrope synchronized php-1.19/extensions/WikimediaMaintenance/jobs-loop.sh 'r114834'
22:24 logmsgbot: reedy synchronized php-1.20wmf1/extensions/CentralAuth/ 'g4102'
22:23 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AntiSpoof/ 'g4103'
21:20 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling mobile URL template for mediawiki.org (using "mediawikiwiki" this time)'
21:18 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling mobile URL template for mediawiki.org'
21:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf1
21:04 logmsgbot: reedy synchronized php-1.20wmf1/extensions/MobileFrontend/javascripts 'minified JS'
20:55 logmsgbot: reedy synchronized docroot/ 'Fix symlinks'
20:45 logmsgbot: reedy synchronized docroot/
20:35 logmsgbot: reedy synchronized docroot/
20:31 logmsgbot: reedy synchronized live-1.5/
20:24 logmsgbot: reedy synchronized php-1.20wmf1/ 'Resyncing for apaches with no space'
20:23 logmsgbot: reedy synchronized live-1.5 'Fix symlinks'
20:18 Reedy: Deleting php-1.18 from all apaches due to lack of space
20:14 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PrefSwitch/ 'PrefSwitch is needed by SimpleSurvey'
19:35 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding localisation cache for test2/1.20wmf1
19:24 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf1.php 'Sync ExtensionMessages'
19:23 logmsgbot: reedy synchronized php-1.20wmf1/extensions/ 'Would you like some extensions to go with that, sir?'
19:21 LeslieCarr: restarting gmond on db1004 after removing it's 5gig log
19:07 logmsgbot: reedy synchronized php-1.20wmf1/LocalSettings.php 'Push LocalSettings out'
19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf1
19:00 logmsgbot: reedy synchronized php-1.20wmf1/ 'Pushing files for 1.20wmf1'
18:03 logmsgbot: aaron synchronized wmf-config/swift.php 'Catch e bogus empty file names from listings'
14:17 robh: search in eqiad is being reinstalled, no need to be alarmed (thats a pun!)
14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgLanguageConverterCacheType for git deployment later'
11:50 mutante: pxe boot / reinstall cp1029 - cp1036
11:24 mark: Imported varnish 3.0.2-2wm3 into the Wikimedia APT repository
09:30 apergos: restarted slaving on es1003, it will be a bit before it catches up. patience, young nagios
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Tue Apr 10 02:16:58 UTC 2012
01:33 Tim: on sodium: enabling mod_auth on lists.wikimedia.org by running puppet

April 9

23:14 mutante: migrated foundation-l to wikimedia-l (users/passwords/archive urls/settings stay, old mail address & siteinfo redirect)
22:32 logmsgbot: asher synchronized wmf-config/db.php 'returning db12 as enwiki recentchange/watchlist db'
21:39 LeslieCarr: restarted mysql on es1004 and cleared out its disk space
17:49 LeslieCarr: moving es monitoring to nrpe and variables, may cause false pages if i did it wrong :)
17:36 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 35426 - WebFonts on mr.wikisource.org'
14:54 RobH: i killed eqiad search nodes, woooo
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Apr 9 02:17:22 UTC 2012

April 8

08:45 Nemo_bis: Servers have been very slow, almost unresponsive, and network had a drop of ~0.3 Gb/s, at ~8.35-40.
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sun Apr 8 02:16:58 UTC 2012

April 7

17:55 logmsgbot: reedy synchronized wmf-config/codereview.php 'Remove deferred paths'
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sat Apr 7 02:16:54 UTC 2012

April 6

22:23 LeslieCarr: deploying new squid config to all squids
22:14 LeslieCarr: added neon into tiertwo of squid allowed hosts
22:13 LeslieCarr: deploying new squid config to amssq35
21:55 LeslieCarr: restarted puppet on spence
21:35 LeslieCarr: moved jenkins_1.458_all.deb to /srv/wikimedia/incoming/ on brewster
21:32 LeslieCarr: restarted squid on brewster
18:27 Ryan_Lane: updating OpenStackManager to r114758 on virt0
17:33 mark: Sending Japanese upload traffic to varnish in eqiad
17:15 mark: Power cycled down host lvs5
16:43 mutante: changed master and started slave on es1004
15:55 mutante: used gerrit create-project to create operations/debs/wikistats.git
14:13 mutante: manganese (gerrit) now sends SSL CA certificate on https, (curl -vvv says verify ok), should resolve RT:2777 and BZ:35709
11:51 mutante: es1004 - rsync was finished, deleted all binlogs from old host, mysqld_safe& , but did not "change master.." and "start slave" (see mail)
11:39 notpeter: restarting lsearchd on search3... again...
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Apr 6 02:17:37 UTC 2012
01:21 Ryan_Lane: updating OpenStackManager to r114757 on virt0
00:18 Ryan_Lane: updating OpenStackManager to r114754 on virt0

April 5

23:49 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Change guwikisource logo to point to the unscaled file instead'
21:46 notpeter: halting db15 for it to await decom
21:39 binasher: started enwiki.revision sha1 migration on db12
21:32 notpeter: restarting lsearchd on search18
21:22 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12, moving enwiki watchlist,recentchange,etc to db53'
21:19 logmsgbot: asher synchronized wmf-config/db.php 'returning db53'
21:17 logmsgbot: py synchronized wmf-config/lucene.php 'pushing all search traffic back to pmtpa'
18:34 Ryan_Lane: updating OpenStackManager to r114746 on virt0
18:19 Ryan_Lane: updating OpenStackManager to r114744 on virt0
16:49 RobH: brewster puppet running again, cisco installs wont work again until i finish puppetizing the files later today
15:41 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool4 to eqiad. this is the smaller wikis shard'
15:40 notpeter: pointing search pool4 to eqiad (this is the "smaller languages" shard)
15:14 Rob_H: puppet daemon being halted on brewster, i need to make local test changes to dhcp
14:52 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search prefix pool live in eqiad'
14:51 notpeter: pushing search prefix pool live in eqiad
14:51 mutante: gallium - disabled incompatible GitTool plugin on jenkins and restarted it
14:34 mutante: importing jenkins_1.458_all.deb to wikipedia apt repo and upgrading it on gallium
14:08 apergos: started rsync in screen session as root on es1003 copying snapshot from es1001 to /a/
14:04 andrewbogott: created labs account for cneubauer
14:02 logmsgbot: py synchronized wmf-config/lucene.php 'pointing enwiki search and enwiki.prefix at eqiad'
14:00 notpeter: pointing enwiki and enwiki.prefix at eqiad search cluster
13:48 mutante: gallium - upgraded all pear packages
13:45 mutante: gallium - upgraded phpunit and php_codesniffer via pear (have been installed via pear before, distro outdated)
13:43 mutante: gallium - upgrading pear
13:33 mutante: installing package upgrades on gallium. apache,apt,postgres,php5-*,ruby,...various libs
13:24 logmsgbot: py synchronized wmf-config/lucene.php 'pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad'
13:21 notpeter: pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad
12:27 notpeter: search1 and search4 seem to be dead. restarting lsearchd
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Thu Apr 5 02:16:52 UTC 2012
00:33 Ryan_Lane: updating OpenStackManager to r114730 on virt0
00:24 Ryan_Lane: updating OpenStackManager to r114729 on virt0
00:19 Ryan_Lane: updating OpenStackManager to r114728 on virt0
00:12 Ryan_Lane: updating OpenStackManager to r114726 on virt0
00:00 Ryan_Lane: updating OpenStackManager to r114724 on virt0

April 4

22:16 maplebed: deployed (3rd time's the charm!) udp-filter changes to emery for diederik
22:14 logmsgbot: py synchronized wmf-config/lucene.php 'pointing all search back to pmtpa'
22:13 notpeter: flipping all search back to pmtpa (until tomorrow...)
22:00 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback 'r114717'
21:24 cmjohnson1: replacing power cable to psu1 (bottom) es1
21:22 cmjohnson1: replacing power cable to psu1 (top) es1
21:14 logmsgbot: py synchronized wmf-config/lucene.php 'pointing de, fr, and ja search at lvs pool in eqiad for live testing'
21:12 notpeter: moving de, fr, and ja search to eqiad
21:04 cmjohnson1: replacing power cable on labstore2 array psu2 (right side)
21:00 cmjohnson1: replacing power cable on labstore1 array psu1 (left side)
20:57 cmjohnson1: removing power from bottom power supply labstore 2
20:54 cmjohnson1: removing power from top power supply on labstore2
19:44 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wmgArticleFeedbackv5AbuseFiltering on enwiki'
19:40 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable wmgArticleFeedbackv5AbuseFiltering on enwiki'
19:14 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114716'
19:12 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wmgArticleFeedbackv5AbuseFiltering on enwiki'
19:04 RobH: dns update for zhen mgmt
18:54 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying AFTv5 update
18:52 logmsgbot: py synchronized wmf-config/lucene.php 'pointing ru, nl, pl, pt, zh, and sv search at lvs pool in eqiad for live testing'
18:51 notpeter: moving ru, nl, pl, pt, zh, and sv search to eqiad
18:27 mutante: nuked /a contents on es1004, started rsync from es1001
18:16 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add code for wmgArticleFeedbackv5AbuseFiltering'
18:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgArticleFeedbackv5AbuseFiltering, enabled on testwiki only'
17:55 RoanKattouw: Running AFTv5 schema changes on enwiki
17:47 RobH: i didnt crash the site, weeee
17:46 RobH: gracefully restarting apaches
17:46 RobH: pushing out redirects change to apaches for wikipedia.org/com.il redirect to he.wikipedia.org
17:41 binasher: started enwiki.revision sha1 migration on db53
17:38 logmsgbot: asher synchronized wmf-config/db.php 'returning db52, pulling db53'
17:32 RobH: update done, all nameservers still online
17:31 RobH: dns update for wikipedia.org/com.il being resolved
17:08 RoanKattouw: Applying AFTv5 schema change on testwik
15:30 logmsgbot: py synchronized wmf-config/lucene.php 'pointing eswiki search at lvs pool in eqiad for live testing'
15:28 notpeter: pointing eswiki search at eqiad
12:51 mutante: db1007 - add mysql startup via 'update-rc.d mysql defaults'
12:42 apergos: started mysqld on db1007 via /etc/init.d/mysql (this doesn't seem to point to a special fb build, and can't seem to find one on this host, what's up with that?)
12:31 apergos: rebooted bd1007, it was dead in the water (also no helpful messages on console, bah)
11:16 mutante: enabled Renameuser extension on wikitech, renamed tchay per RT request, disabled extension again (it was installed but disabled)
02:19 logmsgbot: LocalisationUpdate completed (1.19) at Wed Apr 4 02:19:03 UTC 2012
01:50 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileOp.php 'deployed r114697'
01:39 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'

April 3

23:17 LeslieCarr: updating bgp policies on cr1.sdtpa
22:44 LeslieCarr: reinstalling neon
22:04 maplebed: rolled back changes to emery in udp-filter due to the new binary crashing.
21:50 maplebed: ran /etc/init.d/udp2log reload on emery to enact the puppetted changes
21:41 maplebed: deploying new udp-filter and teahouse filters to emery for diederik
20:13 notpeter: restarting lsearchd on search7. was taosted
18:37 logmsgbot: root synchronized wmf-config/mc.php
18:37 RobH: syncing new mc.php, forgot to check for all three of the servers i took down, opps.
18:28 RobH: shutting down mw28, mw49, & mw58 for rack relocation due to power overload in d2-pmtpa, relocation to d1-sdtpa per rt 2692
17:59 K4-713: Synchronized payments cluster to r114642
17:52 logmsgbot: reedy synchronized php-1.19/extensions/MobileFrontend/
17:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix commas in guwikisource namespaces'
17:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix commas in guwikisource namespaces'
16:38 RobH: bringing down srv237 for phase balancing
16:37 RobH: srv230 back in rotation
16:26 RobH: shutting down srv230 for power phase move per rt 2759
16:10 RobH: updating brewster to use new dhcp files for cisco, no more local hackin.
15:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35482 - Add Patroller & Autopatroller groups on ml.wikisource'
15:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35482 - Add Patroller & Autopatroller groups on ml.wikisource'
15:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35545 - Grant the abusefilter-log-detail right to patrollers on Commons'
15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35545 - Grant the abusefilter-log-detail right to patrollers on Commons'
15:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35624 - Subject namespace for the Vietnamese Wikibooks'
15:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35603 - Enable Transwiki import on KN:WP'
15:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35581 - Closure of nz.wikimedia.org'
15:15 logmsgbot: reedy synchronized closed.dblist 'Bug 35581 - Closure of nz.wikimedia.org'
13:35 Tim: manually reloaded rsyslogd on all apaches
06:16 Tim: deploying limited/split apache syslog (https://gerrit.wikimedia.org/r/#change,4149)
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Tue Apr 3 02:16:32 UTC 2012
00:37 logmsgbot: aaron synchronized php-1.19/includes/Block.php 'deployed r114672'

April 2

23:54 Tim: restarting all apaches with apache-restart-all-hard
23:51 logmsgbot: tstarling synchronized php-1.19/extensions/ConfirmEdit/FancyCaptcha.class.php
23:37 logmsgbot: tstarling synchronizing Wikimedia installation... :
23:36 maplebed: cleared the varnish cache for preilly
23:34 Tim: on all apaches: running logrotate -f and deleting the resulting backup syslog files, to free up disk space
23:32 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114673'
23:21 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version number'
23:05 logmsgbot: awjrichards synchronizing Wikimedia installation... : Deploying MobileFrontend changes at r114671 per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#2_April.2C_2012
21:43 maplebed: reverted changes to emery's logging due to a broken package in the deploy.
21:30 LeslieCarr: turned down ms7's secondary ethernet port to prevent the flapping (stupid sun boxes)
19:51 maplebed: deploying new udp-filter to emery rt-2501 gerrit/r4120
19:51 notpeter: running authdns-update on dobson
18:30 RobH: brewster puppet daemon stopped, doing local hacks
18:17 RobH: removed old bin files on db1004 and prolly borked it by removing the wrong files
17:54 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php '35436 - Enable Narayam at Hindi Wikipedia'
17:47 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add disable images option for default on zero domain'
17:45 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 35328 - Enable WebFonts for fr.wikisource.org'
17:40 logmsgbot: nikerabbit synchronized php-1.19/languages/Names.php 'I18ndeploy r114656'
17:15 preilly: carrier testing push for DIGI
17:15 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
16:46 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Mon Apr 2 02:16:47 UTC 2012

April 1

02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Apr 1 02:17:22 UTC 2012

March 31

10:22 mutante: srv222,225 were also upgraded but stopping there for now in favor of reinstalls
09:58 mutante: nuked /usr/shared/doc on a couple srv's, hey at least 700MB or something, and yes we really should reinstall with a decent partitioning scheme as M ark said
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 31 02:18:10 UTC 2012

March 30

19:37 hashar: configured jenkins on gallium to use smtp.pmtpa.wmnet as outgoing SMTP server
19:28 RobH: puppet daemon restarted on brewster
18:13 RobH: killing puppet daemon on brewster, i need to hack at local configuration for cisco server stuff
12:56 mutante: db1047 - added system startup for /etc/init.d/mysql
12:47 mutante: powercycling db1047
12:28 mutante: deleted old kernel sources on upgraded srvs for that little extra space during peaks, suggesting to nuke /usr/share/doc if there should be more disk space warnings
10:41 mutante: same for srv223
09:18 mutante: srv224,srv219,srv220, upgrade apache, dist-upgrading w/ kernel, disabling ureadahead, rebooting one by one
08:06 mutante: storage3 - gmond unable to find the metric information for any mysql_* .."module has not been loaded", starting mysql, running puppet ...
07:57 mutante: powercycling storage3
07:03 Tim: running bug 35578 cleanup script in screen on fenari
06:41 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
06:40 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
06:39 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
06:15 Tim: killed vi on fenari owned by awjrichards, locking CommonSettings.php for two days
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 30 02:17:56 UTC 2012
01:13 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove more crap'
01:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove some dupe code'
01:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove wmgUsabilityPrefSwitch'
00:59 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove wmgUsabilityPrefSwitch'
00:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove unused wmgUseUsabilityInitiativeAlpha'

March 29

23:49 logmsgbot: aaron synchronized php-1.19/includes/revisiondelete/RevisionDeleteUser.php 'deployed r114619'
21:20 LeslieCarr: rebooting db47
20:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Swap wgUseCommaCount to wgArticleCountMethod'
20:07 notpeter: restarting lsearchd on search2 to del the logfile to end all logfiles
20:05 RoanKattouw: Stopping and starting Gerrit on manganese to apply Chad's change of the -1 text in the DB
20:02 notpeter: restarting lsearchd on search7 to del the logfile to end all logfiles
18:11 logmsgbot: catrope synchronized php-1.19/extensions/ClickTracking/ClickTracking.hooks.php
17:59 RobH: search1021 coming back up, done with tests
17:53 RobH: search1021 coming down for ssd fit test
17:07 notpeter: disabling notifications for search lvs nagios checks for 24 hours to test fix
15:42 notpeter: finished clearning up all pmtpa search hosts. hey look! they all have lots of space now!
15:15 notpeter: restarting lsearchd on search3
15:02 RobH: brewster puppet re-enabled
15:02 RobH: virt1001 pxe boots via dhcp and fails tftp download, i have to hold off on further troubleshooting until i have a network admin
14:47 RobH: did virt1001 wrong, reupdating dns
14:39 RobH: all nameservers still online after udpate
14:37 RobH: updating dns for virt1001 testing
14:29 RobH: stopping puppet runs on brewster so my hacking at the dhcpd.conf file won't get overwritten until I have it working right
14:01 Jeff_Green: restarted varnish on on cp3002 because it was thrashing futiley
13:45 notpeter: rebooting (mostly) down cp3001
13:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add participation namespace to metawiki per request'
13:11 notpeter: trimming logs and such on search1-20
09:59 mutante: srv221, disabling ureadahead, installing package upgrades and new kernel, rebooting
09:40 mutante: kill and start lsearchd on search7
09:36 mutante: restarted defunct lsearchd on search6
09:10 mutante: gallium - added demon,hashar,reedy to group jenkins as it's a problem using puppet when users and groups already exist
06:25 mutante: powercycling sq40
06:21 mutante: installed more package upgrades on sodium
05:58 mutante: installed security upgrades on brewster, cadmium, capella (apache,mysql,ruby,apt..)
05:49 mutante: db42 - mysql did not autostart after boot, added using update-rc.d
05:42 mutante: db42 - reboot worked despite the grub warning about unreliable blocklists
05:37 mutante: rebooting db42 to finish upgrades
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 29 02:17:53 UTC 2012

March 28

23:27 Tim: running apt-get upgrade on mw22,mw66,srv193,srv250,srv253,srv236
23:25 Tim: cleaned up stuck apt-get process on srv236
23:22 Tim: cleaned up stuck apt-get processes on mw22,mw66,srv193,srv250,srv253
21:44 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobile frontend resrouce version'
21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/application.min.js 'r114576'
21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/banner.min.js 'r114576'
21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114576'
21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114576'
21:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/application.js 'r114576'
21:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/banner.js 'r114576'
20:43 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 28 20:43:20 UTC 2012
20:29 notpeter: restarted search1020. nothing conspicuous in logs
19:56 RoanKattouw: Running a patched version of l10nupdate that rebuilds the localization cache
18:49 logmsgbot: catrope synchronizing Wikimedia installation... : Bugfixes for ArticleFeedbackv5, ArticleFeedback and ClickTracking
16:47 cmjohnson1: msw1-d1-pmtpa replacement complete
16:34 cmjohnson1: replacing msw-d1-pmtpa per rt2639
15:36 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech'
15:34 Reedy: srv221 is full
15:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech'
14:39 RobH: restarted morebots in screen on wikitech, no longer as catrope, as roan has root on that box
14:36 RobH: got virt1001 to pxe, but dhcp doesnt know how to handle, need subnet details.
14:34 notpeter: lucene hosed on search9 and search15. restarting, then will look after cause
13:14 Jeff_Green: restarting puppet/puppetmaster on stafford to experiment with report settings
02:10 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 28 02:10:34 UTC 2012

March 27

23:12 logmsgbot: tstarling synchronized php-1.19/cache/trusted-xff.cdb
20:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33789 - Enable botadmin usergroup on ml.wikipedia'
19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix lezwiki namespace'
19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove ruwiki arbcom talk from namespaceprotection'
19:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
18:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
18:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35290 - Create Slovenian Wikiversity'
18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35290 - Create Slovenian Wikiversity'
17:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
16:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
16:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
16:48 logmsgbot: reedy ran sync-common-all
16:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'prep work for new wikis'
16:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34527 - Create a Arbcom namespace on Russian Wikipedia'
16:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34527 - Create a Arbcom namespace on Russian Wikipedia'
15:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35161 - Incubator configuration updates'
15:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32825 - Favicon for siwiki'
14:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35516 - Add Skin: namespace to MW.org'
08:15 apergos: test you silly morebot

07:59:56 hashar: archived old server admin logs since the old page was too long for my connection to download :-/
06:59:02 apergos: !log powercycled emery, it was unresponsive via the mgmt console and not pingable
02:17:52 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 27 02:17:52 UTC 2012
00:56:51 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114507'
00:55:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/LeaveFeedbackTemplate.php 'r114508'
00:42:50 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bmping resource version for MobileFrontend'
00:41:58 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r114509'
00:37:30 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version #'
00:36:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/MobileFrontendTemplate.php 'r114507'
00:36:09 logmsgbot: awjrichards[00:36:36] synchronized php/extensions/MobileFrontend/templates/LeaveFeedbackTemplate.php 'r114508'
00:35:50 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r114508'
00:08:55 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114506'

March 26

23:18:17 logmsgbot: awjrichards synchronizing Wikimedia installation... : Syncing MobileFrontend to r114504 changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#26_March.2C_2012
22:44:53 RobH: !log also rolling firmware to ps1-d[1|2|3]-pmtpa
22:28:10 RobH: !log pushing firmware updates to servertechs in sequence: ps1-[a2|a3|a4|a5|b2|b3|b4|b5|c1|c2|c3|d1|d2|d3]-sdtpa, disregard any errors from rebooting alerts
19:55:09 notpeter: !log stopping puppet on search6 and search15 for 24 hours to test new log rotation script
19:19:35 RobH: !log cp1019 memory replaced per rt 2651
19:07:14 apergos: rebooting ms1001 (new kernel)
17:53:34 RobH: cp1019 coming down for memory replacement per rt 2651
17:51:39 RobH: fluorine disk upgrade done, os install pending, details on rt 2350
17:43:48 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r114492'
17:36:51 RobH: fluorine coming down for new disks
17:14 notpeter: backingup plwiki.nspart1 index on search7, deleting working copy, and restarting lsearchd. (note: this will probably cause some downtime on some languages while the proc restarts...)
15:18 RobH: db59 has errors, but as it was a fusion io testbed server, it is more than likely tweaked for such, it is not in any rotation
14:54 RobH: db59 shutting down for io card removal per rt 2589
13:37 mutante: while on it, installing a whole bunch of package updates on db42
13:25 mutante: db42 was out of disk , caused by ~5G citations.csv in /tmp, gzipped the file
09:59 mutante: ..and on ms-be-3. running puppet on db59
09:43 mutante: another corrupted .yaml file on ssl2
09:33 mutante: brewster - delete puppet lock file, restart lighttpd, puppet ...
09:05 mutante: brewster was out of disk - deleted lighttpd access.log.1, gzipped access.log
08:24 mutante: on several mw* boxes puppet did not run because .yaml files on the puppetmaster became corrupted. need to delete the $hostname files in /var/lib/puppet/yaml/node on stafford and re-run. puppet bug similar to http://projects.puppetlabs.com/issues/7836
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 26 02:18:03 UTC 2012

March 25

22:26 RobH: row b servertech firmware in eqiad all updated, should clear alarms as they come back online
22:18 RobH: firmware updates on servertechs in row b eqiad, disregard alarms
20:14 RobH: to fellow ops, you can disregard those observium errors, as I caused them
20:13 RobH: firmware updated on all power strips in row a eqiad.
16:22 RobH: ps1-a1-sdtpa firmware update complete
16:15 RobH: updating firmware on ps1-a1-sdtpa
16:14 RobH: ps1-b1-sdtpa firmware updated successfully
16:14 RobH: ps1-a1-eqiad firmware updated successfully
16:09 RobH: updating firmware on ps1-s1-eqiad and ps1-b1-sdtpa
16:07 RobH: updated firmware successfully on ps1-a8-eqiad, if it has observium alarms now then there are bigger issues.
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 25 02:17:21 UTC 2012
00:59 LeslieCarr: admin down asw-a-eqiad xe-1/1/2 and cr2-eqiad xe-5/0/0 due to framing errors causing packet loss and lacp sporadic timeouts. source of the issue

March 24

19:46 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
19:46 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
17:35 mark: Migration from br1-knams to cr2-knams completed.
17:09 mark: Migrated second knams-esams dark fiber link from br1-knams to cr2-knams
16:36 mark: Corrected MTU setting on cr2-knams's AMS-IX interface
16:20 Reedy: Some european users reporting oruting issues
16:01 mark: Cleared OSPF session between csw1-esams and csw2-esams which magically made some internal routes reappear
15:40 mark: Brought up AMS-IX ipv4 BGP sessions
15:30 mark: Brought up AMS-IX ipv6 BGP sessions
15:25 mark: Moved AMS-IX connection to cr2-knams:xe-1/1/0
15:22 mark: Shutdown all AMS-IX BGP sessions
15:06 mark: Disabled BFD on OSPF3 between cr2-knams and csw1-esams
14:49 mark: Moved AS6908 and AS1257 PIs to cr2-knams
14:18 mark: Brought up AS13030 and AS1299 BGP sessions on cr2-knams
13:57 mark: Shutdown AS1299 BGP session on br1-knams
13:14 mark: Established full iBGP mesh with added router cr2-knams. cr2-knams now has full Internet connectivity.
12:48 mark: Moved fiber from br1-knams:e1/2 to cr2-knams:xe-0/0/0
12:44 mark: Disabled br1-knams:e1/2 (DF leg 1 to esams)
12:43 mark: Rack mounted and powered up cr2-knams
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 24 02:17:02 UTC 2012

March 23

23:49 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ 'r114466'
23:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia'
23:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia'
23:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce'
23:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce'
23:07 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce'
23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce empty arrays'
22:24 RobH: scs-a1-eqiad back online
21:58 RobH: scs-a8-eqiad coming down for re-grounding
19:51 RobH: all power strips in eqiad are now properly grounded
18:12 maplebed: removed ms1 and most of ms2 from the production swift rings. no effect expected.
18:04 logmsgbot: asher synchronized wmf-config/db.php 'returning db32, pulling db52 for migration'
16:44 RobH: cp1019 in middle of firmware update, please dont touch
16:44 RobH: cp1017 memory error seems ot have cleared post firmware update, will keep an eye on it for the rest of the day
16:09 RobH: raid rebuilding on magnesium, however swift stuff is kind of black box mystery right now to me, need Ben to review magnesium later for that
15:53 RobH: magnesium coming back online
15:44 RobH: shutting down magnesium for disk swap
15:37 RobH: firmware updating on cp1017, no one touch it please
15:30 RobH: db1020 can go back into whatever rotation Asher wants it in
15:29 RobH: db20 memory error on raid controller resolved with firmware updarte
06:39 logmsgbot: tstarling synchronized php-1.19/includes/filerepo/file/LocalFile.php 'r114442'
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 23 02:18:35 UTC 2012
01:55 mutante: deleting puppet report files older than 60hours on stafford to free disk space

March 22

23:30 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
23:18 RobH: db1020 firmware still updating, will check on it later tonight. offline until then
22:19 notpeter: all 3 dns servers are responding to digs after reload
22:10 notpeter: pushing a new zone file to add 2 more search-related vips for eqiad
20:52 notpeter: stopping puppet on brewster temporarily
20:25 notpeter: rebuilding search1015 and 1016 for disk shuffles
20:01 RobH: magnesium goign down and up again, troubleshooting the disks
19:47 apergos: rebooting ms1002, had stuck rsyncs, and kswapds at 100% cpu, weirdness like "ls /export/upload/wikipedia/am/0/00" hanging.
18:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
15:45 RobH: search 1015 and search1016 back up with added disks
15:08 RobH: shutting down search1015 & search1016 for hdd additions
14:45 RobH: db1020 still offline, requires firmware update on raid controller per rt 2621, will perform later today
14:33 logmsgbot: reedy synchronizing Wikimedia installation... :
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 22 02:17:47 UTC 2012
01:14 K4-713: Re-enabled the donations queue consumer in Jenkins
00:28 binasher: started enwiki.revision alter on db32
00:26 binasher: disabled lvm snapshots and puppet on db32 for revision sha1 alter
00:24 logmsgbot: asher synchronized wmf-config/db.php 'pullin db32 for revision alter'

March 21

22:27 ^demon|away: wmf-deployed extensions now r/o in SVN
21:52 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php
21:27 Ryan_Lane: bringing up all instances on virt3
21:08 cmjohnson1: swapped 2 DIMMS in virt3 (b2 and b5)
21:01 Ryan_Lane: shutting down virt3 to replace dimms
20:47 ^demon: /trunk/phase3 is now r/o in SVN
20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable prefswitch'
20:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgArticleFeedbackv5OversightEmails on enwiki'
18:59 maplebed: rebooted ms-be3 after it crashed.
18:51 binasher: brought db24 back up after hang, and reslaving, but leaving out of db.php. just replicating until a replacement s2 snapshot host is built
18:51 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 update
18:46 logmsgbot: asher synchronized wmf-config/db.php 'returning db36'
18:35 logmsgbot: asher synchronized wmf-config/db.php 'pulling db24, failing hw'
18:03 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php
18:01 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily disable ShortUrl on testwiki because we think it might conflict with ArticleFeedbackv5'
17:59 K4-713: updated and synchronized payments cluster to r114382
17:47 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
12:25 notpeter: disabling notifications for search-pool1
08:58 mutante: rebooting ms-be4
08:37 mutante: stopped/started lsearchd on search9
08:05 mutante: ms-be4 down but cant powercycle it yet..Unable to establish LAN session / ipmitool /ipmi_mgmt
07:58 mutante: restarted lsearchd on search3 and 9
05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/CoreParserFunctions.php
05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/Parser.php
05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/StripState.php
05:22 logmsgbot: tstarling synchronized php-1.19/tests/parser/parserTests.txt
03:51 mutante: added "lez" to langlist and running authdns-update, for lez.wikipedia per RT-2665
03:29 mutante: magnesium - shutting down, has existing RT-2669 to replace disk
03:18 mutante: magnesium - "..drive on port B of the Srial ATA controller is operating outsde of normal specifications.. Strike F1 key to continue"..
03:16 mutante: powercycling magnesium - down and just "init: tty4 main" on mgmt, frozen
03:10 mutante: running puppet on aluminium
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 21 02:18:10 UTC 2012
01:06 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114342'
00:25 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo'
00:03 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo'

March 20

23:19 Ryan_Lane: fixing the zero redirect
22:46 logmsgbot: reedy synchronized wikipedia.dblist 'test'
22:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/api/ApiQueryExtracts.php 'r114319'
22:09 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping resrouce version # for MobileFrontend'
21:54 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#20_March.2C_2012
21:46 binasher: stopped eqiad bits servers from udplogging to emery, packet loss is back to zero
20:59 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
20:17 binasher: killed enwiki.revision sha1 migrator (upgrade-1.19wmf1-2.php). after db36 completes, will run the rest by hand
19:52 Ryan_Lane: pushing change for zero.wikipedia.org to redirect to the english message
19:41 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page'
19:16 cmjohnson1: pulling disk 5 on virt1 for reseating
18:34 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page'
18:02 pgehres: flipped Template:CC-status on wmfwiki since credit cards are still disabled on payments.wikimedia.org
17:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35193 - Enable sub page feature in Telugu Wikisource'
17:49 notpeter: restarting lsearchd on search10
17:30 logmsgbot: aaron synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed r114285'
17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that then'
17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Test something for sewikimedia'
16:42 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Bug 35355 - Reset of permissions for Hindi Wikipedia'
16:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove hiwiki botadmin from whGRoupsRemoveFromSelf'
15:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
15:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
15:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
15:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35161 - Incubator configuration updates'
15:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31209 - Enable the WikiLove extension for incubator'
14:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove more group dupes'
14:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35355 - Reset of permissions for Hindi Wikipedia (hiwiki)'
14:14 logmsgbot: reedy synchronizing Wikimedia installation... : sscapping for r114268
14:08 logmsgbot: reedy synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'r114268'
09:12 mutante: new URL pointing to Wikipedia Education Program - http://education.wikimedia.org
08:59 mutante: several srv's said they were unable to contact NTP server
08:57 mutante: apache-graceful-all to deploy changed redirects.conf
08:53 logmsgbot: tfinc synchronized wmf-deployment/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'Fixes file pages showing data charge warnings'
07:42 mutante: running authdns-update after adding education.wm for redirect RT:2634
06:21 logmsgbot: tstarling synchronized php-1.19/includes/User.php
05:35 logmsgbot: asher synchronized wmf-config/db.php 'pulling db36 durring db migration'
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 20 02:17:55 UTC 2012
00:25 logmsgbot: awjrichards synchronizing Wikimedia installation... : Reverting MobileFrontend to r113973
00:15 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114221'
00:07 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling zero rated mobile access everywhere'
00:05 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumpging version number for MobileFrontend resources'

March 19

23:54 logmsgbot: awjrichards synchronizing Wikimedia installation... : Redoing accidentally aborted scap, Pushing changes to MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#19_March.2C_2012
23:51 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing changes to MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#19_March.2C_2012
23:35 AaronSchulz: fixed a few files, on commons and other wikis, with empty oi_archive_name values even though the file was on NFS
23:20 Ryan_Lane: restarting all nginx servers
23:20 Ryan_Lane: added a new proxy to the ssl configuration to temporarily proxy access to wikimania videos being transcoded
21:38 binasher: creating "ops" db and related grants on prod db clusters 2-7 to prep rollout of ishmael / pt-digest beyond s1
21:17 binasher: started enwiki.revision sha1 alter on production side
20:57 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Removing debugging code from MobileFormatter'
20:54 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
20:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
20:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
20:31 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Adding debugging code to MobileFormatter'
20:07 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r114176'
19:41 Ryan_Lane: bringing virt3 instances back up
19:33 binasher: deploying new frontend squid conf to add support for mf_useformat cookie [rt 2645]
19:18 K4-713: CiviCRM 4.1.1 update script finished executing on prod.
19:12 Ryan_Lane: shutting down virt3 for memory reseating
19:09 K4-713: Started the CiviCRM 4.1.1 update script on prod.
19:08 mark: Rebuilding RAID arrays on brewster
18:58 K4-713: Put production civicrm / drupal instance in offline mode for upgrade
18:54 K4-713: Disabled all production CiviCRM Jenkins jobs, for CiviCRM upgrade.
18:54 cmjohnson1: brewster HDD replacement complete
18:42 mark: Shutting down brewster for HDD replacement
18:26 Jeff_Green: killed kill-slow-queries on db1008 for the duration of the civicrm upgrade
18:19 logmsgbot: nikerabbit synchronized php-1.19/includes/Linker.php 'i18ndeploy r114160'
18:19 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/resources/ext.webfonts.fontlist.js 'i18ndeploy r114160'
18:14 mark: Running smartctl -t long /dev/sdb on brewster
12:58 logmsgbot: hashar synchronized php-1.19/includes/SiteStats.php 'Reenable SiteStatsInit::articles() for bug 35169. SiteStatsInit::doAllAndCommit() still disabled since it breaks the site'
10:28 logmsgbot: tstarling synchronized wmf-config/PoolCounterSettings.php 'increased max queue from 50 to 100 on reports that the limit was reached on the enwiki main page in normal operation'
09:11 mutante: nomcom and langcom wikis look kind of broken , redirecting to pages on incubator with "Error: This page is unprefixed! "
08:49 mutante: making (almost) all private wikis https-only per RT-2565, vi remnant.conf,sync,graceful...
07:30 mutante: running sync-apache after making a change to remnant.conf to make grants.wm https-only
05:09 Ryan_Lane: bringing up most instances on virt3, doing so by project priority
04:42 Ryan_Lane: bringing up all instances on virt4, waiting 30 seconds between instances
04:25 Ryan_Lane: bringing up all instances on virt2, waiting 30 seconds between instances
04:09 Ryan_Lane: bringing up all instances on virt1, waiting 30 seconds between instances
04:00 Ryan_Lane: attempting to bring some instances up
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 19 02:17:17 UTC 2012
01:15 mutante: killed, updated, restarted wikibugs bot per request in RT:2656, should have fixed bugzilla:18831

March 18

23:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35308 - Install mw:Extension:DynamicPageList (Wikimedia) on Portuguese Wikipedia (ptwiki)'
19:20 Ryan_Lane: stopping all labs instances, manually recovering gluster volume
15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35295 - Missing a in abusefilter-hide-log permission for oversighters'
10:49 Ryan_Lane: rebooting virt4 thanks to defunct libvirt process
03:43 Ryan_Lane: bringing all labs instances up
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 18 02:18:51 UTC 2012
01:09 Ryan_Lane: rebooting all of the virt hosts, gluster is having major issues
00:43 Ryan_Lane: rebooting virt2
00:40 Ryan_Lane: restarting glusterfs on virt2
00:11 Ryan_Lane: rebooting virt3 libirt is non-responsive
00:00 Ryan_Lane: bringing up instances that were downed on virt3

March 17

23:50 Ryan_Lane: virt3 crashed, powercycling it
23:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove old comments'
23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove old comments'
23:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat'
23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat'
23:02 logmsgbot: catrope synchronizing Wikimedia installation... : Have to scap for that AFTv5 change to propagate i18n change
22:52 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ 'r114087'
21:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35289 - Add wikisource logo to mobile wikisource gateway'
02:21 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 17 02:21:03 UTC 2012
01:23 AaronSchulz: FindFilesMissingDBRows.php done, list under aaron/output/missingFileDBRows
00:11 AaronSchulz: Running FindFilesMissingDBRows.php on all wikis

March 16

21:21 binasher: running enwiki.revision sha1 schema migrations on eqiad side
20:12 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild moodbar messages
20:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable moodbar on enwiki'
19:53 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ 'r114030'
19:15 Reedy: Ran namespaceDupes on stewardwiki
17:11 RobH: hdd in search1017/1018 replaced per rt 2583
16:54 RobH: search1017 and search1018 coming down for hdd swap
16:53 RobH: cp1017 back in service pool
16:43 RobH: cp1019 back in full service
16:22 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r114021'
16:22 RobH: cp1017 memory error, coming down for troubleshooting.
16:18 RobH: cp1019 memory error cleared after reseating, notes on rt 2651
16:09 mark: Migrated all varnish3 packages to newer varnish packages from git
16:08 RobH: cp1019 coming down for memory error troubleshooting
15:58 RobH: cp1040 repaired per rt 2611
15:48 RobH: cp1040 down for memory replacement
15:09 logmsgbot: reedy synchronized stylize.php 'Test for hume'
15:04 logmsgbot: root synchronized ufg.sql 'test sync to see if hume is fixed'
14:55 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php
14:04 apergos: restarted swift-container-auditor on ms-be3, it had died for some reason
08:07 mutante: i reverted that (star cert for wikitech), no worries i "shred"ded the files
07:51 mutante: replaced self-signed cert on wikitech with the star cert
04:19 mutante: on stafford, deleting spence's puppet report files to free some disk space (they are like the largest report files of all)
03:09 mutante: stafford - - /var/lib/puppet/reports is getting quite large (18G), and we got the first disk space warning, do we want to keep those?
02:45 mutante: killing nrpe on several hosts where it was running as the wrong user again (somehow through the use of dsh)
02:21 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 16 02:21:35 UTC 2012
01:12 mutante: stopping nagios-wm temp. while changing nrpe config (will watch it manually until it's back)
00:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe'
00:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe'

March 15

23:17 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113974'
23:12 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/DisableTemplate.php 'r113973, fixes bug 35249'
23:10 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/modules/ext.articleFeedbackv5/ext.articleFeedbackv5.js 'r113972'
22:59 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
22:59 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Bump AFTv4 event logging percentage from 25% to 100%'
22:57 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js
22:48 mutante: purging Lucene monitoring on indexer from db9, remove duplicate service definitions manually anyways (still tons left), run purge script, reload Nagios..
22:24 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
22:23 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Raise AFTv4 event logging percentage from 5% to 25%'
22:21 mutante: getting rid of Swift HTTP checks on non production machines manually (come on spence _purge_ ;P)
22:07 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
22:04 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Raise AFTv4 event logging percentage from 1% to 5%'
21:44 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
21:28 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113961'
21:25 pgehres: K4-713 synchronized payments cluster to r113956
21:25 pgehres: disabled credit cards on donate.wikimedia.org
21:21 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'fix fatal'
21:20 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Bump AFTv4 event logging percentage from 0.27% to 1%'
21:19 Ryan_Lane: rebalancing instances gluster volume
21:18 RoanKattouw: That was r113959
21:18 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js
21:11 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r113958'
21:09 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ 'r113957'
20:46 mark: bits.pmtpa cluster back online
20:44 RobH: dns update for silver and zhen servers
20:37 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php
19:54 RobH: sq67-sq70 have been reinstalled, but not signed in puppet, not sure if they are ready for that or if there are other items mark needs to change first
19:11 RobH: working on sq67-sq70 reinstalls, disregard alerts
19:00 RobH: db1022 resetup and redeployed per rt 2537 and assigned back to asher
18:51 logmsgbot: reedy synchronizing Wikimedia installation... : Running scap to deal with message changes earlier
18:19 RobH: db1022 coming down for reinstall and resetup of raid per rt 2537
17:55 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/ 'r113940'
17:54 logmsgbot: reedy synchronized php-1.19/extensions/CheckUser/ 'r113940'
17:53 logmsgbot: reedy synchronized php-1.19/extensions/wikihiero/modules/ext.wikihiero.css 'r113940'
17:52 logmsgbot: reedy synchronized php-1.19/extensions/NewUserMessage/NewUserMessage.class.php 'r113940'
17:41 logmsgbot: reedy synchronized php-1.19/includes/RecentChange.php 'r113938'
17:38 logmsgbot: reedy synchronized php-1.19/resources/mediawiki/mediawiki.util.js 'r113936'
17:37 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialUndelete.php 'r113936'
17:32 logmsgbot: reedy synchronized php-1.19/languages/messages/ 'r113935'
17:31 logmsgbot: reedy synchronized php-1.19/resources/ 'r113935'
17:31 logmsgbot: reedy synchronized php-1.19/includes/ 'r113935'
17:16 logmsgbot: reedy synchronized php-1.19/includes/SkinTemplate.php 'r113932'
16:13 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php 'r113929'
15:15 mark: Created git repo operations/debs/varnish in gerrit
14:06 apergos: disabled moodbar temporarily on en wikii, see bug 35245
14:02 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disable of feedback dashboard (right config var this time?)'
13:51 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disable of feedback dashboard'
13:11 apergos: on screen as root on dataset1001, copying to gluster volume; if this causes problems feel free to shoot it. ( cp -a 20120211 /mnt/glusterpublicdata/public/enwiki/ )
09:08 mutante: ran puppet on mw1020
08:12 mutante: installing apache,apt,cron,mysql-client upgrades on spence
07:51 mutante: messed with /var/lib/dpkg/status on hume to fix broken packages/remove "marked for purging" on libmysql-php5 without removing a ton of other packages, rather hackish but seems fine anyways, like not broken anymore on simulated dist-upgrade etc
07:01 mutante: uprading apache and apt on hume
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 15 02:17:35 UTC 2012
01:26 Ryan_Lane: labsconsole was missing libapache2-mod-php5. puppet must have tried to upgrade a package unsuccessfully
01:22 mutante: planet back up (installed libapache2-mod-php5 which installed apache2-mpm-prefork and removed apache2-mpm-worker)
01:19 mutante: planet down - apache on singer, syntax error in site config "Invalid command 'php_admin_flag'"
01:03 mutante: fixing nrpe "unable to read output" raid check on srv197,207,243,,244,253.. (nrpe running as wrong user)

March 14

23:16 maplebed: installed the swiftcleaner to run daily from iron. see root's crontab for more info.
20:41 binasher: disabled log_queries_not_using_indexes on all core dbs
20:33 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
20:30 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
19:29 maplebed: rebooting ms-be1 to enable hyperthreading (and make it the same as all the other ms-be hosts)
19:06 preilly: pushing x-images header for vary support
19:06 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
19:05 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'zero needs to add x-images to vary header'
18:58 maplebed: ms-be5 is back in rotatino
18:31 preilly: push zero change for carrier testing
18:31 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
16:19 RobH: updating dns for new domain wikimediacommons.pt (nameservers not yet pointed at us)
16:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'add vcs for extdist updates'
13:03 RobH: cp1029-cp1035 all installed and ready for varnish deployment, puppet has been run
08:24 mutante: running "apt-get -f install" on snapshot3 to fix dpkg, which installed mysql-client- and client-core-5.1
08:02 mutante: stop/start memcached on srv254,srv255,srv257
07:51 mutante: restarting mecached on marmontel
07:51 mutante: fixing owa[1-3] Swift HTTP commands manually
03:44 mutante: ekrem - user agent "AppleDictionaryService" requests cause temp. WAP outage ..it seems
03:38 mutante: free some disk space on spence - deleted user.log.1 on spence, compressing messages.1, apt-get clean,...
02:52 RobH: cp1032-cp1035 reinstall issue wiped mbr causing issues, will reinstall in my AM
02:49 RobH: revoked, cp1032 is some reason in grub error, and its too late at night for me to work on it, will troubleshoot tomorrow
02:48 RobH: realized i forgot to log hours ago that cp1029-cp1036 are installed with puppet run, ready for varnish deployment tomorrow
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 14 02:17:13 UTC 2012

March 13

23:51 mutante: upgrading bugzilla to 4.0.5
23:42 logmsgbot: reedy synchronized php-1.19/resources/jquery/jquery.textSelection.js 'r113786'
23:14 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
22:47 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113779'
22:44 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r113774'
22:44 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/common.css 'r113774'
22:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113771'
22:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/api/ApiQueryExcerpts.php 'r113774'
22:27 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Removing moile URL template for tewtwiki'
21:44 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
21:44 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
21:31 logmsgbot: asher synchronized wmf-config/db.php 'replacing db18 with new s7 slave db56'
21:19 binasher: started slaving db56 from db37
20:30 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
19:27 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
19:17 RobH: iron updated to use ipmi_mgmt script
19:08 preilly: pushing changes for zero to mswiki
19:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
19:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
19:05 binasher: streaming hotbackup of db1041 to db56 (new s7 slave replacing db18)
18:10 maplebed: failover successful, restarted pybal on lvs4, failback successful.
18:09 binasher: power cycling db1020, which also froze this morning
18:08 maplebed: stopping pybal on lvs4 - should fail over to lvs3
17:47 maplebed: pybal restarted on lvs3
17:47 binasher: power cycling db1040, crashed again
17:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 35183 - p include extensions/Renameuser/Renameuser.php instead of extensions/Renameuser/SpecialRenameuser.php'
17:12 mark: Sending all normally-pmtpa upload traffic to upload-lb.eqiad
17:05 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
16:59 preilly: add disable images support to mswiki under zero domain
16:59 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'add disable images option for mswiki on zero domain'
16:58 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add disable images option for mswiki on zero domain'
16:46 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mswiki remove from mywiki'
16:44 mark: Sending traffic from Japan, India, Mexico to upload-lb.eqiad
16:37 LeslieCarr: reinstalling neon
16:23 apergos: stole some free space from the phys volume on ms1002 to give us more time for the rsync to keep going til after the move to swift etc
15:28 mark: Sending traffic from the USA to upload-lb.eqiad
15:27 mark: Rebooting lvs1005 with upgraded kernel/packages
15:12 LeslieCarr: manually deleted cp1025 info from nagios config file - nagios restored for now
14:51 mark: Sending traffic from Canada to upload-lb.eqiad
14:32 mark: Sending traffic from Brazil to upload-lb.eqiad
13:58 mark: Sending traffic from Argentina to upload-lb.eqiad
12:58 mark: Seeding the eqiad upload caches from live upload requests
11:59 mark: Setup squid logging to oxygen, with oxygen relaying to multicast 233.58.59.1
11:02 mark: Rebooting lvs1002 with kernel updates
10:17 mark: Rebooting manutius with newer 2.6.36 kernel to attempt avoiding i/o kernel bug with torrus
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 13 02:18:03 UTC 2012

March 12

22:55 K4-713: synchronized payments cluster to r113679, and tweaked the anti-fraud rules
21:51 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r113671'
21:51 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113671'
21:44 Reedy: Running foreachwiki extensions/WikimediaMaintenance/cleanupBug31576.php in screen as me on hume
21:39 RobH: search1014 repaired per rt 2483
20:26 RobH: cp1040 coming down for hardware stuffs
18:19 Nikerabbit: Assuming scap has finished
17:48 logmsgbot: nikerabbit synchronizing Wikimedia installation... : Deploying updated Translate
17:46 notpeter: restarting indexer on searchidx2
17:24 logmsgbot: nikerabbit synchronized php-1.19/includes/Title.php 'r113635'
17:22 logmsgbot: nikerabbit synchronized php-1.19/languages/ 'r113635'
17:14 logmsgbot: nikerabbit synchronized php-1.19/extensions/Narayam/ 'Updating Narayam'
17:13 mark: PXE booting cp1025-cp1028
17:11 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'Updating WebFonts'
15:16 mark: Rebooted manutius, stuck in a similar state as streber always did
06:10 mutante: turning off debug mode in nagios-nrpe, again had to kill it , restart fails
05:53 mutante: dunno, copper was stuck (no mgmt output after reboot) but powercycling it and back
05:43 mutante: rebooting copper to make sure grub update didnt break it and asked for restart anyways
05:37 mutante: copper - installing (security) updates (apt,grub,openssl,ruby,libc6..)
04:19 mutante: wanted to restart nagios-nrpe-server on spence with debug=1 to investigate permission issue. arr! "Address already in use" "cant write to pidfile", killed the one started on Feb18, and reordered allowed_hosts, spence talks to itself again now :p
03:40 mutante: same (and nscd) on fenari
03:35 mutante: upgrading libc6 and related packages on spence
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 12 02:17:28 UTC 2012

March 11

08:14 apergos: restarted lighttp on dataset2
07:49 apergos: removed current htcp log file, restarted purger, it seems to be logging normallynow
07:35 apergos: current ls shows 17416851456 2012-03-11 07:34 HTCPpurger.log while current du -sh shows 175M for /var/log. Sparse file that gets rotated badly? lots of leading nulls (many gb worth), why?
07:33 apergos: on ms1004 the HTCPpurger.log file after rotation was 17 gb, filling the disk. Removed it.
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 11 02:17:35 UTC 2012

March 10

22:09 Reedy: Make that wikimania2012, not wikimediawiki
22:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable anon page creation for wikimediawiki'
19:28 binasher: set sync_binlog = 1 on all current masters and eqiad dbs
19:22 binasher: reslaved db1033
07:03 mutante: ran puppet on db1022, another one that works fine manually but somehow did not by itself
05:11 mutante: doing more (cp*, db*, msbe-* ,mw*) by hand / for loop
05:01 mutante: starting nagios-nrpe-server on all via dsh (fail to restart on config change issue)
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 10 02:16:57 UTC 2012
01:07 maplebed: started swiftcleaner on owa1 looking for (and purging) bad objects
01:06 maplebed: rebalanced the swift rings to finish decreasing traffic sent to ms1 and ms2
00:18 Ryan_Lane: powercycling ssl1003
00:18 Ryan_Lane: powercycling ssl1001

March 9

20:34 notpeter: stopping search indexer on searchidx2 for fresh rsync to searchidx1001
19:58 preilly: pushed change to remove description from landing page
19:57 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
18:59 Ryan_Lane: sending test.m.wikipedia.org to the same place as test.wikipedia.org via squid
18:58 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Fixing wgMobileUrlTemplate settings for domains that do not have .m. domains configured'
18:48 logmsgbot: reedy synchronized php-1.19/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.css 'r113497'
18:40 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Changing the way in which wgMobileUrlTemplate is configurable by InitialiseSettings.php'
18:39 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling wgMobileUrlTemplate for testwiki - hopefully for real this time'
18:34 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Making wgMobileUrlTemplate configurable by InitialiseSettings.php'
18:34 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling wgMobileUrlTemplate for testwiki'
17:40 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113489'
17:32 maplebed: set swift storage device weight on ms2 to 0 and pushed out rings
15:52 apergos: cleared up a little bit of space on root partition of snapshot2, but that's about it. I hope we never have 3 versions of mw in test at the same time, the tmp caches will kill us
15:52 mark: Turned off vcc_err_unref on all varnish servers, so varnish doesn't complain when ACLs/probes/backends are unused
15:44 Jeff_Green: hume apt upgrades, puppetd --test, switch to mysql 5.1.53-fb3753-wm1
06:38 Ryan_Lane: reloading autofs on all labs instances
06:13 Tim: running svn cleanup on extdist trunk
04:18 Tim: switched php and wmf-deployment symlinks over to php-1.19 instead of php-1.18
04:18 Tim: restarted morebots
00:57 pp-pdf2: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
00:57 pp-pdf3: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
00:57 pp-pdf1: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
00:38 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mywiki'
00:32 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.js 'fixes to code push'
00:32 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.min.js 'fixes to code push'
00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fixes to code push'
00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.js 'fixes to code push'
00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.min.js 'fixes to code push'
00:01 RobH: oxygen install done, booting successfully after multiple tests, now running puppet for initial config
00:01 K4-713: updated the paypal IPN listener on aluminium to r1450

March 8

23:57 logmsgbot: awjrichards synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113428'
23:56 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'per-wiki memory limit configuration, with extra memory for zh* for converter tables'
23:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'per-wiki memory limit configuration, with extra memory for zh* for converter tables'
23:42 mutante: rebooting ms-be5
23:37 logmsgbot: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments
23:24 binasher: streaming hotbacking of db1017 to db1033 - no snapshots of enwiki in eqiad til db1033 is back
23:19 Tim: started changing the php symlink to 1.19 instead of 1.18, but then changed my mind and changed it back.
23:16 logmsgbot: tstarling synchronizing Wikimedia installation... :
23:07 logmsgbot: tstarling synchronized php-1.19/extensions/ExtensionDistributor/svn-invoker.conf
23:01 logmsgbot: asher synchronized wmf-config/db.php 'returning db24 to service'
22:58 maplebed: powercycled ms-be3 - it crashed 2.5 hours ag.
22:52 logmsgbot: asher synchronized wmf-config/db.php 'pulling db18'
22:40 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend 'deployed r113413, r113414'
22:39 LeslieCarr: poked hole to allow labs machines to reach gluster machines in tampa
22:13 logmsgbot: catrope synchronized php-1.19/includes/MagicWord.php 'r113411'
22:13 logmsgbot: catrope synchronized php-1.19/includes/Cdb.php 'r113411'
22:13 logmsgbot: catrope synchronized php-1.19/includes/WebRequest.php 'r113411'
22:11 RobH: udpating dns for oxygen
22:03 RobH: oxygen coming down for reinstall
20:42 cmjohnson1: power to msw-c1-sdtpa restore
20:40 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
20:40 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.php 'changes for zero'
20:39 cmjohnson1: removing and relocating power to msw-c1-sdtpa
19:38 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
19:34 RoanKattouw: Running scap for ArticleFeedbackv5 updates
19:30 RoanKattouw: Running AFTv5 schema changes on enwiki
19:29 logmsgbot: catrope synchronized wmf-config/CommonSettings.php '$wgArticleFeedbackv5OversightEmails'
19:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '$wgArticleFeedbackv5OversightEmails'
19:26 RoanKattouw: Applying AFTv5 schema changes to en_labswikimedia
19:09 preilly: push zero rated changes
19:09 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
19:09 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
19:04 RoanKattouw: Clearing message blobs
18:53 RoanKattouw: Running rebuildLocalisationCache.php
18:49 binasher: power cycling cp1044
18:46 binasher: purging entire mobile varnish cache - the main mobile template included robots no-follow
18:43 preilly: needed to fix a google issue with robots
18:43 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ApplicationTemplate.php 'remove ROBOTS metatag'
18:40 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ApplicationTemplate.php 'remove ROBOTS metatag'
18:40 binasher: deploying new squid frontend.conf to fix epic fail - all googlebot traffic was being redirected to mobile. now just if it's mobilegooglebot.
18:29 RoanKattouw: Applying AFTv5 schema changes on testwiki
18:27 RoanKattouw: Pushing new AFTv5 code to testwiki, do not sync to the live site just yet
17:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'ptwikipedia to ptwiki'
17:14 cmjohnson1: shutting down db18 for memory testing
16:57 RobH: search1014 still down per rt2483
16:47 maplebed: took ms-be5 out of rotation in the swift cluster - it's crashed 3 times now.
16:36 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'r113368'
16:31 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Revert live hack because it works, will come in properly'
16:30 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Test for bug 27246'
16:16 RobH: search1008 repaired
15:52 RobH: mw1103 finally repaired and ready for os and such
14:48 pp-pdf1: installed python faulthandler 2.1
14:47 pp-pdf3: installed python faulthandler 2.1
14:47 pp-pdf2: installed python faulthandler 2.1
14:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35012 - Namespace aliases for wikipedia and wikipedia-talk namespaces on Sanskrit wiki'
09:17 mutante: running puppet on mw1010 - finished quickly without problems - uh, wonder why Nagios reported puppet freshness then
08:22 mutante: cp1019 - Hitting F1 to continue reboot ( "Alert! System fatal error during previous boot")
08:21 mutante: cp1019 went down, then rebooted by itself (i think) after showing "idrac-8W82BP1 Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted"
07:54 mutante: cadmium fixed by adding groups::wikidev
07:41 mutante: puppet on cadmium broken due to dependency Group[500] for User[catrope]
07:20 mutante: ms1004 ran out of disk - caused by 17G HTCPurger.log.1, trying to gzip it now
06:52 logmsgbot: tstarling synchronized multiversion/MWMultiVersion.php
06:51 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
03:04 Guest32353: powercycled ms-be5; it has been unresponsive for 2 hours.
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 8 02:18:02 UTC 2012
01:32 AaronSchulz: fixBug34995.php done
01:26 AaronSchulz: running fixBug34995 on all wikis
00:17 Ryan_Lane: adding zero cnames
00:16 Ryan_Lane: installing newer wikimedia-task-dns-auth on all dns servers
00:15 Ryan_Lane: added wikimedia-task-dns-auth_0.18 to the repo, to add support for zero

March 7

23:05 logmsgbot: aaron synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed r113319'
22:39 maplebed: set swift weight for ms1 to 0 initiating the process to move data off the host in preparation for decomissioning it.
21:17 Jeff_Green: running apt upgrades and puppetd --test on srv194, srv197, srv203, srv212, srv213, srv230, srv244, srv245, srv252, srv282 and manually restarting nrpe because they're reporting funky in nagios
20:20 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
20:17 Jeff_Green: yet another redirects.conf change, per RT#2498 redirect wikimedia.com-->wikimedia.org
20:05 binasher: reverted no-pagecache rsync on search nodes - without corresponding index warmup in lsearchd, it just pushes back the pain a bit and does more harm than good
20:04 binasher: deployed support for zero.wikipedia.org and carrier tagging to mobile varnish servers
19:38 logmsgbot: catrope synchronized php-1.19/includes/resourceloader/ 'r113278'
19:27 Jeff_Green: manual apt-upgrade, puppetd --refresh, and repeat on srv265 because it was running on outdated apache config
18:44 RobH: correction sq39
18:36 RobH: pulled sq39 from text pybal config, pulled sq46 from upload pybal config
18:36 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
18:36 logmsgbot: catrope synchronized php-1.19/extensions/CustomUserSignup/modules/AccountCreationUserBucket.js 'touch'
18:12 RobH: shutting down sq38 and sq46 per rt 2581 for testing
16:02 cmjohnson1: replacing hdd for disk 10 on db22
16:00 cmjohnson1: pulling disk 10 from db22
13:28 mark: Removed torrus from streber
13:00 pp-pdf2: updated mwlib to 0.13.6
13:00 pp-pdf3: updated mwlib to 0.13.6
13:00 pp-pdf1: updated mwlib to 0.13.6
11:29 logmsgbot: hashar synchronizing Wikimedia installation... : trigger a rebuild of l10n cache
04:53 mutante: added ms-be5 drives to swift cluster
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 7 02:18:01 UTC 2012
02:11 logmsgbot: catrope synchronized php-1.19/includes/api/ApiBase.php 'r113212'
01:58 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'bumped max file size to 4GiB'
00:27 maplebed: put ms-be4 into rotation as a new production swift backend storage node
00:21 maplebed: put ms-be3 into rotation as a new production swift backend storage node
00:05 maplebed: put ms-be2 into rotation as a new production swift backend storage node

March 6

23:54 logmsgbot: catrope synchronized php-1.19/extensions/CustomUserSignup/ 'Belated sync of r113056'
23:52 binasher: deploying new frontend squid config to include googlebot in mobile redirects
23:36 logmsgbot: reedy synchronized php-1.19/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php 'r113200 reverting r113198'
23:25 Tim: patched 5xx-filter.c live on locke and reloaded udp2log to stop the segfaults
23:20 logmsgbot: reedy synchronized php-1.19/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php 'r113198'
21:46 logmsgbot: catrope synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r113183'
21:41 notpeter: restarting puppet on brewster
21:03 Jeff_Green: pushing another change to redirects.conf and doing a graceful apache restart
20:32 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild message cache stuffs for r113129
20:31 Jeff_Green: disabled Global Connect nagios test (check_gcsip) on payments cluster because GC is down and nagios is spammy
20:25 notpeter: reimaging search1001-1020 with new partman recipe :/
20:22 notpeter: temp stopping puppet on brewster
20:21 logmsgbot: reedy synchronized php-1.19/resources/mediawiki.action/mediawiki.action.edit.js 'r113175'
20:20 logmsgbot: reedy synchronized php-1.19/maintenance/populateRevisionSha1.php 'r113175'
20:19 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialContributions.php 'r113175'
20:18 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialUserlogin.php 'r113176'
20:00 pp-pdf1: installed log-wikimedia-operations (which can be used for automated logging to #wikimedia-operations)
19:53 Ryan_Lane: restarting labs mysql to allow for more connections
19:26 Ryan_Lane: installing nova-api on virt0
19:09 Ryan_Lane: upping FLAGS.sql_max_pool_size for nova-api
18:47 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
18:46 Ryan_Lane: rebooting all instances
18:34 Ryan_Lane: restarting nova-network on virt2
18:19 Ryan_Lane: rebooting virt1
18:15 Ryan_Lane: rebooting virt2
18:11 Ryan_Lane: rebooting virt3
18:07 Ryan_Lane: rebooting virt4
17:57 Ryan_Lane: taking the opportunity to apply security updates to virt0-4
16:25 logmsgbot: catrope synchronized docroot/foundation/FrameResize.html 'Put Jobvite frame resize file in foundationwiki docroot per Erik'
11:40 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching sr* to 1.19
11:15 logmsgbot: hashar synchronized php-1.19/languages/messages/MessagesSa.php 'r1113039 for bug 34938 : title is sometime empty on Sanskrit wikis'
11:13 logmsgbot: tstarling synchronized php-1.19/includes/OutputPage.php 'r113128'
10:41 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching zh* from 1.18 to 1.19
08:36 mutante: on hooper: puppet broken due to dependency Package[libapache2-mod-php5] for Service[apache2]
03:33 mutante: rebooting bast1001 for kernel upgrade
03:32 mutante: upgrading apache2 packages, base-files, kernel, several libs on bast1001
03:27 mutante: installing a couple upgrades on fenari (apache2-utils, update-manager-core, cvs, ruby, libxml*, libopenssl-ruby*...)
02:37 logmsgbot: LocalisationUpdate completed (1.18) at Tue Mar 6 02:37:06 UTC 2012
02:36 logmsgbot: tstarling synchronizing Wikimedia installation... : updating to r113119
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 6 02:18:13 UTC 2012
01:27 Jeff_Green: manually updated packages and restarted apache on srv198, srv229, srv262, srv268, mw40 because their apache redirect configs failed to update after sync-apache and restart
01:07 Jeff_Green: another adjustment to redirects.conf and apache-graceful-all for RT#2488

March 5

22:24 Jeff_Green: modified redirects.conf per RT #2488
21:21 Reedy: Ran foreachwiki cleanupUploadStash.php
20:36 maplebed: enabled swift for 100% of thumbnails in production
18:18 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r113058'
18:11 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'WebFonts: bugwiki bug 34550; sawikisource bug 34159; amwiktionary amwikiquote bug 34700'
18:01 mark: Raised MTU between cr1-sdtpa - (csw1-sdtpa) - cr2-pmtpa to 9192
17:35 Jeff_Green: removed 3GB db30:/tmp/gmond.log and force-restarted gmond b/c the init script failed to restart it
17:16 Jeff_Green: adjusted LVS partitions on hume, moved /usr/local/apache to a new 5GB mount
15:18 mark: Fixed DNS resolving on the core routers by allowing DNS replies in the loopback filter
14:44 logmsgbot: reedy synchronized php-1.19/includes/Title.php 'r113036'
14:43 logmsgbot: reedy synchronized php-1.19/includes/AjaxResponse.php 'r113036'
14:35 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/ 'r113035'
14:34 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/ 'r113035'
13:50 mark: Set increased OSPF/OSPFv3 metric 30 on both directions of the link cr1-eqiad:xe-5/2/1 <--> cr1-sdtpa:xe-0/0/1, to combat higher than normal jitter and packet loss on the link
12:53 mark: Upgraded observium to latest version
09:41 mutante: restarting memcached on marmontel
09:40 mutante: restarting squid backend on knsq25
06:52 Ryan_Lane: all of the instances are accessing the file descriptors of files inside of the _base directory, and fuse has an issue with this. gluster can't recreate the base directory because of the processes holding open the old one.
06:50 Ryan_Lane: I've corrupted the _base directory on the instance's glusterfs share. I'm recovering the files from file descriptors using lsof. Not totally sure how I'm going to get the _base directory back, yet.
02:33 logmsgbot: LocalisationUpdate completed (1.18) at Mon Mar 5 02:33:04 UTC 2012
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 5 02:16:39 UTC 2012

March 4

21:48 logmsgbot: reedy synchronized wmf-config/ 'Bug 32726 - Set =true for Commons'
21:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix .'
21:41 logmsgbot: reedy synchronized wmf-config/ 'Bug 32726 - Set =true for Commons'
21:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34897 - Enable Special:Import on Catalan wikisource'
20:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34567 - New logo for Arabic Wiktionary'
20:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34715 - Please modify the import sources for the Spanish Wikiversity'
20:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34694 - Install the Quiz extension on de.wikibooks'
20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wmgMoodBarCutoffTime'
20:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Create wmgMoodBarCutoffTime'
20:14 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Variablise moodbarconfig infoUrl'
20:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Variablise moodbarconfig infoUrl'
20:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34618 - Install MoodBar on fr.wikisource'
20:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34766 - Logo of Sanskrit Wikisource'
19:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34867 - Switch Sango wiktionary logo'
19:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34931 - Add namespaces aliases on as.wikipedia.org'
19:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34690 - Changing the name in the title bar to Assamese'
02:35 logmsgbot: LocalisationUpdate completed (1.18) at Sun Mar 4 02:35:16 UTC 2012
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 4 02:17:34 UTC 2012

March 3

18:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34728 - Categories added to user pages by Babel in pt.wiktionary'
13:04 logmsgbot: aaron synchronized php-1.19/includes/Revision.php 'deployed r112949'
02:35 logmsgbot: LocalisationUpdate completed (1.18) at Sat Mar 3 02:35:08 UTC 2012
02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 3 02:18:04 UTC 2012

March 2

21:41 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'disabled logging hack'
20:47 Jeff_Green: added redirect/301 from http://static.wikimedia.org --> http://dumps.wikimedia.org now that archival static html dumps are located there
19:53 mark: Decommissioned csw5-pmtpa from AS14907 service. rest in pieces ;)
19:10 mark: Did a hot cut to remove csw5-pmtpa out of the path of cr1-sdtpa -> csw1-sdtpa -> csw5-pmtpa -> cr2-pmtpa
17:46 cmjohnson1: powering down msw1-pmtpa for relcocation to d1-pmtpa
17:40 cmjohnson1: disconnecting management fiber from msw1-pmtpa
16:59 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'r112904'
16:55 RobH: ms-be4 boot order fixed, fixing ms-be5 & ms-be2
16:49 RobH: fixed boot order on ms-be3, fixing ms-be4
16:33 RobH: poking at bios on ms-be3
16:05 RobH: wikitech outage resolved
15:20 RobH: shutdown frdev offsite vm per email to engineering last week
15:18 RobH: backing up wikitech in hopes of upgrading some of its software
08:36 apergos: on ms1004, low on space, HTCPpurger.log.1 had about 16 gb of nulls before any real content, I tailed off the real stuff and tossed the original. The current log file has the same problem, why?
02:34 logmsgbot: LocalisationUpdate completed (1.18) at Fri Mar 2 02:34:34 UTC 2012
02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 2 02:17:51 UTC 2012
01:36 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/lockmanager/LockManager.php 'deployed r112867'
00:41 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree 'deployed r112862'

March 1

23:33 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'log agent'
23:29 logmsgbot: reedy synchronizing Wikimedia installation... : Push message updates from r112848
23:22 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'logging fix'
23:22 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
23:20 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
23:17 logmsgbot: reedy synchronized php-1.19/includes/filerepo/backend/FSFileBackend.php 'r112850'
23:16 logmsgbot: reedy synchronized php-1.19/includes/Article.php 'r112850'
23:11 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
23:06 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ApiFeedbackDashboardResponse.php 'r112848'
23:05 logmsgbot: reedy synchronized php-1.19/extensions/Collection/Collection.body.php 'r112848'
22:12 logmsgbot: aaron synchronized php-1.19/includes/specials/SpecialContributions.php 'deployed r112844'
22:06 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend 'deployed r112841'
21:04 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'enabled FileBackend debug log'
19:57 cmjohnson1: replaced disk 3 labstore1 chassis
19:54 cmjohnson1: removing disk 3 from labstore1 chassis
19:47 Ryan_Lane: restarted memcached on virt0
19:15 logmsgbot: reedy synchronized php-1.19/cache/interwiki.cdb 'Updating interwiki cache'
17:39 Jeff_Green: Removed >5GB /tmp/gmond.log on db25, db32, db33, db37
17:36 logmsgbot: hashar synchronized php-1.19/includes/EditPage.php 'r112819 - Bug 34849 diff during editing an old version compares to the old version instead of the current one'
17:36 Jeff_Green: Removed >5GB /tmp/gmond.log on db13
17:35 Jeff_Green: Removed >5GB /tmp/gmond.log on db11
17:25 Jeff_Green: Removed 5.3GB /tmp/gmond.log on db1018
17:24 Jeff_Green: Removed 5.3GB /tmp/gmond.log on db1017
17:13 Jeff_Green: Removed 4.8GB /tmp/gmond.log on db1008. Tried to resist urge to make snarky comment about ganglia but failed.
14:54 RobH: strontium server rebooting to set HT to enabled
14:26 mark: Moving bits traffic back from pmtpa to eqiad
14:24 mark: Cleared dnsmasq cache on virt2
14:16 mark: csw5-pmtpa: Mar 1 14:01:42:A:Power Supply 2 , 2nd from left, bad
14:14 mark: mr1-pmtpa rebooted/lost power for some reason
14:07 mark: pmtpa/sdtpa management network went down
13:54 mark: Pooled new eqiad bits servers strontium and palladium
12:45 logmsgbot: hashar synchronized php-1.19/includes/specials/SpecialWatchlist.php 'r111882 for Bug 34835 - watchlist shows times in UTC'
10:53 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: reverting sr* wikis back to 1.18 per Siebrand's recommendation due to bug 34832
06:26 logmsgbot: tstarling synchronized php-1.19/extensions/SpamBlacklist/SpamBlacklist.php 'r112781'
05:46 maplebed: started swift deletion run on owa1, 2, and 3.
02:33 logmsgbot: LocalisationUpdate completed (1.18) at Thu Mar 1 02:33:53 UTC 2012
02:16 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 1 02:16:52 UTC 2012
02:15 Ryan_Lane: vlan tagged virt5's eth0 and eth1 ports on csw1-sdtpa
02:12 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'debug logging'
02:02 logmsgbot: reedy synchronized php-1.19/resources/mediawiki.action/mediawiki.action.history.diff.css 'r112750'
01:59 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: all zh wikis back to 1.18
01:50 logmsgbot: aaron synchronized php-1.19/extensions/WikiLove 'deployed r112758'
01:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last 265 wikipedias over to 1.19wmf1
01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: s7 to 1.19wmf1
01:23 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'r112754'
01:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: s2 to 1.19wmf1
00:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Meanwhile, on wikipedia.... Hello ruwiki!
00:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki to 1.19wmf1
00:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwiki to 1.19wmf1
00:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.19wmf1
00:05 logmsgbot: tstarling synchronized php-1.19/extensions/Collection/Collection.body.php 'r112745'

@@ Line 1: / Line 1: @@
 == August 16 ==
-* 10:29 pp-pdf3: restarted all services
-* 10:29 pp-pdf1: restarted all services
-* 10:29 pp-pdf2: restarted all services
-* 10:29 pp-pdf1: update mwlib.epub to 0.14.0
-* 10:28 pp-pdf2: update mwlib.epub to 0.14.0
-* 10:28 pp-pdf3: update mwlib.epub to 0.14.0
-* 10:28 pp-pdf2: update mwlib.rl to 0.14.0
-* 10:28 pp-pdf1: update mwlib.rl to 0.14.0
-* 10:28 pp-pdf3: update mwlib.rl to 0.14.0
-* 10:28 pp-pdf2: update mwlib to 0.14.0
-* 10:27 pp-pdf3: update mwlib to 0.14.0
-* 10:27 pp-pdf1: update mwlib to 0.14.0
 * 03:57 mutante: nagios back up after adding missing monitor groups misc_pmtpa in appserver role (srv194)
 * 02:56 K4-713: synchronized payments cluster to 37e31eddf5a4