Server admin log/Archive 20

September 20

03:55 Tim: in LDAP, added myself to the wmf and ops groups
03:48 AaronSchulz: Removed $oldLabsWikis cruft from CommonSettings.php
03:47 logmsgbot: aaron ran sync-common-all 'enabled FlaggedRevs on test2wiki'
03:18 Tim: created a password for myself in LDAP
02:40 binasher: completed 1.18 schema migrations against all slaves. masters still need to be rotated and upgraded
02:29 logmsgbot: LocalisationUpdate completed (1.18) at Tue Sep 20 02:32:24 UTC 2011
02:24 logmsgbot: LocalisationUpdate completed (1.17) at Tue Sep 20 02:26:48 UTC 2011
02:14 binasher: finished 1.18 migrations on s6 slaves, started s7
01:48 Tim: stopped puppet on fenari since I need to change a file and puppet on wikitech doesn't have any documentation about the new git system
01:43 binasher: finished 1.18 migrations on s3/DEFAULT slaves, started s6
01:38 preilly: sync file to fix uncaught type error with reading null property
01:38 logmsgbot: preilly synchronized php/extensions/MobileFrontend/javascripts/application.js 'fix javascript error with null object'
01:30 logmsgbot: tstarling synchronized php-1.18/includes/profiler/ProfilerSimpleUDP.php 'r97591'
01:18 logmsgbot: aaron synchronized wikiversions.cdb 'set test2wiki to 1.18 again (for real this time)'
01:17 binasher: finished 1.18 migrations on s5 slaves, started s3/DEFAULT
01:16 logmsgbot: aaron synchronized wikiversions.cdb 'set test2wiki to 1.18 again'
01:16 logmsgbot: aaron synchronized wikiversions.dat 'set test2wiki to 1.18 again'
01:12 logmsgbot: aaron synchronized wikiversions.cdb 'set test2wiki back to 1.17'
01:12 logmsgbot: aaron synchronized wikiversions.dat 'set test2wiki back to 1.17'
00:33 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'fixed $wgTexvc dir for 1.18 (override extension default)'
00:28 logmsgbot: reedy synchronized php-1.18/includes/OutputPage.php 'r97588'
00:21 logmsgbot: reedy synchronized php-1.18/extensions/TitleBlacklist/TitleBlacklist.list.php 'r97583'
00:02 binasher: finished 1.18 migrations on s4 slaves, started s5

September 19

23:58 logmsgbot: reedy synchronized php-1.18/StartProfiler.php 'r97581'
23:52 binasher: finished 1.18 migrations on s2 slaves, started s4
23:48 maplebed: took ms1 out of rotation in the external data store hosts to create a new slave.
23:47 logmsgbot: ben synchronized wmf-config/db.php 'taking ms1 out of rotation to create a new slave in eqiad RT-1530'
23:29 logmsgbot: aaron synchronized php-1.18/includes/profiler/Profiler.php 'deployed r97577'
23:12 logmsgbot: aaron synchronizing Wikimedia installation... : switching testwiki back to regular 1.17
22:55 logmsgbot: aaron synchronizing Wikimedia installation... : switching test2wiki to 1.18
22:54 binasher: test2wiki on s3 has been migrated by hand
22:52 binasher: 1.18 schema migrations completed on all s1 slaves, starting s2
22:04 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Pre-emptively disable file movers for "user" group (for 1.18, r93871)'
19:48 binasher: 1.18 schema migrations completed against enwiki on the first slave (db36)
19:04 logmsgbot: neilk synchronizing Wikimedia installation... : regular UploadWizard update, a few changes to make JS message parser throw less fatals, other misc small fixes
19:02 binasher: begin run of /home/w/common/php-1.18/maintenance/upgrade-1.18wmf1-1.php
18:39 logmsgbot: LocalisationUpdate failed
18:38 preilly: removing show/hide from blackberry for now
18:38 logmsgbot: preilly synchronized php/extensions/MobileFrontend/DeviceDetection.php 'remove show/hide for now from blackberry'
18:05 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy Narayam changes
17:50 RoanKattouw-BSc: Moved /home/wikipedia/common/php-1.17-test/includes/.swp to /home/midom/php-1.17-test-includes-.swp , was causing sync errors due to restrictive perms
17:47 logmsgbot: aaron ran sync-common-all 'pushing 1.18 files out to apaches'
17:40 logmsgbot: aaron ran sync-common-all 'pushing 1.18 files out to apaches'
17:37 AaronSchulz: created php-1.18/cache/trusted-xff.cdb
17:28 AaronSchulz: Ran checkoutMediaWiki script just add php-1.18 non-svn files/symlinks (reedy checked out already)
16:54 logmsgbot: reedy synchronizing Wikimedia installation... : Deployment of 1.18 files to apaches
16:51 Reedy: Copy of php-1.18 to /home/wikipedia/common/php-1.18 finished, taking just under 46 minutes
16:03 Reedy: copying /tmp/php-1.18 to /home/wikipedia/common/php-1.18 in a screen session on fenari as reedy
16:02 Reedy: svn co done
16:01 Reedy: Running svn co svn+ssh://svn.wikimedia.org/svnroot/mediawiki/branches/wmf/1.18wmf1 on fenari to /tmp/php-1.18 in a screen session as reedy
14:24 mark: Upgraded asw-a4-dtpa and asw-a5-sdtpa to 2.7.02d to resolve memory corruption issues (RT #1113)
13:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30984 - Filemover group in ruwiki, allow sysop to remove filemover'
12:27 mark: Fixed Puppet on amslvs* [I4a710a8d] (RT #1448)
12:01 mark: Corrected wrong LVS service ip of search_pool3 [Icd27ac04]
11:49 mark: Changed depool thresholds for search servers LVS services [Ib58b6178]
11:07 mutante: owa2 - power up,run puppet,dist-upgrade,reboot - RT #1510
09:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30984 - Filemover group in ruwiki'
02:28 logmsgbot: LocalisationUpdate completed (1.17-test) at Mon Sep 19 02:31:18 UTC 2011
02:23 logmsgbot: LocalisationUpdate completed (1.17) at Mon Sep 19 02:25:34 UTC 2011

September 18

23:52 Tim: added comments to lsearch-global-2.1.conf indicating that the "x" hostnames are to disable services
23:13 Tim: restarted lsearchd on search7, that seems to have done something
23:09 Tim: search7 has been broken since ~09:00 UTC, trying to fix
07:18 apergos: restarted lsearchd on search7 about 5 min ago
04:06 AaronSchulz: created l10n in php-1.17-test/cache
02:19 logmsgbot: LocalisationUpdate failed (1.17-test)
02:19 logmsgbot: LocalisationUpdate completed (1.17) at Sun Sep 18 02:21:34 UTC 2011

September 17

17:43 AaronSchulz: fixed 10update script, same bug with $mwVerDbSets
13:00 domas: manually placed /etc/php5/conf.d/apc-aux.conf on lucid apaches, apparently working APC cache helps a lot with request processing
11:26 mark: PyBal on lvs1-3 was in a BGP colliding loop with cr2-pmtpa, restarted PyBal on all three hosts
10:44 domas: made profile hitrate collection less crashy, restarted it (from ~tstarling/src/udpprofile/, not sure where else it should've been)
10:44 mark: Upgrade of cr2-eqiad complete
10:34 mark: Upgrading re1.cr2-eqiad to junos 10.4R7.5
10:28 mark: Performing failover from cr2-eqiad re1 to re0
10:18 mark: Upgrade of cr1-eqiad complete
10:08 mark: Upgrading re0.cr1-eqiad to junos 10.4R7.5
09:55 mark: Upgrading re0.cr2-eqiad to junos 10.4R7.5
09:46 mark: Performing failover from cr1-eqiad re0 to re1
09:33 mark_: Upgrading re1.cr1-eqiad to junos 10.4R7.5
02:22 logmsgbot: LocalisationUpdate completed (1.17=aawiki 1.17-test) at Sat Sep 17 02:25:20 UTC 2011
00:00 notpeter: removing srv288 from the apaches pool for upgrade to lucid

September 16

23:48 maplebed: started copying data from all srv hosts with external storage databases to es1001. I expect it to run all weekend.
23:39 notpeter: putting srv289 back in the apache pool
23:12 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'added require for Math extension, checks $wmgUseMath and version compare with $wmfVersionNumber so it does nothing yet'
22:50 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'added wmgUseMath setting (on by default); not yet used'
21:27 AaronSchulz: fixed usage of $mwVersionNums in set-group-write2 (same double quotes bug the sync scripts had before)
20:51 logmsgbot: aaron synchronized php-1.17/extensions/FlaggedRevs/specialpages/ReviewedVersions_body.php 'deployed r97331'
20:51 Jeff_Green: putting http redirect back in front of donate.wikimedia.org b/c fundraisers aren't ready for it
20:51 logmsgbot: aaron synchronized php-1.17/extensions/FlaggedRevs/api/FlaggedRevsApi.hooks.php 'deployed r97331'
20:29 AaronSchulz: fixed geshi on test2 & secure testwiki
20:25 logmsgbot: aaron ran sync-common-all
20:22 AaronSchulz: changed mwscript & mwscriptwikiset to use the /home version of scripts (people usually did this before and some things like TrustedXFF expect it)
20:07 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Set wgAutopromoteOnceLogInRC for plwiki,plwikisource ahead of time - no effect yet - bug 29655'
20:02 logmsgbot: aaron synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevision.php 'deployed r97323'
19:55 Jeff_Green: repooled payments4 after its long strange testing hiatus
19:55 logmsgbot: aaron synchronized php-1.17-test/extensions/FlaggedRevs/FlaggedRevision.php 'deployed r97323'
19:03 preilly: fix for undefined method on wml card
19:02 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix method call to undefined method'
18:45 AaronSchulz: added php-1.17-test/libs/geshi symlink to common/libs geshi, working on testwiki now
18:43 AaronSchulz: created common/lib directory and placed a copy of GeSHi-1.0.8.10 inside
18:19 preilly: add version to mobile application js to bust cache
18:18 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'remove jquery add version id to bust cache'
17:55 preilly: size correction for mobile icons
17:55 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'config project icons for mobile size corrections'
15:19 logmsgbot: reedy synchronized php/extensions/OggHandler/OggHandler_body.php 'r97289'
15:18 logmsgbot: reedy synchronized php/extensions/OggHandler/OggPlayer.js 'r97289'
15:17 logmsgbot: reedy synchronized php/languages/messages/MessagesKk_cyrl.php 'r97287'
15:17 logmsgbot: reedy synchronized php/includes/HttpFunctions.php 'r97287'
14:53 domas: killed whoever was running full table scans with subselects on enwiki and hitting http://bugs.mysql.com/bug.php?id=46947 - thanks, you locked up enwiki for many users :)
12:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30928 - Make Portal namespace for shwikipedia'
11:47 Reedy: Run namespaceDupes on hywiki
02:22 logmsgbot: LocalisationUpdate completed (1.17=aawiki 1.17-test) at Fri Sep 16 02:24:43 UTC 2011
01:49 logmsgbot: aaron synchronizing Wikimedia installation... :
01:41 AaronSchulz: moved test2wiki to php-1.17-test
01:36 AaronSchulz: testing with testwiki on php-1.17-test
01:04 logmsgbot: aaron synchronizing Wikimedia installation... :
00:47 logmsgbot: aaron synchronized wikiversions.dat 'set testwiki to 1.17-test'
00:47 logmsgbot: aaron synchronized wikiversions.cdb 'set testwiki to 1.17-test'

September 15

23:32 Tim: running apt-get upgrade on hume
22:31 binasher: upgraded varnish on mobile cache servers to pull in udplog fix
22:15 preilly: today's push for mobile frontend
22:15 logmsgbot: preilly synchronized php/extensions/MobileFrontend/javascripts/application.js 'remove jquery'
22:14 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'wml paging'
22:14 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'remove jquery'
21:08 Jeff_Green: purged hume:/a/static/uncompressed and switched it to xfs. see RT #1474 for more info
20:59 robh: updated puppet for new pdus in row d-pmtpa as well as chris shell access
20:50 notpeter: removing srv289 from apaches pool for upgrade to lucid. also stopping memecache and apache.
20:40 Ryan_Lane: restating gerrit to change subject lines in change emails
19:30 notpeter: putting srv219 back into rending pool
18:40 mark_: Deactivate static routes on cr2-pmtpa
17:50 mark_: Upgraded cr2-pmtpa to JUNOS 10.4R7.5
17:50 notpeter: rebooting spence
17:44 notpeter: restarting torrus
17:40 mark_: Rebooting cr2-pmtpa
16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert bug 30594 (namespaces for plwiki)'
15:15 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'bug 30808: wgRestrictDisplayTitle for enwikibooks, should've been false'
02:27 logmsgbot: LocalisationUpdate completed (1.17) at Thu Sep 15 02:29:44 UTC 2011

September 14

23:32 logmsgbot: aaron synchronizing Wikimedia installation... :
23:25 logmsgbot: aaron synchronizing Wikimedia installation... :
23:08 logmsgbot: aaron synchronized multiversion/MWScript.php 'r96507'
23:07 logmsgbot: aaron synchronized multiversion/MWVersion.php 'r97110'
22:59 Ryan_Lane: finished switchover to git for puppet repos
22:14 Ryan_Lane: switching the svn puppet repo to the git one. going to lock the svn repo
22:11 binasher: re-imaged db14 to lucid
21:28 binasher: disabled nagios notifs for db14, preparing to re-image
20:50 logmsgbot: reedy synchronized php/resources/startup.js 'touch'
20:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 28081 - Install ArticleFeedback on Portuguese Wikipedia (ptwiki)'
20:40 logmsgbot: reedy synchronized php/resources/startup.js 'touch'
20:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 28081 - Install ArticleFeedback on Portuguese Wikipedia (ptwiki)'
19:46 Reedy: That was bug 30726
19:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Setup of confirmed group for ptwiki'
19:42 logmsgbot: py synchronized wmf-config/CommonSettings.php 'syncing out correct location of oggThumb'
19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30830 - Please add a Portal namespace to hywiki'
19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Swap NS_PORTAL for 100'
19:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30432 - set meta namespaces for ay.wikipedia'
19:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30808 - Set to false on English Wikibooks'
18:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add cp1043 and cp1044 to wgSquidServersNoPurge'
18:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30798 - Logo for Sami Wikipedia'
17:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30899 - Timezone for lmo.wikipedia'
15:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fawiki -'
15:02 Jeff_Green: correction: removed donate.wikiPedia.org from redirects.conf, sync-apache, apache-graceful-all
15:01 Jeff_Green: removed donate.wikimedia.org from redirects.conf, sync-apache, apache-graceful-all
14:57 robh: ps1-b5-eqiad
14:57 robh: lowered the humidity alarm on ps1-b25-eqiad, also fixed some of its configuration
14:24 logmsgbot: catrope synchronized php/extensions/LiquidThreads/classes/Hooks.php 'Fix stupid typo'
14:17 logmsgbot: catrope synchronized php/extensions/LiquidThreads/classes/Hooks.php 'More live hacks for fatals that Ariel has been seeing in the dumps'
14:10 logmsgbot: catrope synchronized php/extensions/LiquidThreads/classes/Hooks.php 'Live hack to work around fatal'
14:08 logmsgbot: catrope synchronized php/extensions/LiquidThreads/classes/Hooks.php 'Live hack to work around fatal'
14:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30832 - Set $wgOverrideSiteFeed to use Special:NewsFeed from GNSM extension, on Persian Wikinews (fawikinews)'
14:01 Reedy: Created ArticleFeedback tables on ptwiki
13:59 logmsgbot: reedy synchronized php/resources/startup.js 'Touch'
13:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ClickTracking on ptwiki as prequisite for AF'
12:24 mutante: srv281 - powercycle (one more time but created a hardware check ticket now)
11:56 mark: Upgrading asw-d-pmtpa to junos 10.4R3.4
11:45 mark: Added asw-d-pmtpa to rancid, torrus, observium
11:36 mark: Configured asw-d-pmtpa
11:12 RoanKattouw: Started Apache again on srv219 now that it's depooled
11:10 Tim: removed srv219 from /home/wikipedia/conf/pybal/pmtpa/rendering on reports that it is giving inappropriate PHP errors
10:59 RoanKattouw: Stopped Apache on srv219, was spewing lots of PHP errors. Needs an ops person to depool the box
10:51 RoanKattouw: logmsgbot: !log catrope synchronized wmf-config/CommonSettings.php 'Raise account creation throttle for India event per IRC request'
10:51 RoanKattouw: Restarted morebots
02:30 logmsgbot: LocalisationUpdate completed (1.17) at Wed Sep 14 02:32:57 UTC 2011
02:16 logmsgbot: root synchronized wmf-config/InitialiseSettings.php 'Enabling protocol-relative urls for donatewiki'
02:04 logmsgbot: root ran sync-common-all
00:34 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'remove feedback'
00:15 preilly: fix for wml issues
00:15 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix for wml issues'

September 13

23:26 notpeter: removing srv219 from rendering pool for upgrade to lucid
23:23 notpeter: putting srv220 back into the rendering pool
22:26 notpeter: removing srv220 from the rendering pool for upgrade to lucid
22:14 notpeter: putting srv221 back into rendering pool
21:35 binasher: srv207 was serving traffic with an old deploy and incorrect db.php. synced, and added back to mediawiki-installation grp
21:23 binasher: mysql 4.0.40 is dead!
21:22 binasher: S6 master changed to db43, master-pos is db43-bin.000227 144900190
21:21 logmsgbot: asher synchronized wmf-config/db.php 'setting S6 to writeable on new master'
21:18 logmsgbot: asher synchronized wmf-config/db.php 'setting S6 ({ru,fr,ja}wiki) to read only, replacing mysql master'
19:46 notpeter: removing srv221 from rendering pool for upgrade to lucid
19:39 maplebed: wikitech ran out of space on /. removed old log files. installed logrotate.
18:44 logmsgbot: asher synchronized wmf-config/db.php 'setting S5 (dewiki) to read only, replacing mysql master'
18:37 binasher: preparing to replace S5 master - will set to readonly shortly
18:32 notpeter: removing srv222 from rendering pool for upgrade to lucid
18:20 preilly: fix mismatched div
18:20 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix for wml card mismatched div'
18:05 mark: Rebooted asw-d-pmtpa to recycle VC member IDs
18:04 preilly: wml fixes
18:04 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.wml.php 'remove element title'
18:03 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix for wml card id attribute'
14:30 logmsgbot: reedy synchronized php/languages/messages/MessagesEn.php 'r96969'
09:01 mutante: knsq10 - powercycle, dist-upgrade/kernel,squid clean,reboot
08:55 mutante: srv281 - powercycle,start apache
02:24 logmsgbot: LocalisationUpdate completed (1.17) at Tue Sep 13 02:27:04 UTC 2011
01:16 neilk_: just deployed a config change (wmf-config r2266) that should fix bug #30797
01:13 logmsgbot: neilk synchronized wmf-config/CommonSettings.php 'the URL for the UploadStash image scaler should not be protocol-relative -- this is invoked with curl, behind the scenes to fetch a scaled image.'
00:48 notpeter: putting srv223 back into rendering pool

September 12

23:37 Ryan_Lane: add CVE-2011-3192 workaround to noc.wm.o
23:35 logmsgbot: midom synchronized wmf-config/db.php 'srv157 down'
23:23 notpeter: removing srv223 from rendering pool for upgrade
22:35 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'config project icons for mobile wikiquote correction'
22:32 preilly: pushing mobile project icon change
22:31 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'config project icons for mobile'
22:06 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix section title with link'
21:50 logmsgbot: reedy synchronized php/extensions/UploadWizard/resources/jquery/jquery.mwCoolCats.js 'Bug 30879 - UploadWizard adds categories without [[Category:]] prefix'
21:47 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Scrap that'
21:43 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Fix UW category addition bug'
21:38 robh: mc1001-mc1008 have sas controllers, ready for installation
21:30 notpeter: readding srv224 to rending pool
20:34 notpeter: removing srv224 from rendering pool for upgrade to lucid
19:23 logmsgbot: neilk synchronizing Wikimedia installation... : deploying for UploadWizard (up to r96823) - minor fixes to CSS/button position, WLM, categories
19:04 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'fix view'
18:22 preilly: fix wml view for mobile
18:22 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.wml.php 'fix wml view'
18:04 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable Narayam on tawikibooks and tawikisource per bug 29798'
17:40 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy Narayam bugfixes
17:39 logmsgbot: catrope synchronized php/extensions/CategoryTree/CategoryTreeFunctions.php 'r96696'
17:36 logmsgbot: catrope synchronized php/resources/mediawiki/mediawiki.js 'r96699'
16:48 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'r96864'
16:47 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeRevisionAuthorView.php 'r96864'
16:40 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'Revert'
16:38 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'r96862, r96858'
16:34 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'Revert push'
16:32 logmsgbot: reedy synchronized php-1.17/extensions/EmailCapture/SpecialEmailCapture.php 'r96860'
16:31 logmsgbot: reedy synchronized php-1.17/extensions/EmailCapture/EmailCapture.i18n.php 'r96860'
16:31 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeRevisionAuthorView.php 'r96860'
16:30 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'r96858'
16:26 robh: updating dns for new misc servers mgmt
15:41 Reedy: That was scapping r96856
15:40 logmsgbot: reedy synchronizing Wikimedia installation... :
15:01 logmsgbot: catrope synchronized php/extensions/UploadWizard/resources/mw.UploadWizardDeed.js 'r96848'
14:25 logmsgbot: reedy synchronized php-1.17/skins/common/shared.css 'r96829'
11:42 mutante: sq78 - powercycle, dist-upgrade/kernel, squid clean, reboot
11:06 mutante: sq66 - powercycle, dist-upgrade/kernel, squid clean, reboot
10:44 mark: Setup IPv6 transit to AS2828 on cr1-sdtpa
10:42 mutante: srv187 - power up, dist-upgrade/kernel, reboot
10:29 mutante: amssq56 - power cycle, dist-upgrade/kernel, squid clean,reboot
07:09 apergos: restarted searchd on search7
02:23 logmsgbot: LocalisationUpdate completed (1.17) at Mon Sep 12 02:25:37 UTC 2011

September 11

02:21 logmsgbot: LocalisationUpdate completed (1.17) at Sun Sep 11 02:23:26 UTC 2011

September 10

22:02 Ryan_Lane: purged binlogs on db9 to free up some space
02:24 logmsgbot: LocalisationUpdate completed (1.17) at Sat Sep 10 02:26:45 UTC 2011

September 9

19:06 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'touch'
19:05 logmsgbot: reedy synchronized php-1.17/resources/mediawiki/mediawiki.js 'r96682'
18:54 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'touch'
18:51 logmsgbot: reedy synchronized php-1.17/resources/mediawiki/mediawiki.js 'Revert push of r96679, stuff seems broken'
18:40 logmsgbot: catrope synchronized php/resources/mediawiki/mediawiki.js 'r96679'
18:04 Ryan_Lane: gave Danny B. editor rights on wikitech
16:36 mutante: amssq37 - power up, dist-upgrade/kernel,squid clean, reboot
15:44 mutante: ran sync-apache and apache-graceful-all to add 2 redirects (RT 1445 + 1449)
11:33 logmsgbot: catrope synchronized php/resources/jquery/jquery.highlightText.js 'r96625'
10:41 mutante: sq60 - power cycle,dist-upgrade/kernel ,reboot
10:28 mutante: sq65 - power up, dist-upgrade/kernel,squid clean,reboot
09:07 mark: Removed configuration for XO on csw5-pmtpa
09:00 mark: Set local-pref of most XO routes to 90, to favour traffic to AS 3257
02:22 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 9 02:24:42 UTC 2011
01:24 preilly: fix random link
01:24 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix random link'
01:11 Tim: restarted parser cache hit rate stats collector on fenari
00:38 preilly: only show feedback link on en wiki
00:38 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'only show feedback on en wiki'
00:23 logmsgbot: aaron synchronizing Wikimedia installation... :

September 8

23:32 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'fix footer links'
23:32 preilly: pushing fix for footer links
23:31 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix footer links'
22:02 binasher: changed m.wikipedia.org dns cname to point to mobile-lb.eqiad.wikimedia.org
21:56 binasher: upgraded wikimedia-task-appserver to 1.49 (het deploy) in the hardy repo
21:49 binasher: moving m.wikipedia.org nagios monitor to point to mobile-lb.eqiad.wikimedia.org
16:21 mark: Shutdown csw5-pmtpa:e5/3
16:21 mark: Migrated XO transit connection from csw5-pmtpa:e5/3 to cr1-sdtpa:xe-0/0/2
14:55 mark: Shutdown BGP sessions to AS 2828 on csw5-pmtpa
12:50 logmsgbot: catrope synchronizing Wikimedia installation... :
12:50 RoanKattouw: Seems I forgot to deploy my 1.17wmf1 merges from yesterday, scapping them out now
12:35 mark: Activated XO BGP session on cr1-sdtpa
02:28 logmsgbot: LocalisationUpdate completed (1.17) at Thu Sep 8 02:30:26 UTC 2011
00:42 preilly: fix for home page with styles that float images
00:42 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix homepage'
00:11 preilly: fix wildcard selections for mobile main page
00:10 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix homepage'
00:01 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.i18n.php 'fix homepage'
00:00 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix homepage'
00:00 preilly: pushing fix for homepage
00:00 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/notices/notice_1.html.php 'fix homepage'
00:00 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'fix homepage'

September 7

23:59 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'fix homepage'
23:59 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/information/leave_feedback.html.php 'fix homepage'
19:46 logmsgbot: reedy synchronized php-1.17/includes/OutputPage.php 'r96472'
18:49 Reedy: That sync was Bug 30621 - Set $wgSiteFeed['rss'] to use the new Special:NewsFeed from GNSM on English Wikinews
18:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:50 logmsgbot: catrope synchronized php/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'r96455'
17:50 logmsgbot: catrope synchronized php/extensions/UploadWizard/resources/mw.UploadWizard.js 'r96455'
17:03 Reedy: Run namespaceDupes on sourceswiki again, this time with --fix
16:07 mutante: srv281 power cycled
14:22 logmsgbot: catrope synchronized php/includes/SquidUpdate.php 'r96439'
14:22 logmsgbot: catrope synchronized php/includes/GlobalFunctions.php 'r96439'
14:21 logmsgbot: catrope synchronized php/includes/Defines.php 'r96439'
13:50 logmsgbot: catrope synchronized php/includes/SquidUpdate.php 'Reinstate Tim's live hack'
12:35 Reedy: Run namespaceDupes.php on sourceswiki
12:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30724 - Author namespace at multilingual wikisource'
12:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30724 - Author namespace at multilingual wikisource'
12:29 mark: Setup port and BGP configuration for XO on cr1-sdtpa
12:29 mark: Setup generic BGP_transit_in import policy statement on cr1-eqiad, cr2-eqiad, cr1-sdtpa, cr2-pmtpa, and migrated TiNet neighbor to use it
12:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30724 - Author namespace at multilingual wikisource'
12:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30724 - Author namespace at multilingual wikisource'
12:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30756 - Enable WikiLove on Swedish Wikipedia'
11:05 mutante: db15 - dist-upgrade/kernel, reboot
10:53 mutante: db15 - reset/power up via Sun LOM
10:20 logmsgbot: catrope synchronized php/includes/SquidUpdate.php 'Revert emergency patch for bug 30792'
10:19 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'bug 30792: set $wgInternalServer'
09:19 mutante: sq72 - powercycle,dist-upgrade/kernel,squid clean,reboot
02:53 logmsgbot: tstarling synchronized php-1.17/extensions/OggHandler/OggHandler_body.php 'live patch to fix total breakage of OggHandler due to protocol-relative upload URLs'
02:45 logmsgbot: tstarling synchronized php-1.17/includes/SquidUpdate.php 'emergency patch for bug 30792'
02:21 logmsgbot: LocalisationUpdate completed (1.17) at Wed Sep 7 02:24:09 UTC 2011
01:40 logmsgbot: asher synchronized 503.html 'running sync-file to test new wikimedia-task-appserver'

September 6

22:38 logmsgbot: catrope synchronized php/languages/messages/MessagesEn.php 'r96194'
22:20 logmsgbot: ben synchronized wmf-config/mc.php 'moved mc servers that passed mctest.php from the down list to the spares list. Whitespace cleanup. Linked to memcache wikitech page. -ben'
22:13 binasher: wiped enwiki slave on db1048, re-enabled binlogging
20:06 Jeff_Green: rebooting db1008 for kernel update
19:58 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Make wgUploadPath protocol-relative for commons, meta and test'
19:22 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable HTTPS on meta'
19:19 logmsgbot: catrope synchronized php/includes/Title.php 'r96360'
19:18 logmsgbot: catrope synchronized php/extensions/CentralNotice/CentralNotice.php 'r96360'
19:18 logmsgbot: catrope synchronized php/extensions/Collection/Collection.body.php 'r96360'
17:42 robh: ns1 pdns restarted
17:42 robh: ns1/linne pdns died. initial post sync check with dig showed new info populated, odd.
17:38 robh: all dns servers still responsive after update
17:37 robh: updating dns for sodium server per RT#1423
07:56 apergos: rsyncd set up and running on dataset2 for mirroring recent xml dumps
07:30 apergos: restarted lighttpd on dataset2
02:24 logmsgbot: LocalisationUpdate completed (1.17) at Tue Sep 6 02:26:41 UTC 2011

September 5

18:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30756 Enable wikilove on sewiki'
16:21 RoanKattouw: Restarting indexer on searchidx2
16:20 RoanKattouw: Installing new build of LuceneSearch.jar on searchidx2. Contains fix for protocol-relative URLs that only applies to the indexer
15:48 logmsgbot: catrope synchronized php-fatal-error.html 'Update for r96283'
15:42 logmsgbot: catrope synchronized live-1.5/404.php 'Per bug 30733, fix HTTPS detection and make URLs protocol-relative. Grr, why is this file not versioned?'
13:30 mutante: sq31 shutdown -> RT #1431 (degraded RAID, just on one disk and squid would not run anyways)
13:18 mark: Extended /var LVM LV on lily by 5 GB
13:00 mutante: sq33 - dist-upgrade/kernel, reboot | sq31 - hardware failure? ->RT #1431
12:22 mutante: sq31 - installed smartmontools to check disk health, squid backend does not start due to missing /dev/sdd | srv266 run puppet manually
11:42 mutante: srv217,srv266 - fix NTP/ntpdate | sq31 - dist-upgrade/kernel, start squid
11:34 mutante: srv266 - power cycle (too often since July)
11:25 mutante: sq76 - power up,dist upgrade/kernel,squid clean,reboot
11:12 mutante: also cleared paniclogs on mchenry (spamd socket fail on Aug31) and lily (temp. failure to write log due to disk space, see Tim's log entries Sep 2nd)
11:05 mutante: reviewd paniclog on srv207 due to non-zero size. ran out of mem once on Aug 11, cleared logs to stop email alerts
10:43 mutante: sq55 - power up,dist upgrade/kernel,squid clean,reboot
02:27 logmsgbot: LocalisationUpdate completed (1.17) at Mon Sep 5 02:29:45 UTC 2011

September 4

16:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'From bug 29940, kill MW/MWT namespace aliases from hiwiki. MW is an interwiki, and mwt is a languagecode'
02:27 logmsgbot: LocalisationUpdate completed (1.17) at Sun Sep 4 02:30:04 UTC 2011

September 3

02:19 logmsgbot: LocalisationUpdate completed (1.17) at Sat Sep 3 02:22:07 UTC 2011

September 2

22:08 logmsgbot: catrope synchronized php/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'r96154'
22:07 logmsgbot: catrope synchronized php/extensions/UploadWizard/includes/UploadWizardCampaign.php 'r96154'
22:07 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.config.php 'r96154'
22:07 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.i18n.php 'r96154'
19:15 logmsgbot: catrope synchronizing Wikimedia installation... :
19:14 RoanKattouw: Running scap to repair any damage that might have been caused by the above
19:13 RoanKattouw: Removed empty line in /etc/dsh/group/mediawiki-installation that seemed to be breaking dsh and therefore break sync
18:59 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
18:52 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 2 18:54:34 UTC 2011
18:17 RoanKattouw: Removing the entire LocalisationUpdate cache and regenerating it from scratch
18:17 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.i18n.php 'r96136'
17:52 RoanKattouw: Put l10nupdate cron job on fenari back at 2:00 UTC
17:31 logmsgbot: catrope synchronized php/includes/parser/Preprocessor_DOM.php 'Pass XML_PARSE_HUGE flag'
16:21 Tim: restarting mailman on lily
15:45 notpeter: starting lighttpd on dataset2
15:11 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 2 15:13:32 UTC 2011
14:53 mutante: srv217 - apache2 kept becoming a defunct zombie, there were apache2 package updates, installed them
14:50 RoanKattouw: Testing new l10nupdate cron job by hacking the time to 14:53 (one minute from now)
14:45 RoanKattouw: Moved l10nupdate cron job from hume to fenari
14:30 mutante: srv217 - there were zombies again "[apache2] <defunct>", kill/restart, hints on debugging?
14:12 Jeff_Green: depooling payments4 for fundraising testing/staging
14:04 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Make $wgLogo protocol-relative on foundationwiki'
13:24 mutante: fenari - loaded apache php module by creating symlinks from mods-available to mods-enabled (to fix http://noc.wikimedia.org/conf/)
12:47 mutante: changed puppet to make fenari include class mediawiki::packages which installs wikimedia-task-appserver (RT#1420)
11:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
11:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30680 - Enable Collection for KM WP'
10:56 mutante: sq59 - power up,dist-upgrade/kernel,squid clean,reboot
10:03 Tim: on lily: reduced /var reserved space to 1% to get an extra 2GB free
09:56 Tim: on lily: did "apt-get clean" to get an extra 1 GB of space and restarted exim4
09:51 Tim: mail on lily is broken due to full /var partition
02:20 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 2 02:22:27 UTC 2011
00:33 logmsgbot: aaron synchronizing Wikimedia installation... :
00:26 logmsgbot: aaron synchronizing Wikimedia installation... :
00:21 AaronSchulz: updated set-group-write2 for het deploy

September 1

23:03 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Setting CentralNotice fundraising banners to use https on foundationwiki'
20:49 logmsgbot: awjrichards synchronized php/extensions/ContributionTracking/ContributionTracking_body.php 'r96038'
20:49 logmsgbot: awjrichards synchronized php/extensions/ContributionTracking/ContributionTracking.processor.php 'r96037'
20:35 maplebed: removed all vim turds (foo~) from within ~/puppet on sockpuppet
19:11 preilly: fix issue with enable/disable links for images on Mobile Frontend
19:11 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix caching issue on enable/disable image links try two'
18:17 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix caching issue on enable/disable image links'
18:17 preilly: fix a caching issue with enable/disable images links
18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Synchronise wmgWikiLoveDefault with wmgUseWikiLove, seems most places want it on by default, not off by default'
15:30 logmsgbot: catrope synchronized php/includes/parser/Preprocessor_DOM.php 'Revert debugging hack'
15:22 logmsgbot: catrope synchronized php/includes/parser/Preprocessor_DOM.php 'udp:// not udp:/ , d'oh'
15:19 logmsgbot: catrope synchronized php/includes/parser/Preprocessor_DOM.php 'Debug DOMDocument exceptions'
14:54 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
14:03 mutante: srv266 - power cycled
13:53 notpeter: removing srv155 from apache pool for bug fix
13:51 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Don't use the protocol-relative interwiki table on secure'
12:26 mutante: sq74 - power up, dist-upgrade/kernel, squid clean,reboot
07:35 Ryan_Lane: dropped the nova database on virt1; restarting virt2, virt3, and virt4
02:16 logmsgbot: LocalisationUpdate completed (1.17) at Thu Sep 1 02:18:53 UTC 2011
00:20 maplebed: started gathering stats about storage blobs in a screen session on fenari. feel free to kill if it's causing problems.

August 31

23:12 preilly: fix the broken links for disable/enable images
23:12 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix broken links'
23:02 preilly: add view that was missed in merge
23:02 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'add view missed in merge'
22:35 maplebed: increased freshness timer from 6 hours to 10 hours to reduce flapping. rt-1408
21:21 awjr: accidentally ran 'rsync -var grosley:/ /' rather than 'rsync -nvar grosley:/ /' on aluminium >_<
21:08 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Enable HTTPSExperiment on commonswiki and set protocol-relative settings'
21:00 logmsgbot: asher synchronized wmf-config/db.php 'putting db44,db45 (s5) and db46,db47 (s6) into production'
20:27 RoanKattouw: Fix ownership of l10n cache directory (from nagios to mwdeploy) on all Apaches using dsh as root
20:26 Ryan_Lane: (I gave Roan permission to run that as root)
20:25 RoanKattouw: rsyncing /h/w/c/php/cache/l10n to the Apaches manually as root to fix permissions/ownership issues (hopefully)
20:12 logmsgbot: LocalisationUpdate completed (1.17) at Wed Aug 31 20:15:00 UTC 2011
20:10 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Fix Apple Touch Icon URL on foundationwiki'
20:05 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable HTTPSExperiment on foundationwiki and make wgServer protocol-relative'
19:56 RoanKattouw: Running LocalisationUpdate again
19:55 logmsgbot: LocalisationUpdate failed
19:48 notpeter: restarting apache on srv155.
19:45 Ryan_Lane: changing all cnames in wikimediafoundation.org to point to foundation-lb and changing the A record as well
19:38 Ryan_Lane: changing cname of wikimediafoundation.org to point to foundation-lb address
19:26 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy HTTPS fixes
19:09 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Define wgCanonicalServer for all wikis, needed for upcoming deployment'
18:33 logmsgbot: LocalisationUpdate failed
18:33 RoanKattouw: Running LocalisationUpdate to hopefully fix Arabic message issue in UploadWizard
18:28 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy MoodBar update
18:25 Ryan_Lane: adding new log collector on locke, running on port 8421, for aft clicktracking
18:24 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.i18n.php 'touch'
17:58 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.i18n.php 'r95893'
17:48 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy WikimediaIncubator fixes
17:46 binasher: started replicating db46 + db47 from db29 (s6 master)
17:31 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying UploadWizard bug fixes
17:18 binasher: started replication on db44 + db45 from db23 (s5 master)
15:03 mutante: db1025 apt-get --reinstall libmysql-common libmysqlclient16 - fixed the broken dpkg status
14:18 mutante: db1025 - dist-upgrade/kernel, kill mysql,reboot..
13:50 mutante: sq35 - dist-upgrade/kernel,squid clean,showed I/O errors,but squid-backend runs now,RAID recovering
12:38 mutante: sq64 - powercycle,dist-upgrade/kernel,squid clean,reboot
12:20 mutante: sq35 - start squid backend but exits again soon after (Exiting due to repeated, frequent failures-> RT 1404
12:10 mutante: srv217 - restart apache ([apache2] <defunct>)
12:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30491'
12:04 mutante: srv266 - power cycle
02:19 logmsgbot: LocalisationUpdate completed (1.17) at Wed Aug 31 02:21:24 UTC 2011

August 30

22:26 Ryan_Lane: scratch that. instances in labs will need LDAP access as well
22:25 Ryan_Lane: remove ldap from virt1, push new schema to svn ldap. point labs at svn ldap.
22:20 binasher: imaged db44-db47
22:12 Ryan_Lane: imported svn ldap to labs ldap
21:09 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
21:06 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'bug 30638, set upload wizard policy page'
19:56 logmsgbot: hashar: previous sync was for r2238: removing old commented out parts
19:55 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php
19:18 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '30523 - Portal namespace for id.wikisource'
19:10 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '30594 - Change namespaces configuration - pl.wikipedia'
19:09 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '30594 - Change namespaces configuration - pl.wikipedia'
18:38 Ryan_Lane: wiping the LDAP database on virt1, re-creating using same domain as svn LDAP.
18:26 logmsgbot: aaron synchronizing Wikimedia installation... :
17:37 logmsgbot: aaron synchronizing Wikimedia installation... :
17:34 logmsgbot: aaron synchronizing Wikimedia installation... :
17:30 logmsgbot: aaron synchronizing Wikimedia installation... :
17:30 aaron: updated scap script file
17:21 Ryan_Lane: removed duplicate ip in badbadip list in squid
16:31 mutante: srv266 - power cycle
16:22 mutante: amssq38 - power up,dist-upgrade/kernel,squid clean,reboot
12:31 Tim: on db9 and db10, raised max_connect_errors from 10 to 1 billion so that it will stop blocking kaulen. These servers are not publically accessible, nobody can SYN flood them
12:25 Tim: ran "flush hosts" on db9 to temporarily fix bugzilla connection error
09:22 RoanKattouw: Reverted yesterday's changes on srv162 (disabled core dumps and suid_dumpable)
02:22 logmsgbot: LocalisationUpdate completed (1.17) at Tue Aug 30 02:25:02 UTC 2011
01:16 binasher: upgrading varnish on mobile caching servers to 3.0.0-1wmf5
00:19 binasher: added new udplog-1.7 pkg to lucid-wikimedia repo, will be upgraded everywhere via puppet on next run

August 29

22:01 awjr: updating payflowpro_gateway.i18n on payments1-4 - r95708
19:41 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95690:
19:41 RoanKattouw: Running scap to deploy Narayam changes
19:03 logmsgbot: neilk synchronized php/includes/Exception.php 'adding these subclasses in an attempt to stop logged errors in prod - r95683'
18:54 RoanKattouw: Re-enabling Apache core dumps on srv162, this time with suid_dumpable enabled
18:18 Jeff_Green: taking payments3 out of production to test a mediawiki config change
18:03 logmsgbot: neilk synchronizing Wikimedia installation... Revision: 95681:
15:36 mutante: srv207 -stop NTP/ntpdate dobson.wm/start NTP (fixes Nagios CRIT), start apache | sq35 -start squid
15:15 mutante: srv278, srv281 - started apache
15:09 mutante: srv207 - was unusable due to overload/freeze - powercycle, dist-upgrade/kernel, puppet run, reboot (log entry from July 31st (RAID issues) not confirmed)
14:57 mutante: srv266 started apache
14:51 mutante: srv281 - power up, dist-upgrade/kernel, puppet run, reboot (note: see 'srv281' in comments of RT#22 and Server_admin_log)
14:39 mutante: srv278 - power up, dist-upgrade/kernel, puppet run, reboot
14:28 mutante: srv266 - power up, dist-upgrade/kernel, puppet run, reboot
14:08 mutante: srv217 - power up, dist-upgrade/kernel, puppet run, reboot
14:06 mutante: nagios-wm - ok, just needed restart to talk again
13:54 mutante: srv188 - power up, dist-upgrade/kernel, puppet run, reboot
13:45 mutante: nagios-wm is on channel but does not speak!? (not ignoring it)
13:45 mutante: srv174 - confirmed hardware failure, new RT#1379, acked in Nagios
13:29 mutante: srv156 - power up, dist-upgrade/kernel, puppet run, reboot
12:57 RoanKattouw: Reverted all of my changes to srv162 and started puppet again. Need to do more to get a core dump, will do that later
09:26 RoanKattouw: ... on srv162
09:26 RoanKattouw: Changed the core dump directory to /a/tmp/apachecore because the root partition doesn't have much free space but /a does
09:23 RoanKattouw: Set up Apache core dumping on srv162 *correctly* by uncommenting CoreDumpDirectory /tmp/apache-core locally in /etc/apache2/wmf/main.conf
09:03 RoanKattouw: Changed ownership of /mnt/upload6/math/8/0/0/800618943025315f869e4e1f09471012.png from root:root to apache:apache, permissions errors were causing PHP warnings
07:39 RoanKattouw: Reverted my changes on srv163 and started puppet
07:38 RoanKattouw: Stopped puppet on srv162, set Apache's cwd to /a/tmp/apachecore in /etc/apache2/envvars , and set ulimit -c 1000000 in /etc/default/apache2
07:34 RoanKattouw: Moving my core dump for segfault debugging test to srv162 instead of srv163, for disk space reasons
07:32 RoanKattouw: Stopped puppet on srv163 to prevent it from reverting my hacks
07:26 RoanKattouw: Restarting Apache on srv163 so these changes take effect
07:26 RoanKattouw: Enabled core dumps for Apache on srv163 by editing /etc/default/apache2
07:19 RoanKattouw: Changing Apache's cwd on srv163 by editing /etc/apache2/envvars
02:18 logmsgbot: LocalisationUpdate completed (1.17) at Mon Aug 29 02:20:17 UTC 2011

August 28

17:55 logmsgbot: ariel synchronized php-1.17/includes/upload/UploadFromStash.php 'fix fatal Call to a member function getId() on a non-object'
02:23 logmsgbot: LocalisationUpdate completed (1.17) at Sun Aug 28 02:26:07 UTC 2011

August 27

02:21 logmsgbot: LocalisationUpdate completed (1.17) at Sat Aug 27 02:23:37 UTC 2011

August 26

21:06 mutante: amssq48 - power back up, clean squid, dist-upgrade
16:37 robh: updating text-settings to move sq36 into the squid api cluster. puppet updated already for the same, and pybal updated to remove sq36 frontend from normal text service
02:27 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 26 02:29:15 UTC 2011
00:11 robh: change reverted, nothing bad, but undesired result. hooper back to normal
00:09 robh: hooper apache config change for https redirection on etherpad
00:09 robh: i meant to paste the rt link
00:09 robh: testing something in hooper apache config, should result in nothing noticeable to users, unless i did it wrong.
00:08 maplebed: changed puppet client run interval from the default (30m) to 2hrs to reduce load on the master.

August 25

23:29 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 95505:
23:20 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 95505:
23:10 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
23:09 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php
23:08 logmsgbot: awjrichards synchronized php/extensions/LandingCheck/LandingCheck.i18n.php 'r95542'
23:08 logmsgbot: awjrichards synchronized php/extensions/LandingCheck/SpecialLandingCheck.php 'r95542'
22:48 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 95505:
22:47 mark: Rebooting lvs1002, lvs1003, lvs1005, lvs1006
22:43 robh: gallium deployed for continuous integration testing per RT#1204. Requires further development input for final system configuration.
22:29 mark: Deployed cr2-pmtpa as backup bootp forwarder and PIM router
22:24 mark: Deployed cr2-pmtpa as backup VRRP router on all production subnets
22:05 mark: Enabled cr2-pmtpa:irb.105 (subnet virt-hosts) family inet; activated cr2-pmtpa as backup VRRP router for that subnet
21:54 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/centralnotice.css 'r95518'
21:48 mark: Setup iBGP on cr2-pmtpa to all other AS14907 routers
21:46 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/centralnotice.js
21:41 Jeff_Green: taking payments4 out of the pool for testing
21:13 robh: installing gallium for CI testing.
19:52 logmsgbot: catrope synchronized php/includes/api/ApiQuerySiteinfo.php 'Fix PHP warning'
19:49 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95505:
19:49 RoanKattouw: Run scap to deploy HTTPS / prot rel fixes
19:22 awjr: deployed contrib log auditing fixes on Grosley
19:20 logmsgbot: catrope synchronized wmf-config/secure.php 'Factor out URL fixing into fixupUrl, add hook functions for GetCanonicalURL and IRCLineURL'
19:13 logmsgbot: catrope synchronized wmf-config/secure.php 'rm debugging'
19:12 logmsgbot: catrope synchronized wmf-config/secure.php 'Fix setting of $wgInternalServer for protocol-relative URL wikis (was broken since introduction of prot rel) and set $wgCanonicalServer'
19:05 logmsgbot: catrope synchronized wmf-config/secure.php 'Debugging'
18:54 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak wmgArticleFeedbackBlacklistCategories, thanks Krinkle'
18:54 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgArticleFeedbackBlacklistCategories line for enwiki'
17:12 Ryan_Lane: pushed out workaround for CVE-2011-3192
14:43 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'touch'
14:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30418 - Enable ArticleFeedback on Hungarian Wikipedia'
14:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30523 - Portal namespace for id.wikisource'
13:11 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable WikimediaIncubator on testwiki, is causing issues'
10:47 logmsgbot: catrope synchronized php/languages/messages/MessagesPi.php 'r95472'
02:20 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 25 02:22:40 UTC 2011
02:02 mutante: sq77,sq81,sq85 - power back up, squid clean, dist-upgrade
01:46 mutante: sq52,sq54 - power back up, squid clean, dist-upgrade
00:10 robh: manually pushed ntpdate on dataset1 cuz its ugly red in nagios.

August 24

19:32 logmsgbot: catrope synchronized wmf-config/CommonSettings.php
19:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable WikimediaIncubator on incubatorwiki'
19:07 robh: sq34 unresponsivle to serial console, rebooted
19:04 robh: sq34 offline, no rt ticket or admin log entry for a reason to be offline, investigating
19:03 logmsgbot: catrope synchronized php/extensions/WikimediaIncubator/IncubatorTest.php 'r95416'
18:37 logmsgbot: catrope synchronized php/extensions/WikimediaMessages/WikimediaMessages.php 'r95411, right path this time'
18:37 apergos: doing testing of some new xml dump code on snapshot4 on root, it's writing locally so don't worry about jumps in local disk usage
18:36 logmsgbot: catrope synchronized php/includes/EditPage.php 'r95411'
18:28 Ryan_Lane: powercycling virt3
17:47 Ryan_Lane: disabling access_log on srv193
17:41 Ryan_Lane: temporarily enabled the access log on srv193
17:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Tidying up wmgUseAbuseFilter'
17:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Setting wmgUseAbuseFilter to true for all wikis'
17:06 Reedy: About to enable AbuseFilter accross all wikis
16:38 maplebed: running a puppet experiment - turning off the puppet client on spence from now until 2pm - nagios operation will remain unaffected
16:24 maplebed: added roan's key to root's authorized keys
15:58 robh: rebooting emery, appears to be hung in boot, but cannot see due to serial console not retaining a backlog when not connected
15:37 mark: Reenabled cr1-sdtpa:xe-0/0/0
14:53 mark: Disabled cr1-sdtpa:xe-0/0/0 due to port errors
14:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30538 - Change Cantonese Wikipedia (zh-yue.wikipedia) autoconfirmed threshold'
09:38 logmsgbot: catrope synchronized php/extensions/MobileFrontend/MobileFrontend.php 'r95388'
02:19 logmsgbot: LocalisationUpdate completed (1.17) at Wed Aug 24 02:21:20 UTC 2011

August 23

23:49 apergos: limited downloads on dataset2 to 2 simultaneous connections per ip, except our hosts
23:12 apergos: forgot to log earlier, limited bandwidth in lighttpd on dataset2 per download except for our hosts
21:48 Ryan_Lane1: moved catrope from mortals to roots
21:12 notpeter: upgrading squid and squid-frontend on sq86
20:10 notpeter: upgrading squid and squid-frontend on sq85
19:56 logmsgbot: demon synchronized php/includes/specials/SpecialEmailuser.php 'r95328'
18:22 notpeter: upgrading squid and squid-frontend on sq83
18:03 robh: wiped old binlogs off db9 older than 20 days
17:16 maplebed: installed a new /etc/sudoers on db36 to placate nagios
16:36 apergos: added webzilla.com hosts to urls denied on dataset2, bulk garbage requests and using a pile of bandwidth
16:27 robh: reviewed paniclog on lily and mchenry due to non-zero size. cleared logs to stop email alerts
10:58 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable WikimediaIncubator on testwiki for debugging'
10:51 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable WikimediaIncubator again'
10:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Comment out $wmincClosedWikis'
10:38 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable WikimediaIncubator extension so I can do some profiling'
10:37 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Add profiling group for debugging incubatorwiki slowness, triggered by a request parameter'
10:30 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily disable WikimediaIncubator extension due to slowness on incubatorwiki'
09:15 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95304:
09:15 RoanKattouw: Running scap to undo the accidental deployment of yesterday's failed UploadWizard deployment
02:19 logmsgbot: LocalisationUpdate completed (1.17) at Tue Aug 23 02:21:27 UTC 2011
00:28 ^demon: ran refresh-dblist and sync-dblist
00:21 ^demon: fixed refresh-dblist since wmf-config was moved

August 22

22:39 RoanKattouw: Added UploadWizard tables to testwiki, for real this time
22:32 preilly: pushing fix for disable link
22:32 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/information/disable.html.php 'issue with en always being the target of the form'
22:21 preilly: remove content type header for now from XHTML view on Mobile Frontend
22:21 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'remove content type header for now'
22:07 notpeter: upgrading squid and squid-frontend on sq55
21:45 notpeter: upgrading squid and squid-frontend on sq52
20:59 logmsgbot: catrope synchronized php/extensions/TitleBlacklist/api/ApiQueryTitleBlacklist.php 'r95249'
20:30 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95237:
20:30 RoanKattouw: Deploy new TitleBlacklist code (trunk version) site-wide
20:22 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php 'Bug 30507 - Please lock flaggedrevs.labs.wikimedia.org'
20:19 logmsgbot: jeluf synchronized closed.dblist
20:17 notpeter: restarting torrus. again....
20:13 RoanKattouw: ..or not. The SQL is not in 1.17wmf1 yet, will have to wait a few minutes
20:09 RoanKattouw: Creating tables for UploadWizard on testwiki and commonswiki
20:09 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php 'Bug 30509 - Closure of Asturianu Wikibooks'
20:08 logmsgbot: jeluf synchronized closed.dblist
20:08 logmsgbot: jeluf synchronized closed.dblist
20:00 notpeter: restarting torrus
19:33 logmsgbot: catrope synchronized php/extensions/WikimediaIncubator/IncubatorTest.php 'r95234'
19:08 notpeter: upgrading squid and squid-frontend on sq50
18:47 RoanKattouw: Fixed /h/w/bin/scap to update ExtensionMessages-1.17.php instead of ExtensionMessages.php
18:44 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.17.php 'Sync the right file'
18:37 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $mincClosedWikis for Incubator extension per SPQRobin'
18:35 logmsgbot: catrope synchronized wmf-config/ExtensionMessages.php 'Fix i18n issue with WikimediaIncubator, hopefully'
18:21 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95216:
18:21 RoanKattouw: Running scap to deploy WikimediaIncubator bugfixes
18:06 Jeff_Green: added fundraising-analytics.wikimedia.org cname for grosley for RT #854
17:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable HTML5 on mediawikiwiki. Feel free to revert if it breaks too much'
16:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikilove on commons per http://commons.wikimedia.org/wiki/Commons:Village_pump#WikiLove'
15:23 logmsgbot: catrope synchronized php/extensions/WikimediaMessages/WikimediaGrammarForms.php 'r95209'
13:03 Tim: set up ufw on ersch and alsted
11:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Turn wikilove on by default on mlwiki'
11:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'mlwikipedia to mlwiki'
11:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30473 - Please create the rollback group at es.wikibooks'
11:35 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'touch startup for good measure'
11:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30500 - Enable WikiLove Extension on Malayalam Wikipedia'
11:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30491 - Enable WikiLove on Swedish Chapter Wiki'
07:12 Tim: on ersch and alsted: set up a temporary firewall to block external access to miscellaneous swift services such as the object servers
06:41 Tim: on alsted and ersch: adjusted rsyncd.conf to deny access from outside the cluster and restarted rsync
06:22 Tim: on ersch: reconfigured MW to send its debug log to a place which is not on the public web, and did a chmod 000 in case the config is reverted
06:18 Tim: on ersch apache2.conf: deny access to vim editor backup files
02:21 logmsgbot: LocalisationUpdate completed (1.17) at Mon Aug 22 02:23:31 UTC 2011

August 21

18:41 logmsgbot: reedy synchronized php-1.17/extensions/OAI/OAIRepo_body.php 'r95170 for bug 26304'
18:36 rainman_: restarted lucene on search7 which somehow got hung
02:17 logmsgbot: LocalisationUpdate completed (1.17) at Sun Aug 21 02:20:08 UTC 2011

August 20

11:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29834 - Reconfigured AF for hiwiki'
02:21 logmsgbot: LocalisationUpdate completed (1.17) at Sat Aug 20 02:23:29 UTC 2011

August 19

23:43 logmsgbot: asher synchronized wmf-config/db.php 'adding srv155 back to ext store'
20:00 logmsgbot: laner synchronized wmf-config/db.php 'Repooling db12'
19:58 Ryan_Lane: pushing squid config
19:56 Ryan_Lane: banning a bad ip in squid list
19:49 RoanKattouw: Ryan uncommented srv155 and srv156 from the ES list at 19:11 today
19:28 logmsgbot: laner synchronized wmf-config/db.php 'Removing db12 temporarily to let it catch up on replication'
19:16 logmsgbot: laner synchronized wmf-config/db.php 'srv156 is also down, removing it from db.php'
19:09 logmsgbot: laner synchronized wmf-config/db.php 'removing srv155 from external store list'
18:47 logmsgbot: reedy synchronized wmf-deployment/languages/Names.php 'r94830 r95027'
18:46 logmsgbot: reedy synchronized wmf-deployment/languages/messages/MessagesQug.php 'r94830'
17:39 mutante: sq53,sq58 - power back up, squid clean, dist-upgrade
17:35 notpeter: adding srv155 back into pool for testing
17:10 notpeter: upgrading squid and squid-frontend on sq49
16:08 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'Touch startup.js'
16:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ClickTracking on huwiki as dependancy of AF'
14:08 RobH: confirmed serial on scs-c1-pmtpa working for asw-d2-pmtpa, ps1-d2-pmtpa, & cr2-pmtpa
13:16 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'Touch'
13:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix'
13:11 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert'
13:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert'
13:10 Reedy: Fail summary. Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia. Also, add wmg global for $wgArticleFeedbackBlacklistCategories
13:10 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia. Also, add wmg Global for Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia'
13:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia. Also, add wmg Global for Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia'
11:07 Andrew: (in a screen on hume)
11:07 Andrew: Sending 753994 emails to qualified voters in the referendum.
10:57 mark: Converted routing subinterfaces on cr2-pmtpa:ae0 to IRB and configured bridge-domains
02:19 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 19 02:21:42 UTC 2011
01:17 Jeff_Green: flipped $wgContributionReportingDBserver back to db10
00:39 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'Touching startup.js'
00:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29834 Enable AF on hiwiki - Enabling ClickTracking first'

August 18

23:24 notpeter: adding srv155 back to the pool after upgrade to 10.04 and testing
22:43 notpeter: upgrading squid and squid-frontend on sq48
22:01 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94793:
22:00 RoanKattouw: Running scap to deploy WikiLove changes
21:43 logmsgbot: catrope synchronized php/resources/jquery.ui/themes/vector/jquery.ui.button.css 'r94947, JUI CSS fixes for WikiLove'
21:03 Jeff_Green: db10 is re-cloned with innodb-file-per-table enabled and a significantly smaller db footprint, /a is down from 94% to 88% use
20:34 RobH: serial connection to cr2-pmtpa tested working. port 13 on scs-c1-pmtpa
16:21 mark: Allocated IPs for pmtpa row D PDUs and access switches
11:15 Jeff_Green: stopping mysql on db10 for re-clone
11:04 Jeff_Green: swinging $wgContributionReportingDBserver from db10.pmtpa.wmnet to db1008.eqiad.wmnet
02:22 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 18 02:24:15 UTC 2011

August 17

23:59 mutante: deleted srv173 manually from db9 hosts and resources so Nagios would start again
22:53 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'Touch startup'
22:31 preilly: pushing updates to Mobile Frontend for opt-in
22:31 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94793:
22:01 binasher: deploy of frontend.conf to all squids
21:43 Reedy: That was Setting $wgCategoryTreeDisableCache to false. Stupid escaping
21:42 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Setting to false'
21:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable poolcounter for commmonswiki'
21:31 maplebed: extended logic in mysql.pp to always set variables used in my.cnf.erb
21:22 Reedy: Added PoolCounterSettings.php to noc
21:04 Reedy: Allowed StartProfiler.php to be shown on noc
21:02 mutante: sq57 - power back up, clean squid, dist-upgrade
20:53 notpeter: upgrading squid and squid-frontend on sq47
20:50 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94793:
20:49 RoanKattouw: Running scap to deploy Neil's UploadWizard/UploadStash changes
20:43 RoanKattouw: Neil added uploadstash table to all wikis
20:42 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 29983 - Install ArticleFeedback on Portuguese Wikibooks (ptwikibooks). Enabling ClickTracking'
20:37 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 29983 - Install ArticleFeedback on Portuguese Wikibooks (ptwikibooks). Enabling ClickTracking'
20:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable PoolCounter for Commons'
20:24 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 29983 - Install ArticleFeedback on Portuguese Wikibooks (ptwikibooks)'
20:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 29983 - Install ArticleFeedback on Portuguese Wikibooks (ptwikibooks)'
20:14 Reedy: Disabled ReaderFeedback on ptwikibooks, part of bug 29983
19:53 Reedy: Creating ArticleFeedback tables on huwiki
19:53 Reedy: Creating ArticleFeedback tables on ptwikibooks
19:52 Reedy: Creating ArticleFeedback tables on hiwiki
19:16 logmsgbot: catrope synchronized php/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r94792'
19:11 binasher: repooled srv264
19:07 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/backend/CodeRevision.php 'r94793'
19:07 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/CodeReview.i18n.php 'r94793'
18:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'bug 30417 setting use wikilove default to true for huwiki'
18:43 logmsgbot: reedy synchronized php-1.17/includes/parser/ParserCache.php 'Revert debugging here too'
18:41 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Disabling parsercache debug logging. Nothing of interest'
18:32 binasher: pulling srv264 from lvs for testing
18:27 logmsgbot: reedy synchronized php-1.17/includes/parser/ParserCache.php 'Add in a not saved in parser cache message if appropriate'
18:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Log parsercache log groups to file'
18:18 logmsgbot: reedy synchronized php-1.17/includes/parser/ParserCache.php 'Hack in some logging for when parser output is uncacheable'
14:26 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Keep $wgInternalServer on http when visiting wikis that do not have a protocol-relative $wgServer over https. For bug 29925'
14:22 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove debugging code'
14:15 logmsgbot: catrope synchronized php/includes/RecentChange.php 'r94755, fix prot rel links in IRC feeds'
13:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'More debugging'
13:24 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix stupid mistake in debugging hack'
13:23 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add temp debugging hack for $wgInternalServer'
13:14 logmsgbot: catrope synchronized wmf-config/secure.php 'Remove debugging hack'
13:12 logmsgbot: catrope synchronized wmf-config/secure.php 'tweak hack'
13:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30417 Enable WikiLove on huwiki'
13:10 logmsgbot: catrope synchronized wmf-config/secure.php 'Debugging code so I can figure out what $wgInternalServer is on commons via secure'
12:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30362 Enable newusermessage and wikilove on zhwiki'
12:22 mark: restarted etherpad on hooper
02:27 logmsgbot: LocalisationUpdate completed (1.17) at Wed Aug 17 02:29:32 UTC 2011

August 16

22:31 notpeter: pulling srv155 from pybal so I can take it down and upgrade to 10.04
22:10 notpeter: upgrading squid and squid-frontend on sq45 and sq46
21:55 mutante: resolved RT 1274 - we now get mail incl. the diff and comment after every svn commit on sockpuppet
21:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'For bug 26304 set oaiChunkSize to 40'
21:32 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesEn.php 'Push r94698'
18:48 mutante: sq61,sq73 - power back up, clean squid, dist-upgrade, rebootb
18:12 mutante: sq40,sq56 - power back up, clean squid, dist-upgrade, reboot
18:10 preilly: pushing updates to Mobile Frontend extension to allow for search page to work
18:10 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94562:
18:08 binasher: deploying frontend.conf to all squids - change to mobile acl to support opt-in trial
17:53 mutante: amssq42,amssq61 - power back up, clean squid, dist-upgrade, reboot
17:38 mutante: added new class svn::server:notify which installed libsvn-notify-perl on sockpuppet (to get svn commit mails in the future)
17:29 logmsgbot: demon synchronized php/extensions/CodeReview/backend/CodeRevision.php 'r94664'
16:23 mark: Increased prefix limit on BGP session between br1-knams and csw2-esams from 10 to 50 prefixes
16:00 RobH: lvs1004, lvs1005, lvs1006 both network patches attached
15:57 mark: Added Nagios purge script on 'reload' action in its init script
15:44 mark: Setup switchports for cross rack row connections from lvs1001-1006 on asw-a-eqiad and asw-b-eqiad
15:40 Jeff_Green: package upgrade on payments4, no reboot
14:21 mark: Noticed the mobile LVS service had a depool-threshold set at '1.5', preventing it from depooling any hosts, ever. Corrected it by setting it to .6
14:07 mark: Setup 'new-mobile' LVS service on lvs1001 and lvs1004
14:07 mark: Setup eqiad LVS cluster (classes 'high-traffic' and 'testing') consisting of lvs1001 (A4) and lvs1004 (B4); still awaiting cross-row connections
13:54 mark: Increased bgp group PyBal multihop ttl from 1 to 2 on all core routers
13:54 mark: Configured PyBal bgp group and policy statements on cr1/cr2-eqiad
13:14 RobH: reviewed exim paniclog on lily and williams, both from a refusal in spam acl, over 24 hours ago, and no issues since. cleared each paniclog back to 0
11:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'bug 30392: Add Wikipedia_talk and MediaWiki_talk aliases on zh-yuewiki'
10:42 mark: Corrected DNS for lvs1004-1006
10:03 mark: Setup switchports for lvs1001-1006
02:17 logmsgbot: LocalisationUpdate completed (1.17) at Tue Aug 16 02:19:43 UTC 2011
00:37 mutante: gmond.conf was an empty file on search12. copied it in place and restarted gmond

August 15

23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30320 Enable AbuseFilter for cswiki'
23:34 Reedy: Testwiki job queue is now empty
22:57 preilly: pushing Mobile Frontend updates for mobile friendly main page
22:57 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94562:
21:58 logmsgbot: reedy synchronized wmf-config/codereview.php 'Auto defer semantic tags'
21:35 notpeter: upgrading squid and squid-frontend on sq43 and sq44
21:15 notpeter: upgrading squid and squid-frontend on sq41 and sq42
21:14 RoanKattouw: Fixed scap so mergeExtensionMessages didn't fail anymore. Was broken due to het deploy changes
21:14 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94562:
21:12 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94562:
21:12 RoanKattouw: Ended up moving extensions/SecurePoll/auth-api.php.save to /home/catrope . I could do that because I have write access to the parent directory, even though I don't have read access to the file
21:07 RoanKattouw: Permissions for extensions/SecurePoll/auth-api.php.save are messed up, breaking scap. Running set-group-write in an attempt to fix it
21:01 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94562:
21:01 RoanKattouw: Running scap to push my latest 1.17wmf1 commits plus Patrick's MobileFrontend changes site-wide
20:31 RoanKattouw: Running svn up on fenari to deploy my recent 1.17wmf1 revisions to testwiki
19:19 Ryan_Lane: assigning svc ips for bits, upload, and text in 10.2.2.x range for eqiad
18:25 notpeter: upgrading squid and squid-frontend on knsq20 and knsq21 and knsq22
18:10 notpeter: upgrading squid and squid-frontend on knsq18 and knsq19
18:04 preilly: pushing mobile fronted changes for image enable disable
18:04 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94253:
17:53 mutante: added libnet-stomp-perl to puppet to be installed on spence
17:49 notpeter: upgrading squid and squid-frontend on knsq16 and knsq17
17:04 RobH: updating dns with lvs1001-1006 mgmt info
16:58 Ryan_Lane: pushing varnish changes to allow XFF from HTTPS cluster
15:39 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/auth-api.php
15:38 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/auth-api.php
15:36 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/auth-api.php
15:34 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/auth-api.php
15:25 RobH: updating dns for lvs1001-lvs1006
15:19 RobH: restarted dns on ns2
15:04 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/includes/ballots/Ballot.php 'Deploy r94343'
15:04 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/SecurePoll.php 'Deploy r94343'
15:03 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/includes/ballots/RadioRangeCommentBallot.php 'Deploy r94343'
14:37 mark: Enabled NAT on the pmtpa management LAN
12:10 mark: Setup ongoing rsync of ms7 images to ms1002
12:03 mark: Setup rsyncd on ms7
02:17 logmsgbot: LocalisationUpdate completed (1.17) at Mon Aug 15 02:19:19 UTC 2011

August 14

17:49 JeLuF: rebooted srv183
17:30 JeLuF: stopped apache and jobrunner on srv183, logfiles indicate memory problems
17:18 logmsgbot: jeluf synchronized wmf-config/mc.php 'replace srv183 by srv229'
13:22 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Make $wgServer protocol-relative on wikimania2005wiki, for consistency with the other HTTPS experiment wikis'
10:12 JeLuF: rebooting sq40 using echo b > /proc/sysrq-trigger, syslog shows lots of SCSI errors
09:10 apergos: stopped and restarted lsearchd on search7. had to shoot the old one, it wouldn't die
02:19 logmsgbot: LocalisationUpdate completed (1.17) at Sun Aug 14 02:21:23 UTC 2011

August 13

21:23 RoanKattouw: sq40 keeps flapping in Nagios all the time
10:39 RoanKattouw: ...on enwiki
10:39 RoanKattouw: Running a hacked-up version of extensions/ArticleFeedback/populateAFRevisions.php to fix bug 30227 (data corruption in ArticleFeedback tables), using a list of affected revisions produced on the toolserver
02:16 logmsgbot: LocalisationUpdate completed (1.17) at Sat Aug 13 02:18:42 UTC 2011
00:22 notpeter: upgrading squid and squid-frontend on knsq12 and knsq15
00:19 notpeter: upgrading squid and squid-frontend on knsq10 and knsq11

August 12

23:15 preilly: pushing fix for mobile WML view and disable caching on cookie set page
23:14 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94253:
22:40 notpeter: upgrading squid and squid-frontend on knsq8 and knsq9
21:54 notpeter: upgrading squid and squid-frontend on amssq59 and amssq62
21:46 preilly: updating the mobile frontend extension on production
21:46 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94253:
21:31 maplebed: localization update permissions changes complete and tested.
21:29 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 12 21:31:45 UTC 2011
21:26 AaronSchulz: Running LU to confirm permission fix
21:25 notpeter: upgrading squid and squid-frontend on amssq56 and amssq58
21:17 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 12 21:19:39 UTC 2011
21:10 maplebed: changing ownership of /usr/local/apache/common-local/php-1.17/cache/l10n from nagios to mwdeploy on all affected srv hosts
21:03 binasher: deploying new squid frontend.conf - bypass mobile redirector in case of trial opt-in/out pages
20:53 notpeter: upgrading squid and squid-frontend on amssq51 and amssq52
20:52 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 12 20:54:15 UTC 2011
18:44 maplebed: made ariley an account on rt to submit tickets for email list maintenance
18:24 JeLuF: DNS: added bugs.mediawiki.org as alias to text.wikimedia.org
17:25 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30268 - Point eowp logo to Wiki.png'
16:25 logmsgbot: mark synchronized wmf-config/CommonSettings.php 'Raise multicast ttl from 2 to 8'
16:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgVaryXFPForAPI = true for HTTPS experiment wikis. This splits the API Squid cache between HTTP and HTTPS, fixing cache pollution issues'
15:08 mark: Turning off ethernet hw offloading GRO on all lvs servers with Puppet
14:30 mark: Turned off all forms of hardware segmentation on lvs4, fixing the slow upload problem
14:29 mark: Turned tcp segment offloading back on on sq51..86
13:57 mark: Manually turned off TCP segmentation offloading on sq51..86
12:59 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Remove upload profiling, hasn't produced any useful data'
12:52 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Profile uploads on officewiki separately, I wanna try something'
11:39 apergos: reran ppuppet by hand on spence, sq32 entries in nagios conf files were not recreated, restarted nagios, seems to be running
11:11 apergos: er... because sq32 is in the decommissioned list but the script to purge resources from nagios is broken right now, which means nagios fails to start
11:10 apergos: purged sq32 resources and host references from puppet db manually on db9, and from nagios conf files on spence. will run puppet manually on spence shortly
10:52 Andrew: (screen on hume)
10:52 Andrew: Running populatePifEditCount.php on all wikis
10:45 Andrew: Adding pif_edits table to all wikis for personal image filter voter list
10:39 mark: Enabled cr1-sdtpa:xe-0/0/2; a cross connect has been ordered, expect Nagios to complain
10:24 apergos: uncommented the monitor_group line in varnish.pp which defines the cache_mobile_eqiad group in puppet (thanks ma rk), will run puppet shortly on spence
08:44 apergos: revert change to site.pp, try applying to spence
08:29 apergos: doing repeated manual runs of puppet on spence til we catch up to current config (it is quite out of date)
07:21 apergos: nagios was failing to start because of unknown host group cache_mobile_eqiad in /etc/nagios/puppet_hosts.cfg; commented out line $nagios_group = "cache_mobile_${site}" in site.pp, waiting for puppet run to complete on spence
02:15 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 12 02:17:38 UTC 2011

August 11

21:24 JeLuF: added 'ttf-ubuntu-font-family' to the list of required packages for image scalers in puppet (bug 30288)
21:07 JeLuF: virt2 root filesystem has switched to read-only due to a disk failure
21:03 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 21:06:09 UTC 2011
20:24 binasher: re-pooling mobile2
20:16 logmsgbot: LocalisationUpdate failed
18:57 binasher: depooling mobile2 from lvs for mobile extension opt in proxy conf testing
18:19 preilly: pushing new mobile frontend changes to production
18:19 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94253:
18:12 mark: Temporarily serving li.wikipedia.org from srv153 (bypassing LVS) as backend on the text squids
18:02 mark: Test complete, change reverted
17:57 mark: Temporarily moved squids->apaches LVS traffic from lvs4 to lvs3 for testing
16:48 Ryan_Lane: upping the nginx upload size too 100m for the https and ipv6 cluster
16:06 Reedy: SVN and related services should be ok now. High HTTP load causing OOM
15:51 Reedy: SVN may be unavailable due to issues with Formey
15:32 logmsgbot: catrope synchronized php/includes/filerepo/LocalFile.php 'r94252 by Chad - Try wrapping ss_images update in a transaction'
15:17 RoanKattouw: All file uploads were returning HTTP 500 errors between 15:07 and 15:16 UTC, my apologies. It's fixed now
15:16 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Fix it for real this time'
15:15 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Unbreak uploads, oops'
15:07 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Make upload profiling 1:1 instead of 1:50'
15:01 mark: Set maximum TCP window size on nas1-a
15:00 RoanKattouw: ...and that worked. Lesson of the day: it seems you can't use dashes in your profile IDs
14:59 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Remove dashes from profile IDs just because I'm paranoid'
14:56 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Add profiling groups upload-commons and upload-other, based on count($_FILES)'
14:10 mark: Setup semi-sync snapmirror replication from nas1001-a:images to nas1-a:images, started initial transfer
13:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 8886 - Install DynamicPageList extension for Vietnamese Wiktionary'
13:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 18390 Set [bureaucrat][] = sysop on enwiki'
13:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30307 Allow bureaucrats to remove sysop flag on Russian Wikipedia'
12:45 mark: Migrated nas1-a and nas1-b to new 64 bit root volumes
09:45 logmsgbot: catrope synchronized wmf-config/checkers.php 'Remove broken profiling in wfACLBlocks. Not needed because everything in $wgExtensionFunctions is automatically profiled anyway'
09:17 mutante: sq75 - power cycle, squid clean
03:00 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 03:02:09 UTC 2011
02:32 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 02:34:20 UTC 2011
02:31 Reedy: LocalisationUpdate is giving Warning: array_diff_assoc(): Argument #1 is not an array in /home/wikipedia/common/php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php on line 373
01:58 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 02:00:53 UTC 2011
01:56 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 01:58:21 UTC 2011
01:46 logmsgbot: LocalisationUpdate completed at Thu Aug 11 01:48:34 UTC 2011
01:46 logmsgbot: LocalisationUpdate failed
01:30 logmsgbot: LocalisationUpdate failed
01:27 logmsgbot: LocalisationUpdate failed
01:12 logmsgbot: LocalisationUpdate failed
00:42 logmsgbot: demon synchronized wmf-config/StartProfiler.php 'Enabled profiling group for commonswiki'

August 10

23:02 logmsgbot: LocalisationUpdate failed
22:59 logmsgbot: LocalisationUpdate failed
22:50 aaron: testing/debugging Het Deploy updates to l10 script
21:25 preilly: pushing device change back into production
21:25 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960:
20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert my change'
20:49 preilly: fixing mobile view
20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Temporarily disabling MObileFrontend'
20:49 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960:
20:48 RoanKattouw: MobileFrontend broke, serving mobile pages to everyone. Fixing
20:43 preilly: pushing fix for Mobile Frontend device detection
20:43 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960:
19:02 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960:
19:02 preilly: updating mobile frontend extension on production
18:41 notpeter: upgrading squid and squid-frontend on amssq49
17:01 mark: Deployed new HTCP loss ganglia monitoring plugin on ganglia aggregator squids in pmtpa
16:32 mutante: sq51 & sq62 - dist-upgrade (incl. kernel) / reboot
16:01 mutante: sq51,sq62: power cycle, squid clean
02:18 logmsgbot: LocalisationUpdate completed at Wed Aug 10 02:21:00 UTC 2011
00:12 maplebed: trying out a change to puppet's nagios config generation - moving service checks to one file per host.

August 9

19:01 logmsgbot: catrope synchronized php/extensions/ArticleFeedback/SpecialArticleFeedback.php 'Add missing !'
18:33 notpeter: upgrading squid and squid frontend on amssq47 and amssq48
17:41 maplebed: added microsecond query duration logging to sockpuppet's apache log format
02:25 logmsgbot: LocalisationUpdate completed at Tue Aug 9 02:27:12 UTC 2011

August 8

20:41 binasher: added self to auth portion of nagios cgi.cfg
20:35 binasher: powered off knsq30, disabling in nagios
19:23 logmsgbot: reedy synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'blanking file'
18:51 logmsgbot: tfinc synchronized wmf-config/CommonSettings.php 'Disabling WikimediaMobile client side redirect'
18:49 logmsgbot: tfinc synchronized wmf-config/InitialiseSettings.php 'Disabling WikimediaMobile client side redirect'
18:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 25957 - Update'
17:57 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'bug 29911'
17:46 logmsgbot: reedy synchronized php-1.17/languages/Names.php 'r94069'
17:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix language code for bug 30045'
17:15 Ryan_Lane: RobH is removing the sep11.wikipedia.org subdomain redirect. It didn't post to the SAL, and he can't get onto IRC
13:53 mark: Fixed IPv6 router advertisements on cr1-sdtpa
11:57 mark: Killed squid on knsq30
11:52 mutante: restarted squid on knsq30
11:29 mark: Restored routing of pmtpa - esams path
11:19 mark: Restarted lsearchd on search7
02:18 logmsgbot: LocalisationUpdate completed at Mon Aug 8 02:21:05 UTC 2011

August 7

07:56 mark: Rerouted pmtpa <--> esams traffic in both directions
02:22 logmsgbot: LocalisationUpdate completed at Sun Aug 7 02:24:47 UTC 2011

August 6

15:52 mutante: amssq41 - and while we're on it: dist-upgrade incl. kernel and reboot. did not include squid package, that was downgraded on Apr 20
15:25 mutante: amssq41 - power back up after it went down, clean squid cache
13:07 RobH: puppet run on spence is via screen on root user
13:05 RobH: spence is running puppet to update with new nagios info that won't contain the facilities fasle reports. if it fails to work, ~/rob_puppet_services.cfg is on spence
13:03 RobH: nagios is being regenerated, i hope this works!
12:23 RobH: srv169 depooled, needs work and to be restored to service
12:10 logmsgbot: root synchronized wmf-config/mc.php 'replacing srv169 in active mc role due to its constant flapping'
06:46 RobH: emptied the /var/log/exim4/paniclog on project2, formey, lily, mchenry, sockpuppet, williams, srv196. All errors were quite old due to various network blips over time.
02:19 logmsgbot: LocalisationUpdate completed at Sat Aug 6 02:21:34 UTC 2011
01:49 Reedy: srv169 seems to be flapping
01:48 Reedy: Ganglia seems brokened "There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused"

August 5

23:50 tfinc: removed knsq30 from LVS esams rotation
23:36 Reedy: Even though Asher pulled knsq30 from rotation, it seems to be still serving users
21:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'more whitespace'
21:33 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'whitespace'
21:32 ^demon: fixed permissions on new hook scripts so they're executable
21:28 ^demon: deployed new svn pre-commit hook framework
21:01 binasher: pulled knsq30 from all frontend squids
20:44 logmsgbot: root synchronized wmf-config/db.php 'returning db18 to s7'
19:02 apergos: power cycled dataset2, it was unresponsive to pings and form mgmt console
18:24 binasher: squid frontend deploy - no mobile redirect for android 3 (tablets)
18:15 Reedy: running namespaceDupes on plwikibooks
18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30164 Change namespaces configuration - pl.wikibooks'
18:00 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Tidy up some redundant userrights definitions'
17:56 Reedy: That sync was also bug 30207 Enable AbuseFilter on Russian Wikinews
17:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:55 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'bug 30207 Enable AbuseFilter on Russian Wikinews'
17:51 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Remove old abusefilter-private rights entries'
17:48 binasher: deploying squid frontend.conf (commented out nokia mobile acl until a jquery netfront browser compatibility issue is fixed)
17:02 maplebed: dns templates in svn and changes to authdns-update tested successfully
15:34 mark: Uncommented srv301 from mediawiki_installation node list; killed apache; reran puppet
15:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30221 Enable NewUserMessage on ladwiki'
15:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'whitespace'
15:18 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'whitespace'
15:10 logmsgbot: reedy synchronized wmf-config/wgConf.php 'Remove initialise noop call'
15:04 logmsgbot: reedy synchronized php-1.17/includes/SquidUpdate.php 'r93971 r93972'
14:24 Reedy: Running svn up to get r93971/r93972 on test
13:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30247 - Enable Botadmin user group on Hindi Wikipedia'
13:29 Ryan_Lane: added timo alias in exim for krinkle
13:28 Ryan_Lane: added timo alias in exim for krinkle
12:21 preilly: updated the MobileFrontend extension on production
12:13 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960:
11:39 logmsgbot: laner synchronized wmf-config/CommonSettings.php 'Setting HTTPS mode when using XFP'
10:06 mark: Fixed memcached check on spence
09:47 mark: srv301 didn't have the right (API) service IPs bound; manually ran dpkg-reconfigure wikimedia-lvs-realserver to fix this
04:10 maplebed: wikimedia-task-dns-auth package upgraded on ns0 to put zone files in revision control.
02:38 logmsgbot: aaron synchronized php/maintenance/addwiki.php 'deployed r93593'
02:31 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'renamed versionNumber variables to have wmf prefix'
02:25 logmsgbot: LocalisationUpdate completed at Fri Aug 5 02:27:52 UTC 2011
00:59 notpeter: knsq30 is seems to have some kind of hardware failure. stopping squid and squid frontend proc
00:48 AaronSchulz: /home/wikipedia/bin version with a working repo again

August 4

23:57 binasher: deploying mobile squid acls with nokia and netfront enabled for a brief test to see if todays mobile gateway changes will allow it to support the load
23:10 binasher: squid mobile redirect deployment completed
22:38 binasher: mobile redirect deploy complete to all pmtpa squids. mobile cluster looks good, starting amsterdam
22:17 binasher: starting an extra slow rollout of mobile redirect acls to frontend squids, minus nokia and netfront
21:43 binasher: varnish 3.0 is now on all RoR mobile gw servers - only caching static non-article content
21:33 binasher: mobile2 is back / varnish for static content / apache relax
19:52 binasher: pulling mobile2 from lvs for testing
17:17 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php 'r93902'
17:17 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php 'r93901'
17:12 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php 'r93901'
17:11 AaronSchulz: Fixed thumbnails on testwiki (bug 30210)
17:10 logmsgbot: aaron synchronized live-1.5/MWVersion.php
15:40 mark: Mounted nas1001-a:/vol/images on ms1002:/vol/images for testing, started rsync copy from ms1002:/export/upload to /vol/images
15:20 mark: Created 12-disk aggregate test0 on nas1001-a
15:01 RoanKattouw: Staging some HTTPS support / prot rel URL code on testwiki, see today's 1.17wmf1 merges. DO NOT DEPLOY SITEWIDE until tomorrow
13:19 RobH: updated wikibugs on mchenry about 10 minutes ago and forgot to admin log
13:17 mark: Created new 64 bit aggregate and root volume on nas1001-a, copied over root data, destroyed old root volume, moving new aggregate to first disks
13:00 RobH: disabled line numbers in vimrc.local on puppet because i hate line numbers (it already says the line number yer on in the corner)
12:21 mark: Started disk replace operation of 0a.10.0 by 0a.10.3 and 0a.10.1 by 0a.10.4 on nas1-b
12:12 mark: Started disk replace operation of 0b.00.0 by 0a.01.1, and 0b.00.1 by 0a.01.2 on nas1-a
12:08 mutante: ragweed - package updates, restart required
11:05 mutante: amssq46 - power cycle, clean squid, dist-upgrade/kernel, reboot
02:18 logmsgbot: LocalisationUpdate completed at Thu Aug 4 02:20:48 UTC 2011

August 3

21:22 binasher: previous log line should have read $ppsize + 50
21:21 binasher: dropping apache maxclients on mobile servers to ($ppsize - 50)
18:24 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r93845'
18:23 logmsgbot: aaron synchronized multiversion/MWScript.php 'deployed r93845'
16:19 logmsgbot: tfinc synchronized wmf-config/InitialiseSettings.php 'testing collectinos on hewiki'
16:13 mutante: installed Net::Stomp Perl module on spence via cpan shell which fixed RT #703
14:32 RobH: updating dns for magnesium and zinc eqiad misc servers to public IPs RT#1245
13:24 logmsgbot: tstarling synchronized php-1.17/extensions/CodeReview/ui/WordCloud.php
13:24 logmsgbot: tstarling synchronized php-1.17/extensions/CodeReview/CodeReview.php
12:21 mark: Rebooting both nas1001 controllers with corrected network config
10:02 logmsgbot: root synchronized wmf-config/mc.php 'swapped out all down servers returning 0 with ones returning 100, cleaned up spares and down list to be accurate'
09:44 RobH: swapping out down memcached servers iwth tested up ones
09:44 RobH: incorrect, turns out there are some down, they reply but do not return full weight
09:44 logmsgbot: robh synchronized wmf-config/mc.php 'replacing down servers'
09:40 RobH: nagios shows spence memcached error, yet manual testing shows all memcached are pooled fine.
09:01 Ryan_Lane: installing ssl1001-4
08:09 apergos: live hack to /usr/share/etherpad/etherpad/bin/run-local.sh on hooper to force maxthreads to 512 for etherpad. restarted etherpad, this may or may not help connection issues (maybe we'll just oom instead?)
07:40 domas: stupid bot and its quotes, http://p.defau.lt/?UUberPb2hRtPxOzG7Q6OwQ is the url
07:39 logmsgbot: midom synchronized php/includes/db/DatabaseMysql.php 'livehacking out sql_mode and set charset code - http://p.defau.lt/?UUberPb2hRtPxOzG7Q6OwQ'
02:20 logmsgbot: LocalisationUpdate completed at Wed Aug 3 02:22:48 UTC 2011

August 2

23:25 binasher: done with that epic test
23:04 binasher: redeploying revised mobile acls with the redirector only matching /wiki/ urls for brief (15min) test
22:19 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'added $wgExtendedVersionNumber (Het Deploy), should have no effect now'
19:19 binasher: reverted mobile redirect acls
18:58 binasher: deploying mobile redirect acls/helper to all frontend text squids (--slow)
18:16 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30186 - Install WikiLove on the Hebrew Wikipedia'
18:03 notpeter: restarting torrus again. cuz it's my favorite thing to do...
17:16 Reedy: As part of fatal coming from OAIRepository decreased by 10% from 50 to 45. No fatals in logs for nearly a week though
17:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'bug 29304 Set to 45'
17:04 logmsgbot: reedy synchronized php-1.17/extensions/OAI/OAIRepo_body.php 'r93770'
17:03 logmsgbot: reedy synchronized php-1.17/extensions/OAI/OAIRepo.php 'r93770'
17:02 logmsgbot: reedy synchronized php-1.17/includes/parser/Preprocessor_DOM.php 'r93763'
15:28 logmsgbot: catrope synchronized php/includes/media/SVG.php 'r93750'
14:23 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Changing main namespace for ladwiki'
12:49 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'reenabling disk cache'
12:42 domas: synced http://p.defau.lt/?Jze2xAwLV4Fvh8JC2B_KqQ to fix memcached side of pcache
12:40 logmsgbot: midom synchronized php/includes/objectcache/MemcachedClient.php
12:07 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'disabling disk cache'
12:00 Ryan_Lane: changing management dns entry for phosphorus to ssl1004
11:57 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'removed l10n debug log group'
11:56 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'reverted live hack'
11:52 RobH: updated wikibugs on mchenry per rt#1179, restarted the service
11:44 logmsgbot: tstarling synchronized multiversion/MWVersion.php 'r93727 fix l10n cache'
11:38 RobH: restarted pdns on linne
11:31 RobH: updated dns for ssl and silicon servers
11:31 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'log l10n file list changes'
11:30 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'Removed unmatched wfProfileOut()'
11:26 RobH: phosphorus changed to sl1004 in dns and racktables
11:20 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'enough data'
11:16 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'more logging'
11:15 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'more logging'
11:11 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'logging hack for recache tracking'
11:09 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'logging hack for recache tracking'
11:04 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'Removed unmatched wfProfileOut()'
11:04 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Adding sitename for ladwiki, per bug 30181'
10:16 mark: Restarting all PyBal instances in a controlled manner
10:13 mark: Split LVS class high-traffic into high-traffic1 (text, bits) and high-traffic2 (upload), and made a separate class for https with BGP disabled
09:29 Ryan_Lane: powercycling srv281
08:14 mark: Increased TTL of bits.pmtpa.wikimedia.org from 60 to 1H
08:09 mark: Changed IP address of upload.pmtpa.wikimedia.org from .3 to 208.80.152.211
08:03 mark: Increased etherpad memory from 1G to 2G in /usr/local/etherpad/bin/run.sh
07:39 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'in parser cache multiwrite configuration, putting memcached first instead of mysql'
05:51 mark: Restarted nginx on yvon and gurvin; apparently nginx needs to be restarted after ip addresses changes, as it binds each individual address instead of IN_ADDR_ANY
04:23 notpeter: stopping puppet on spence temporarily
02:21 logmsgbot: LocalisationUpdate completed at Tue Aug 2 02:23:56 UTC 2011
00:26 AaronSchulz: deployed r93695 to fix WURFL infinite loop
00:24 logmsgbot: aaron synchronized php/extensions/MobileFrontend/library/WURFL/Storage/Mwmemcache.php
00:23 logmsgbot: aaron synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php

August 1

23:41 ^demon: deleted another one of Roan's silly files from /h/w/c/p/ that contain less --help
23:38 notpeter: upgrading squid and squid-frontend on sq31. sq32 is down and still needs an upgrade.
23:25 logmsgbot: aaron synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'pre-OOM logging'
23:19 notpeter: upgrading squid and squid-frontend on sq33 and sq34
23:15 notpeter: restarting broken torrus again again.
22:58 logmsgbot: aaron synchronized live-1.5/robots.php 'commented out redundant code'
22:57 notpeter: upgrading squid and squid-frontend on sq35 and sq36
22:49 logmsgbot: aaron synchronized multiversion/MWVersion.php 'fix for robots.php hits on bogus wikis'
22:47 logmsgbot: aaron synchronized multiversion/MWVersion.php
22:26 notpeter: upgrading squid and squid-frontend on sq37 and sq38
22:09 notpeter: upgrading squid and squid-frontend on sq39 and sq40
21:50 notpeter: restarting broken torrus again
21:18 Reedy: srv187 although up and alive, isn't reporting any metrics to ganglia, and uptime is completely wrong
21:15 notpeter: upgrading squid and squid-frontend on sq59 and sq60
21:12 logmsgbot: aaron synchronized multiversion/MWVersion.php
21:10 logmsgbot: aaron synchronized wmf-config 'multiversion/MWVersion.php missing.php fix'
20:46 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'moved two lines of code down a bit to allow usage of missing.php'
20:33 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'removed transition hack (cdb rebuilt)'
20:29 AaronSchulz: wikiversions.cdb rebuilt
20:21 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
20:17 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'hardcode to 1.17 for safe transition'
20:09 logmsgbot: aaron synchronized live-1.5/MWVersion.php
19:59 notpeter: upgrading squid and squid-frontend on sq61 and sq62
19:37 notpeter: upgrading squid and squid-frontend on sq64
16:15 mark: Deployed puppet generated pybal.conf on amslvs1,2,3,4
15:47 mutante: amssq55: power cycle, fsck forces add. reboot, clean squid cache, dist-upgrade,..
15:43 mark: Deployed PyBal generated pybal.conf on amslvs3
15:25 mutante: amssq53: cleaned squid cache, now dist-upgrade and another reboot
15:07 mark: Temporarily filtering traffic from 10.0.0.0/16 to puppet on sockpuppet
14:56 mutante: power cycle amssq53
12:36 mutante: restarted ntpd on ms3 to fix a Nagios CRIT
12:14 mark: Changed IP address of bits.pmtpa.wikimedia.org from .118 to 208.80.152.210
11:02 mark: Moved LVS services configuration out of site.pp into lvs.pp, in puppet
10:03 mark: Started db20

July 31

12:20 Reedy: srv207 seems to be dead. Nagios suggests RAID issues
12:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30150 Fix timezone at xmfwiki'
02:21 logmsgbot: LocalisationUpdate completed at Sun Jul 31 02:23:17 UTC 2011

July 30

23:54 logmsgbot: reedy synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'htmlspecialchars url'
23:48 logmsgbot: reedy synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'htmlspecialchars url'
23:39 logmsgbot: reedy synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'Urlencode url'
23:32 logmsgbot: reedy synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'Urlencode url'
23:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump wgWikimediaMobileVersion to .3'
22:59 Reedy: WikimediaMobile javascript giving 404 error whilst log out. Going to bump $wgWikimediaMobileVersion
22:15 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30122 - Add "WP" namespace alias to Test Wikipedia'
22:15 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30122'
22:08 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30138 - Please enable AbuseFilter for ee.wikipedia'
22:06 logmsgbot: jeluf synchronized wmf-config/abusefilter.php '30138 - Please enable AbuseFilter for ee.wikipedia'
20:35 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30121 - Test Wikipedia request for file mover user group'
20:27 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '25181 - Enable transwiki import from zhwiki to other zh projects'
20:20 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30136 - Enable recent changes patrolling on Portuguese Wikibooks'
19:09 binasher: db18 mysqld is back up and catching up on replication. may have a bad dimm (lots of ecc errors), leaving out of rotation for now
18:48 binasher: pulling db18 from rotation, appears to be having hw troubles
18:48 logmsgbot: root synchronized wmf-config/db.php 'pulling db18'
02:38 logmsgbot: LocalisationUpdate completed at Sat Jul 30 02:40:45 UTC 2011
00:09 notpeter: upgrading squid and squid-frontend on sq65 and sq66

July 29

23:34 notpeter: upgrading squid and squid-frontend on sq71 and sq72
23:08 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'remove logging of memory exhausted error'
23:02 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'possibly logging memory exhausted error bump limit to 15000'
23:00 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'remove logging of memory exhausted error'
22:57 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'possibly logging memory exhausted error bump limit to 10000'
22:56 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'possibly logging memory exhausted error bump limit to 1000'
22:55 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'possibly logging memory exhausted error'
22:53 notpeter: upgrading squid and squid-frontend on sq73 and sq74
22:17 notpeter: upgrading squid and squid-frontend on sq75 and sq76
21:57 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'move call for props'
21:11 notpeter: upgrading squid and squid-frontend on sq78 and sq77
21:10 RobH: fixed symlink for wikimedia.org/w/ in docroot
19:28 notpeter: restarting torrus. again. poor torrus, it just wants to die.
19:26 RobH: updated dsh files and restarted job runners
19:15 notpeter: upgrading squid and squid-frontend on knsq24
18:43 notpeter: upgrading squid and squid-frontend on knsq25 and knsq26
18:01 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'add mb_convert_encoding call'
17:01 mutante: amssq40 - dist-upgrade->new kernel->reboot
16:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert last commit'
16:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29911'
16:22 mutante: power cycle amssq40
15:45 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 29911 - customise FlaggedRevs with whole reviewer system on hi.wikipedia.org'
15:28 mark: Setup cross link between cr1-sdtpa:ae0.400 and cr2-pmtpa:ae0.400, using VLAN 400 across csw1-sdtpa/csw5-pmtpa
14:48 Jeff_Green: purging master logs on db9 up to the time of the last full db dump
14:30 mark: Removed old cross link between csw5-pmtpa:ve1 and csw1-sdtpa:ve1, freeing up VLAN 400
14:30 mark: Setup temporary cross link between cr1-sdtpa:ae0.404 and csw1-sdtpa:ve2
14:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30098 - Install extension WikiLove on Arabic Wikipedia'
13:53 mark: Deployed full (modified) replica of configuration of cr1-sdtpa on cr2-pmtpa
13:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 29956 Customise hiwiki add/remove groups'
13:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30003 - Please set wgUploadNavigationUrl + wgUploadMissingFileUrl for nl.wikisource.org nl.wikiquote.org nl.wiktionary.org'
13:34 mark: Upgraded cr2-pmtpa to JUNOS 10.4R6.5
13:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30045 define transwiki import source for kmwiki from be, en and simple'
13:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30065 replace the Hebrew Wikipedia logo'
13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30085 Name update for cy.wikibooks.org'
02:16 logmsgbot: LocalisationUpdate completed at Fri Jul 29 02:18:38 UTC 2011
00:10 AaronSchulz: Updated wikimedia-periodic-update (for FR) to use MW script wrapper
00:05 AaronSchulz: Updated mw-central-notice, mw-tor-list to use MW script wrapper

July 28

23:36 AaronSchulz: Updated update-special-pages, update-special-pages-small, jobs-loop.sh to use MWScript.php
23:06 binasher: power cycled srv 173
22:56 logmsgbot: reedy synchronized wmf-config/db.php 'srv173 is dead to the world commenting out'
22:52 Reedy: srv173 seems deaded
22:40 logmsgbot: hashar: made proofreadpage.php configuration publicly available through http://noc.wikimedia.org/conf/ . Usual notice is at the top of the file. Requested by phe (frwiki).
22:39 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fixing caching'
22:37 logmsgbot: aaron synchronized wmf-config/ExtensionMessages-1.17.php 'removed ArticleAssessmentPilot'
22:31 logmsgbot: aaron synchronized wmf-config/extension-list 'removed ArticleAssessmentPilot'
22:28 Reedy: srv178 flapping due to full OS disk
22:14 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'enable Mobile Frontend'
22:07 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93132:
21:28 Reedy: Full run through of LocalisationUpdate, to take into account r93425, Successful!
21:22 logmsgbot: LocalisationUpdate completed at Thu Jul 28 21:24:14 UTC 2011
20:45 binasher: enabling mobile redirect acls on sq78 (5% of all text requests)
20:27 logmsgbot: LocalisationUpdate failed
19:56 Reedy: New localisation files added for Ks-arab, Jam, Brh, Khw, Ks-deva, Anp, Dtp. Bh renamed Bho. Ks was split
19:55 logmsgbot: reedy synchronized php-1.17/languages/Names.php 'r93423'
19:54 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesBh.php 'r93420'
19:54 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesBho.php 'r93420'
19:51 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesKs.php 'r93420'
19:51 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesDtp.php 'r93420'
19:51 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesAnp.php 'r93420'
19:50 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesKs_deva.php 'r93420'
19:50 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesKhw.php 'r93420'
19:50 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesBrh.php 'r93420'
19:49 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesJam.php 'r93420'
19:49 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesKs_arab.php 'r93420'
19:41 logmsgbot: hashar: ci.tesla: deleted cc/projects/mw/extensions/FCKeditor directory
19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Kill lqt alpha'
19:23 logmsgbot: reedy synchronized wmf-config/liquidthreads.php 'Kill lqt alpha'
18:50 logmsgbot: LocalisationUpdate completed at Thu Jul 28 18:52:41 UTC 2011
18:46 notpeter: pushing out new dns zone file (only changing cname survey to point to argon.w.o)
18:40 Reedy: LocalisationUpdate seems to be working correctly now
18:35 Reedy: Fixed localisationupdate for FR, added symlink in /home/wikipedia/l10n/trunk/extensions/FlaggedRevs for the language folder to presentation/language
18:32 notpeter: removing survey.w.o from sites enabled on singer for limesurvey migration, and doing a graceful on singer
18:22 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove old ArticleAssessment stuff'
18:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove old ArticleAssessment stuff'
18:14 logmsgbot: reedy synchronized php-1.17/languages/Names.php 'r93407'
18:14 logmsgbot: reedy synchronized php-1.17/includes/DefaultSettings.php 'r93407'
17:44 logmsgbot: LocalisationUpdate failed
17:23 Reedy: Manual run of /home/wikipedia/bin/sync-l10nupdate after updated permissions
16:56 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'touched file'
16:52 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 're-pushed Het Deploy changes with $dbSuffix fix'
16:51 RobH: nagios wasnt running, didnt investigate, just restarted service
16:44 AaronSchulz: Updated set-group-write2 to handle wmf-config being move up a dir
12:45 Reedy: Also, LocalisationUpdate doesn't seem to be logging to /var/log/l10nupdate as it is supposed to
12:43 Reedy: Seems LocalisationUpdate may be broken due to permission issues on l10n files on apaches
12:39 mark: Replaced SSDs in amssq31 and dist-upgrade
12:27 mark: Brought up BGP session to AS 38930
12:21 Reedy: Running l10nupdate manually on fenari
12:18 Reedy: Reports of LocalisationUpdate may have stopped working again
12:12 mark: Shutdown session to AS 16265
01:18 andrewS: dongle found. reconnected serial to pdu d3-sdtpa.

July 27

23:47 preilly: pushing bad redirect logging to mobile cluster
23:37 maplebed: started reslave from db42 to db1047; poor performance expected for the duration of the sync.
23:33 notpeter: restarting torrus again
23:19 preilly: pushing badredirect log to mobile cluster
22:46 notpeter: upgrading squid and squid-frontend on knsq27 and knsq28
22:13 notpeter: torrus stuck. assuming in deadlock. following instructions for restart
22:01 AaronSchulz: Re-installed moodbar & edittracking. Cleaned up wmf-config so that we are on HEAD (but with the reverted stuff still gone)
22:00 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
20:19 notpeter: upgrading squid and squid-frontend on knsq29 and knsq30
19:54 binasher: pulled sq78 frontend from lvs
19:40 Reedy: Reports of users using non mobile browsers being redirected to mobile sites (enwiki and dewiki)
19:04 mark: Temporarily disabled facilities.pp file in puppet, until there is time to debug
18:34 binasher: pulled sq78
18:32 binasher: sq78 frontend is in service with mobile redirection
17:08 binasher: pulling sq78 from lvs prior to mobile testing
16:56 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r93255'
13:40 Reedy: Do not push any configuration change from wmf-config without speaking to Reedy/Ariel first
13:33 Reedy: Seems some configuration files between r2087 and r2088 weren't pushed to site, pushing them broke most projects. Aaron to investigate when he's in the office
13:20 logmsgbot: reedy synchronized wmf-config/extension-list 'r2087'
13:19 logmsgbot: reedy synchronized wmf-config/liquidthreads.php 'r2087'
13:19 logmsgbot: reedy synchronized wmf-config/ExtensionMessages.php 'r2087'
13:18 Reedy: Site should be somewhat back to normal, reverted to r2087 of config from private SVN
13:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'r2087'
13:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'r2087'
13:15 logmsgbot: reedy synchronized wmf-config/CommonSettings-old.php
13:14 logmsgbot: reedy synchronized wmf-config/CommonSettings-new.php
13:12 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'r2088'
13:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'r2088'
12:36 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Reverting last push'
12:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Reverting last push'
12:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Reverting last push'
12:31 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Reverting last push'
12:27 andrewS: miguel gave OK to fill up d1 in sdtpa as the heat flow issues have been temporarily resolved (also said construction on new AC units was to begin Aug 1st but has not heard anything yet).
12:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
12:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
11:55 mutante: sq63 - restored software RAID & dist-upgrade & reboot
11:15 Jeff_Green: adjusted firewall rules on payments servers per RT #1223
10:28 apergos: chowned /home/w/logs/*.log to udp (was root, /usr/local/bin/demux.py on nfs1 was failing to write to these files since July 21). This is probably the wrong solution but I don't know what the right setup is; can someone who does, make it right?
08:55 andrewS: 74th r410 dell server has arrived in tampa. uploaded packing slip to rt#611.
06:13 Jeff_Green: repooling payments3 & payments4, they're dist-updated and tested
05:05 logmsgbot: aaron synchronized php/maintenance/jobs-loop.sh 'added --wiki to another spot'
02:49 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'fixed texvc path'
01:31 notpeter: restarting jorb runners on srv151-180. also wishing there was a script for htis
01:05 logmsgbot: aaron synchronized php/maintenance/jobs-loop.sh 'added wiki=aawiki param to nextJobDB.php'
00:54 ^demon: job runners stuck in futex deadlock again :(

July 26

23:39 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
23:34 Jeff_Green: dist-upgrade and reboot payments3, payments4
23:26 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
23:23 notpeter: upgrading squid and squid-frontend on amssq31 and 32
23:23 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
23:15 AaronSchulz: CommonSettings.php is no longer just a wrapper (it now has the CommonSettings-new.php contents)
23:07 Jeff_Green: dist-upgrade and reboot payments2
23:00 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
22:47 notpeter: upgrading squid and squid-frontend on amssq33 and 34
22:42 notpeter: (retroactive) upgraded squid and squid-frontend on amssq38 and 39
22:18 Jeff_Green: dist-upgrading and rebooting payments1
21:51 logmsgbot: catrope synchronized wmf-config/CommonSettings-old.php 'Correct MoodBar policy URL per Howie'
21:50 logmsgbot: catrope synchronized wmf-config/CommonSettings-new.php 'Correct MoodBar policy URL per Howie'
21:38 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'moved setting of $secure back up'
21:21 notpeter: upgrading amssq40 and 41 squid and squid-frontend
20:45 Jeff_Green: purging master logs on db9
20:20 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/special/SpecialCentralNotice.php 'r93242'
20:19 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r93240'
19:42 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r93236'
19:26 notpeter: upgrading squid and squid frontend on amssq35 and 36
19:22 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php 'removed warning in b/c script code'
19:16 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
19:11 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php 'fix b/c code for scripts ran without wrapper'
18:48 RobH: cp1043 and cp1044 installed per rt#1157
18:46 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php
18:44 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php
18:40 AaronSchulz: CommonSettings.php is now at CommonSettings-new.php for now, the former is just a wrapper (making it easy to go back to the old one)
18:38 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
18:34 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'switched all wikis to new commonsettings'
18:33 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php 'make sure this is on all wikis'
18:33 logmsgbot: aaron synchronized wmf-config/CommonSettings-old.php 'make sure this is on all wikis'
18:28 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'r93229'
16:06 mutante: power cycle owa1
15:55 mutante: power cycle srv281
15:53 mutante: power cycle srv266
14:55 mark: Migrated LVS services 'text' and 'upload' to lvs5/lvs6
14:51 mark: Migrated all HTTPS LVS services to lvs5/6
14:45 mark: Migrated bits-lb and bitssvc to lvs5/6
14:16 mutante: power cycle sq84 | clean squid cache & dist-upgrade on sq80,82,84
14:10 mutante: power cycle sq82
14:07 mutante: power cycle sq80
13:50 mark: Deploying lvs5 and lvs6 as LVS balancers (class high-traffic)
13:44 mutante: OS installed on lvs5 and lvs6
13:26 mutante: reinstall lvs6
13:23 mutante: power cycle lvs6
13:23 mark: Fixed switchport settings for lvs5
12:10 mutante: power cycle and dist-upgrade sq79
11:58 mutante: power cycle and dist-upgrade sq63
11:52 mutante: power cycle and dist-upgrade knsq5
11:31 mutante: power cycle and dist-upgrade knsq25
11:18 mutante: power cycle and dist-upgrade knsq23
11:13 mutante: dist-upgrade and reboot knsq20
10:56 mutante: power cycle knsq20
10:50 mutante: power cycle knsq13
10:26 mutante: rebooting amssq54,amssq57,amssq60
10:14 mutante: dist-upgrade on amssq60
10:14 mutante: dist-upgrade on amssq57
10:13 mutante: dist-upgrade on amssq54
09:45 mutante: running dist-upgrade on amssq50
09:40 mutante: power cycled amssq60
09:37 mutante: power cycled amssq57
09:23 mutante: power cycled amssq54
09:22 mutante: power cycled amssq50
00:44 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
00:43 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set privacyUrl properly for MoodBar'
00:39 AaronSchulz: Note - addwiki.php won't let you run it until Het deploy stage 2C is done (planned for tomorrow morning)
00:34 logmsgbot: aaron synchronized php/maintenance/addwiki.php 'deployed r93152'
00:11 notpeter: upgrading squid and squid-frontend on amssq42

July 25

23:52 AaronSchulz: changed scap & CommonSettings.php to use ExtensionMessages-1.17.php (instead of ExtensionMessages.php)
23:50 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
23:45 notpeter: (re)deploying squid.confs to amssq46
23:43 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
23:42 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'switched all wikis to use new Het wrapper'
23:41 logmsgbot: aaron synchronized extract2.php
23:23 notpeter: upgrading squid and squid-frontend on amssq44
23:09 Andrew: MoodBar deployment to enwiki complete.
23:03 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 93132:
23:02 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'avoid /home for secure testwiki'
23:00 Andrew: Deploying MoodBar to enwiki
22:43 logmsgbot: aaron synchronized extract2.php 'done with debug code'
22:40 Andrew: Adding MoodBar tables to all databases.
22:34 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 93132:
22:33 Andrew: Deploying EditPageTracking to enwiki
22:33 logmsgbot: aaron synchronized extract2.php
22:26 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 93132:
22:25 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 93132:
22:24 Andrew: Running scap to push message updates
22:21 Andrew: EditPageTracking tables are all added.
22:12 Andrew: Creating EditPageTracking tables on all wikis
22:08 Andrew: Deploying EditPageTracking to testwiki
22:08 Andrew: EditPageTracking and MoodBar user registration cutoff is 20110725221004.
22:04 Andrew: added EditPageTracking and MoodBar to extension-list
22:02 Andrew: EditPageTracking rollout to enwiki is going ahead.
22:01 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132:
22:00 Andrew: updated prototype MoodBar / EditPageTracking to trunk state
21:57 logmsgbot: aaron synchronized extract2.php
21:33 AaronSchulz: www.wikipedia.org portal site didn't like the change
21:32 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'revert'
21:29 RobH: cp1001-cp1044 have had sas controllers installed and cabled. ready for os installation
21:21 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'switch all wikis to new Het wrapper'
20:57 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
20:56 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 100%'
20:55 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.27%'
20:40 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
20:39 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 95%'
20:38 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.28%'
20:37 logmsgbot: aaron synchronized wmf-config/liquidthreads.php 'fixed reference to undefined var'
20:32 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'fixed testwiki notice'
20:29 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
20:26 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 90%'
20:26 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.3%'
20:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29940 hiwiki namespaces'
20:02 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'fixed use of "tru" and bad NS_PORTAL constants'
20:00 logmsgbot: aaron synchronized php-1.17/maintenance/Maintenance.php 'deployed r93099'
19:45 notpeter: stopping, purging, and upgrading squid and squid frontened on amssq46 (still has faulty version on it)
19:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29956 userrights/groups for hiwiki'
18:44 binasher: installing mobile redirect helper to all caching proxies but leaving out of the squid config
18:43 binasher: added percona xtrabackup to wikimedia-lucid and installing on all lucid based db's via puppet
18:17 notpeter: updating squid and squid frontend on amssq37
17:48 Reedy: Trunk version of CodeReview has been deployed to mediawiki.org. Yaay
17:46 logmsgbot: demon synchronizing Wikimedia installation... Revision: 92888: Deploying updates to CodeReview
17:17 RobH: cp1033-cp1044 sas cards installed
17:12 mutante: Disabled account rcole
16:52 mark: Added user account and full root access to dzahn
16:08 mark: Removed old puppet resources for lvs3.pmtpa.wmnet (replaced by lvs3.wikimedia.org) on db9
01:17 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92888:

July 24

18:12 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '29641 - Create Appendix'

July 23

09:50 logmsgbot: aaron synchronized php-1.17/includes/specials/SpecialUserlogin.php 'deployed r92938'
09:36 logmsgbot: aaron synchronized php-1.17/extensions/CategoryTree/CategoryTree.php 'live hack to efCategoryTreeAjaxWrapper to shut up $option notices'
09:28 logmsgbot: aaron synchronized php-1.17/includes/specials/SpecialResetpass.php 'deployed r92935'
09:27 logmsgbot: aaron synchronized php-1.17/includes/specials/SpecialUserlogin.php 'deployed r92935'
02:57 logmsgbot: catrope synchronized php/extensions/CategoryTree/CategoryTree.php 'remove debugging code; no point as long as the log collector is broken'
02:56 logmsgbot: catrope synchronized php/extensions/CategoryTree/CategoryTree.php 'debugging a PHP warning'
02:54 logmsgbot: catrope synchronized php/extensions/CategoryTree/CategoryTree.php 'debugging a PHP warning'

July 22

21:10 logmsgbot: catrope synchronized php/resources/jquery/jquery.localize.js 'r92890'
19:24 binasher: enabled local file debug logging on amssq45's backend squid
19:13 Ryan_Lane: adding ci2.tesla to DNS
19:03 Jeff_Green: new db10 dump restored to db1008 and replication is started
18:08 notpeter: upgrading squid-frontend/squid on amssq45
18:06 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turning account creation project off for mw.org. Wasnt fully disabled'
18:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29940 hiwiki ns aliases'
16:59 Reedy: Running namespaceDupes.php on ukwiki
16:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29210'
16:56 RobH: rebooting srv187, unresponsive to serial console
16:55 Reedy: srv187 seems somewhat dead, SSH takes over 6 minutes to die with "srv187: ssh_exchange_identification: read: Connection reset by peer"
16:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29210'
16:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29210'
16:40 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'replaced undefined CT_MODE_PAGES with 10, the extension constant value'
14:57 Reedy: sync-common-file seems broken. Not returning after executing a command...
10:21 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561:
10:09 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'setupFundraiserPortal must declare $urlprotocol as global'
10:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'error suppression for REQUEST_METHOD check'
04:49 Jeff_Green: restarted replication on db10
04:47 Jeff_Green: repooled pyayments4
02:36 AaronSchulz: updated scap to new wmf-config dir
02:34 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561:
02:25 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561:
00:33 awjr: updating DonationInterface on payments cluster to r92830
00:30 logmsgbot: aaron synchronized wikiversions.cdb
00:26 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php

July 21

23:54 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php
23:28 Jeff_Green: stopping slave on db10 for a mysqldump
23:21 Jeff_Green: depooling payments4 for awjr staging code changes
23:04 aaron: new MWVersion/MWMultiVersion code running on testwiki
23:04 logmsgbot: aaron synchronized live-1.5/MWVersion.php
23:03 logmsgbot: aaron synchronized live-1.5/MWVersion.php
23:00 logmsgbot: aaron synchronized wikiversions.cdb
22:53 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561:
22:46 RobH: psw1-eqiad racked with serial connection to scs-a8-eqiad. RT#1207
22:25 awjr: updating CiviCRM install on Grosley to r368
22:04 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561:
21:31 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 92561:
21:30 awjr: running scap to deploy CentralNotice updates to r92794
21:18 RobH: psw1-eqiad added into racktables, racked, powered, and outlets configured. need to still run serial and get network port assignemtns
20:55 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
20:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 85%'
20:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.318%'
20:46 Ryan_Lane: make that 5, since that's the lowest allowed
20:45 Ryan_Lane: setting autosupport retry to 0 on eqiad netapp
20:40 Ryan_Lane: disabled autosupport on eqiad netapp
20:28 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
20:28 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 77%'
20:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.35%. Also includes lots of whitespace changes and an unused global removal by Aaron'
19:52 RobH: ms1004 rt1164 passes full dell hardware testing suite
19:08 notpeter: upgrading squid and squid frontend on amssq46
18:56 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '29971 - Set wgUploadMissingFileUrl for nlwiki'
17:26 Ryan_Lane: powering off srv173, since it is dead, and is messing up the stats
16:33 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'unused var in wfNoDeleteMainPage'
16:18 mark: Deployed lvs3 as backup LVS balancer for 'low-traffic' class LVS services, backing up lvs4
15:22 RobH: lvs3 reinstalled, puppet not run
15:07 RobH: updated dns again for more changes pertaining to lvs
15:02 RobH: updated dns to add lvs3.wikimedia.org to forward dns, was already in reverse file.
14:56 RobH: lvs6 reinstalled, puppet not synced and not signed as it may need further work
14:50 mark: Added new service IPs for bits and upload to varnish and squid classes in puppet
14:43 mark: Allocated out of subnet IPs for bits and upload.pmtpa
14:35 RobH: lvs3, lvs5, & lvs6 being (re)installed and setup, please disregard any warnings.
14:34 mark: Redeployed RunCommand monitor on lvs4, now fixed
14:33 RobH: lvs3 being reinstalled

July 20

22:51 Ryan_Lane: started memcached on srv260
21:53 Ryan_Lane: added monitoring for esams and pmtpa -lb addresses
21:10 Jeff_Green: db10 mysqldump is done, replication restarted
21:05 Ryan_Lane: adding lvs IPs for all esams -lb addresses to amslvs servers
20:54 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:52 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 70%'
20:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.386%'
20:47 Ryan_Lane: added all new -lb services to esams pybal
20:45 Ryan_Lane: made esams pybal files consistent. restarting pybal on each amslvs system, in order
20:39 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:37 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 63%'
20:37 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.428571%'
20:25 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 57%'
20:22 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking rate to 0.47%'
19:02 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/SpecialContact.php 'r92653'
19:02 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/README 'r92653'
19:02 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/install.settings 'r92653'
19:01 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/ContactPage.i18n.php 'r92653'
19:01 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/ContactPage.alias.php 'r92653'
18:59 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Variable-izing enabling of ContactPageFundraiser extension, enabling on testwiki and foundation wiki'
18:56 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Variable-izing enabling of ContactPageFundraiser extension'
18:54 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php
18:42 AaronSchulz: Changed sync-file to be relative to common/ instead of php-1.17/
18:41 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561:
18:41 RobH: rebooting the offline sq63 to poll its SAS controller info
18:25 Ryan_Lane: restarting pybal on lvs1
18:12 Jeff_Green: stopping replication on db10 and redoing dump with mysqldump instead of mydumper...
18:03 logmsgbot: awjrichards synchronized php-1.17/extensions/LandingCheck/SpecialLandingCheck.php 'r92647'
18:03 logmsgbot: awjrichards synchronized php-1.17/extensions/LandingCheck/LandingCheck.i18n.php 'r92647'
17:40 RobH: lvs3 racked, but powered off. mgmt confirmed working, network port confirmed up
16:09 mark: Disabling RunCommand monitor on lvs4; not working right
15:50 mark: Restarted PyBal on lvs4
15:27 RobH: powercycling srv283
15:27 RobH: srv283 is hung up again, and now wont take my commands in serial console...
15:22 RobH: srv283 loads are insane, its very slugging, and only serial console works. kicking the apache process over to free up memory and resume normal function
14:39 RobH: srv239 unresponsive, rebooting
14:02 mark: Setup nas1 with clustering failover
08:44 mark: Fixed misc::mediawiki-logger manifest
00:04 Ryan_Lane: test bug 1

July 19

23:59 binasher: also set long_query_time to 0.8 on the enwiki read slaves
23:09 Ryan_Lane: added new ganglia clusters for ssl cluster in pmtpa and esams
22:50 binasher: reduced long_query_time on db36 to 800ms
22:46 binasher: setting up mk-query-digest slow query processing on db36
22:43 binasher: installing maatkit on core dbs via puppet
22:13 Jeff_Green: installing mysql 5.0.51a on db1025 to for mydumper/myloader mysqld version upgrade testing
21:40 Ryan_Lane: make that changing the CNAME to wikipedia-lb
21:40 Ryan_Lane: changing DNS entry for test.wikipedia.org to wikipedia-lb.wikimedia.org
21:04 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
21:03 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Kill underscores in blacklist category name'
20:58 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:58 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set up as an AFT blacklist category on enwiki'
20:45 mark: Flipped all backup static routes for LVS service ips from lvs3 to lvs4
20:42 mark: lvs3 is now unused.
20:40 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:40 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 50%'
20:40 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.54%'
20:38 mark: Migrated apaches LVS service from lvs3 to lvs4
20:22 mark: Migrated LVS service 'api' from lvs3 to lvs4
20:19 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:19 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 43%'
20:19 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.63%'
20:07 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:07 mark: Migrated LVS service 'rendering' from lvs3 to lvs4
20:05 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 37%'
20:04 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.73%'
19:56 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.css 'r92561'
19:56 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r92561'
19:55 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r92561'
19:55 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r92561'
18:58 Ryan_Lane: installing nginx with udplogging module on ssl proxies
18:51 logmsgbot: aaron synchronized php-1.17/maintenance/Maintenance.php 'deploy r92550'
18:27 Ryan_Lane: pushing change to squids to strip X-Forwarded-Proto, if not set by the ssl proxies
18:08 Ryan_Lane: repooling sq69 with geoiplookup xff changes, for load testing
18:02 mark: Fixed LVS on lvs1/lvs2
17:44 mark: Disabled RPF check on ae2.1002 on cr1/cr2-eqiad
17:07 Jeff_Green: rebooting db1025 after dist-upgrade
17:02 Ryan_Lane: depooling sq69 for geoiplookup tests
16:01 mark: Enabled BGP on lvs4 and migrated LVS service search_pool3 to lvs4
15:35 RobH: rebooting sq63 to check its bios, its arleady down due to other issues and depooled in pybal
15:31 mark: Added BGP configuration for PyBal to cr1-sdtpa
14:14 Jeff_Green: commented out puppet-related cron jobs on db1008 as well :-P
13:58 Jeff_Green: stopping puppet daemon on db1008 to facilitate local my.cnf testing
08:55 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324:
07:14 AaronSchulz: Fixed require in PrivateSettings.php
07:13 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php
07:10 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php
01:14 Ryan_Lane1: repooling sq69
00:34 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'bug 29922 Remove abusefilter-private right so it cannot be accidentally granted'
00:34 Jeff_Green: db10 slaving restarted
00:30 Ryan_Lane: depooling sq69 to test varnish changes

July 18

23:57 AaronSchulz: changed 1.17/StartProfiler.php to symlink to the one in wmf-config
23:57 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324:
23:41 logmsgbot: catrope synchronized docroot/bits/test.txt
23:40 AaronSchulz: copied StartProfiler.php to wmf-config
23:40 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324:
22:32 rainman-sr: build search indexes for the recently created wikis
21:34 AaronSchulz: moved PrivateSettings.php to wmf-config and symlinked 1.17 copy of file
21:31 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324:
21:23 JeLuF: removed some privacy related mails from wikide-l archive
20:58 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:56 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 30%'
20:55 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking rate to 0.9%'
20:44 notpeter: running apache-graceful-all
20:38 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'toch'
20:38 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 27%'
20:37 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 1%'
20:37 notpeter: syncing apaches again
20:26 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:25 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 23%'
20:24 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 1.17% and bump cache epoch on HTTPS experiment wikis'
20:20 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'setting wgServer for officewiki to be protocol relative'
20:11 Ryan_Lane: renaming chlorine, germanium, and sulfer to ssl1001, ssl1002, and ssl1003, respectively
20:00 Jeff_Green: stopping replication on db10 and dumping db to db1008
19:40 Jeff_Green: recloned payments3 from payments4, restored db pool config to the way it was before
19:12 Jeff_Green: pointing payments[1-4] db reads to payments2 specifically (rather than localhost) to repair replication on payments3
18:57 notpeter: syncing apaches again
18:41 notpeter: syncing apaches
18:02 notpeter: pushing new dns confs (just a cname, ryan looked over it)
17:55 binasher: varnish3-3.0 packages with udp multicast logging support built and added to the wmf repo
17:31 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324:
17:29 logmsgbot: aaron synchronized php-1.17/maintenance/doMaintenance.php 'deployed r92452'
17:18 logmsgbot: aaron synchronized php-1.17/LocalSettings.php 'use new wmf-conf dir'
17:12 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324:
17:12 aaron copied wmf-config up one directory (to common/)
16:08 RobH: ms1004 started hardware testing. system is using the build in diag tests for dell, but do NOT work via serial redirection. Thus ms1004 appears offline, but is running tests. DO NOT REBOOT IT
14:51 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29942 Enable UploadNavigationUrl for Hindi Wikipedia'
14:49 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29942 Enable UploadNavigationUrl for Hindi Wikipedia'
14:29 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'revert bug 29942'
14:21 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29942 Enable UploadNavigationUrl for Hindi Wikipedia'
14:08 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php
14:03 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29941 Enable Extension Quiz on hiwiki'
13:55 RobH: ms1004 coming back online, the UEFI doesn't support console redirection
13:52 RobH: ms1004 coming down for testing

July 17

22:00 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29689'
20:59 logmsgbot: hashar: investigating hacked sysop account on hewiki for matanya
20:28 Reedy: Running namespaceDupes on brwiktionary
20:27 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29924 Add appendix namespace for brwiktionary even'
20:24 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29924 Add appendix namespace for brwiki'
19:50 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29210 Add Portal alias for ukwiki'
19:28 Reedy: running cleanupImages.php against commons in dry-run mode
19:20 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 25181 allow import from zhwiki to other zh projects'
18:34 Reedy: Ran populateSha1 on commons for images with the hash "phoiac9h4m842xq45sp7s6u21eteeq1"
18:05 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29534 set to true multilingual Wikisource'
17:57 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29486 Rename user namespace on ka.wiki'
17:44 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 25752 Add some rights to enwiki crats'
17:35 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29830, set wmgWikiLoveDefault for hiwiki'

July 16

23:36 Ryan_Lane: depooling sq63
21:25 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'Fix fatal'
12:49 logmsgbot: ariel synchronized php-1.17/wmf-config/abusefilter.php 'disable abusefilter-private for es wikt, it would reveal ips, see bug 29910'
07:34 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29648 - Set $wgCollectionHierarchyDelimiter = "/" for Wikibooks projects'
07:33 logmsgbot: jeluf synchronized php-1.17/wmf-config/CommonSettings.php '29648 - Set $wgCollectionHierarchyDelimiter = "/" for Wikibooks projects'
07:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28221 - Romanian Wiktionary $wgSitename change and change of namespace name for extra namespace'
07:01 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29913 - ScoWp Import sources'
00:06 Ryan_Lane: restarting job runners

July 15

23:19 RobH: ms1004 refusing to boot of cd burned, updating ticket and will work on it later, booted back online
23:09 RobH: upgraded blog.wikimedia.org to newest wp version 3.2.1
23:04 RobH: rebooting ms1004 into hardware tests per RT#1164
22:11 logmsgbot: reedy synchronized php-1.17/extensions/ProofreadPage/proofread.js 'r92321'
22:10 logmsgbot: reedy synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 'r92321'
22:10 logmsgbot: reedy synchronized php-1.17/extensions/ProofreadPage/ProofreadPage.php 'r92321'
22:10 logmsgbot: reedy synchronized php-1.17/extensions/ProofreadPage/ProofreadPage.i18n.php 'r92321'
21:54 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'bumping epoch for https experiment'
21:50 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'global wmfConfigDir needed, breaks otherwise'
21:48 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Setting wgServer to be protocol relative for testwiki'
21:45 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Fix fatals'
21:37 logmsgbot: reedy synchronized php-1.17/includes/ImageGallery.php 'r92310'
21:25 RobH: db1037 controller replaced, ready for installation
21:25 logmsgbot: reedy synchronized php-1.17/includes/ImageGallery.php 'r92294'
21:24 Reedy: "log reedy synchronized php-1.17/skins/common/shared.css 'r92278'" also merged r92294
21:23 logmsgbot: reedy synchronized php-1.17/includes/upload/UploadFromUrl.php 'r92289'
21:23 logmsgbot: reedy synchronized php-1.17/includes/MessageBlobStore.php 'r92289'
21:22 logmsgbot: reedy synchronized php-1.17/includes/HttpFunctions.php 'r92289'
21:21 logmsgbot: reedy synchronized php-1.17/languages/Names.php 'r92287'
21:21 logmsgbot: reedy synchronized php-1.17/includes/media/SVGMetadataExtractor.php 'r92884 r98226'
21:20 logmsgbot: reedy synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php 'r92283'
21:19 logmsgbot: reedy synchronized php-1.17/includes/api/ApiQueryCategories.php 'r92283'
21:19 logmsgbot: reedy synchronized php-1.17/resources/Resources.php 'r92278'
21:18 logmsgbot: reedy synchronized php-1.17/includes/libs/CSSMin.php 'r92278'
21:18 logmsgbot: reedy synchronized php-1.17/skins/common/commonPrint.css 'r92278'
21:17 logmsgbot: reedy synchronized php-1.17/skins/common/shared.css 'r92278'
21:16 logmsgbot: reedy synchronized php-1.17/extensions/CentralAuth/CentralAuthGroupMembershipProxy.php 'r92270'
21:15 logmsgbot: reedy synchronized php-1.17/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'r92265'
21:03 RobH: mobile1001-mobile1005 & stat1001 racked and mgmt online in eqiad. available for allocation and installation.
19:35 binasher: restarted pdns on ns2
19:29 RobH: updating dns for mgmt on mobile1001-1005, stat1001, and all of the now named misc servers in eqiad.
18:51 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29910 - Activation of AbuseFilter on Spanish Wiktionary'
18:51 binasher: finished setting up replication for s1-7 to eqiad dbs
18:50 logmsgbot: jeluf synchronized php-1.17/wmf-config/abusefilter.php '29910 - Activation of AbuseFilter on Spanish Wiktionary'
18:14 logmsgbot: aaron synchronized php-1.17/wmf-config/CommonSettings.php 'Use $wmfConfigDir temp in the other places'
18:11 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29830 - Enable Extension:WikiLove on Hindi Wikipedia'
18:08 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29825 - Enable Extension:NewUserMessage on Hindi Wikipedia'
18:07 awjr: disabled OWA tracking on all payments boxes ~1 hour ago (forgot to log)
18:07 Jeff_Green: testing mydumper by pulling db1040's database to db1008
17:38 logmsgbot: aaron synchronized php-1.17/wmf-config/CommonSettings.php
17:30 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable Narayam on tawiki per bug 29798'
11:36 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'remove some old commented code'
04:56 Jeff_Green: rebooting db1008 after dist-upgrade
01:12 Ryan_Lane: restarting pybal on lvs2
00:42 Ryan_Lane: adding nginx configuration for all domains to the https clusters
00:41 Ryan_Lane: adding all -lb addresses to the https cluster for LVS. also adding the text.esams.wikimedia.org address to the esams https cluster, since that address will switch to a -lb address in the future.
00:33 Jeff_Green: installed packages on db1008 to support compiling mydumper: python-sphinx libglib2.0-dev libmysqlclient15-dev zlib1g-dev libpcre3-dev gcc make cmake g++
00:21 Ryan_Lane: adding wiktionary-lb, wikiquote-lb, wikibooks-lb, wikisource-lb, wikinews-lb, wikiversity-lb, mediawiki-lb, and foundation-lb addresses to geodns, and forward and reverse for esams and pmtpa

July 14

23:38 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'moved $wgCacheDirectory out'
23:38 logmsgbot: aaron synchronized php-1.17/wmf-config/CommonSettings.php 'moved $wgCacheDirectory here'
23:23 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'removed commented out wgCacheDirectory code'
20:49 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:49 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT percentage from 17% to 20%'
20:48 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Take down AFT clicktracking from 1.6% to 1.35%'
20:27 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:27 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT percentage from 13% to 17%'
20:26 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Take down AFT clicktracking percentage from 2% to 1.6%'
20:14 Ryan_Lane: reenabling portmap and nfs-common services on fenari
20:13 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
20:10 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT percentage from 10% to 13%'
20:08 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Take down AFT clicktracking percentage from 2.7% to 2%'
18:47 Ryan_Lane: disabling portmap and nfs-common services on fenari
18:46 logmsgbot: catrope synchronized php-1.17/includes/parser/Parser.php 'r92178'
18:44 JeLuF: importing 6 huge images to the commons (bug 29872)
18:31 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump cache epoch for HTTPS experiment wikis'
18:25 logmsgbot: catrope synchronized php-1.17/cache/interwiki-pr.cdb 'touching'
18:13 logmsgbot: laner synchronized wmf-deployment/cache/interwiki-pr.cdb 'Re-pushing the protocol relative interwiki cdb, since it had bad write permissions'
18:12 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Adding testwiki back into the https experiment'
18:07 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Taking testwiki out of https experiment'
18:04 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Changing stdlogopr to prstdlogo'
18:03 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Changing stdlogopr to prstdlogo'
17:59 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Adding stdlogopr, for protocol relative url stdlogo for https experiment'
17:57 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Putting testwiki into the https experiment'
17:33 RobH: replaced disk in db1006, fixed
17:07 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29738 - add auto-patrolled group to he.wikiquote'
16:13 mark: Disabled STP on asw-d3-sdtpa
16:02 mark: Halted nas1001-a and nas1001-b for recabling
15:54 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29798 - Install Narayam on Tamil wiki projects'
15:40 mark: Reran initial setup on nas1001-b, rebooting it now
15:30 mark: Reran initial setup on nas1001-a, rebooting it now
15:23 mark: Setup switch ports and LACP LAGs for nas1001
14:16 awjr: disabling puppet on grosley (including its cronjobs)
11:54 mark: Increased MTU on both pmtpa-eqiad links to 9192
11:32 mark: Renamed core-database puppet class into db::core
01:47 RoanKattouw: nfs1 overloaded because I tried to scp too much data to fenari at once. nfs1 and fenari have high wait CPU, /home kinda useless right now. This should sort itself out over time, right?

July 13

23:50 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Fixing the x-forwarded-proto check'
23:47 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Enabling wgSecureCookie for wikis in the https experiment'
23:13 RobH: more dns updates for same servers, forgot to set the mgmt to the new nice names instead of just the asset tag names
23:00 RobH: updating dns for new servers argon, chlorine, germanium, sulfur
22:09 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/ext.narayam.rules.ta.js 'r92105'
22:09 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/ext.narayam.rules.ta99.js 'r92105'
22:00 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Making the apple touch icon for officewiki protocol relative'
21:58 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling protocol relative URLs for officewiki'
21:53 Ryan_Lane: changing cname for office wiki to wikimedia-lb
21:42 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28192 - Patrolled edits on ro.wikipedia'
21:09 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Touch startup.js, workaround for RL bug'
21:06 Jeff_Green: repooling payments3 & payments4
21:02 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Increase AFT percentage from 7% to 10%'
21:01 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Lower AFT clicktracking rate from 3.85% to 2.7%'
21:00 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29727 - Enable the NewUserMessage extension on Wikimedia Incubator'
20:58 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29727 - Enable the NewUserMessage extension on Wikimedia Incubator'
20:57 Jeff_Green: rebooting payments3 & payments4
20:50 notpeter: taking down grosley for reimaging/upgrading
20:43 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29825 - Enable Extension'
20:42 Jeff_Green: depooling payments3 and payments4 for package updates and reboots
20:38 Jeff_Green: repooling payments2
20:38 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise ArticleFeedback percentage from 4% to 7%'
20:38 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Lower AFT clicktracking from 6.75% to 3.85%'
20:27 Jeff_Green: rebooting payments2
20:22 Jeff_Green: depooling payments2 for patches and reboot
20:20 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Increase AFT percentage from 2.7% to 4%'
20:19 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Lower AFT clicktracking rate from 10% to 6.75%'
20:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29796 - Create Wikiversity Arabic'
15:17 RobH: pushed dns change to fix dns error for db12 mgmt
13:26 mark: Rebooting asw-b-sdtpa
13:09 mark: Temporarily disabled BGP aggregates announcement on cr1/cr2-eqiad
00:32 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Setting the powered-by link to a protocol relative url for wikis in https experiment'
00:08 Ryan_Lane: updated dumpInterwiki.php to add a protocolrelative option, which will output the interwiki cdb with all wikimedia interwiki urls set as protocol relative

July 12

23:55 logmsgbot: laner synchronized wmf-deployment/cache/interwiki-pr.cdb 'Pushing out updated protocol relative interwiki cdb again'
23:51 logmsgbot: laner synchronized wmf-deployment/cache/interwiki-pr.cdb 'Pushing out updated protocol relative interwiki cdb'
23:33 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Use interwiki-pr.cdb in a more sane way'
23:21 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Setting interwiki cdb to interwiki-pr.cdb for wikis in https experiment'
23:20 logmsgbot: laner synchronized wmf-deployment/cache/interwiki-pr.cdb 'Pushing protocol relative interwiki cdb for https experiment'
22:55 Jeff_Green: payments1 is back online and repooled
22:40 Jeff_Green: rebooting payments1 for kernel upgrade
22:39 Jeff_Green: depooling payments1 for kernel upgrade
22:20 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Setting more URLs to be protocol relative for HTTPS experiment'
22:20 binasher: running xtrabackup of db33 for clone to db1020
22:20 Jeff_Green: running db hot backup on db43 for clone to db1006
20:57 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Disabling interwiki cache for wikis in the https experiment'
20:56 Ryan_Lane: disabling interwiki cache for wikimedia2005wiki
19:39 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Setting url relative URL for apple touch icon for wikimania2005wiki'
19:34 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Set $wgCacheEpoch to today for wikis with $wmgHTTPSExperiment'
19:26 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'enabling protocol relative urls for wikimania2005wiki'
19:26 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling protocol relative urls for wikimania2005wiki'
19:11 logmsgbot: demon synchronized php-1.17/extensions/LiquidThreads/classes/Hooks.php 'r91986'
19:03 Ryan_Lane: switching meta and commons from using text.wikimedia.org to use wikimedia-lb.wikimedia.org
18:56 Ryan_Lane: updated CommonSettings.php and InitialiseSettings.php, if you push changes right now, you're crazy ;)
15:42 logmsgbot: ariel synchronized php-1.17/includes/Import.php 'er, now with the svn updated file :-P'
15:41 logmsgbot: ariel synchronized php-1.17/includes/Import.php 'add LIBXML_PARSEHUGE to override 10b text node limit in recent versions of libxml'
15:41 logmsgbot: ariel synchronized php-1.17/maintenance/backupPrefetch.inc 'add LIBXML_PARSEHUGE to override 10b text node limit in recent versions of libxml'
09:42 JeLuF: created four new wikis, bugs 29456, 29715, 29758, 29796
09:19 logmsgbot: jeluf ran sync-common-all
07:44 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27557 - Create Sakha Wikisource'
05:50 Tim: restarting job runners
00:27 Tim: running chown/chmod to fix up the permissions on the php-1.17/cache directory, broken by sync-l10nupdate
00:20 Tim: fixed sync-l10nupdate to run as mwdeploy
00:01 MegaTroll: installing vmwaretools on internproxy

July 11

23:59 MegaTroll: updated internproxy. rebooting
23:37 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.5.xsd
23:32 logmsgbot: reedy ran sync-common-all
23:28 Tim: fixed sync-docroot and sync-common-all, they were both trying to push files out without switching user to mwdeploy
23:28 logmsgbot: tstarling ran sync-common-all
23:10 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.5.xsd
23:09 Reedy: Downloaded export-0.5.xsd on fenari
22:41 Jeff_Green: payments1 up and repooled
22:23 Jeff_Green: rebooting payments1
22:14 Jeff_Green: depooled payments1 for package updates and reboot
21:16 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r91909'
21:14 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Doh! $wmgClickTrackingLog needs udp:// prefix'
21:11 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Try using locke's IP address instead of FQDN'
20:56 Ryan_Lane: powercycling payments1.wikimedia.org
20:14 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Using FQDN for lock, for udp clicktracking logging instead of the short name'
19:40 Jeff_Green: purged master logs before 20110611 on db9
19:18 notpeter: pushing out new zone file (only one minor change to cname)
19:00 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'CommonSettings code for wmfClickTrackingLog'
19:00 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add wmgClickTrackingLog setting, logging enwiki to hume:8421'
18:57 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/ApiClickTracking.php 'r91899'
18:57 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php 'r91899'
18:56 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/ClickTracking.php 'r91899'
18:15 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29702 - Enable desysoping by bureaucrats on Hindi Wiktionary'
18:12 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29707 - New logo for Khmer Wiktionary km.wiktionary.org'
12:48 apergos: rebooting snapshot1, after upgrade to new kernel
11:31 mark: Enabled LLDP on csw1-sdtpa
06:06 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling mysql parser cache on all wikis'
06:02 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling mysql parser cache on enwiki'
05:59 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling mysql parser cache on enwikinews'
05:51 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'increased parser cache expiry to 365 days'
05:47 Tim: dropping existing pc* tables and recreating them with different table options
05:40 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling mysql parser cache for server configuration change'
05:38 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
05:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling mysql parser cache on enwikinews only'
05:25 Tim: restarted profile-stats-logger.py
05:22 Tim: fixed umask in reedy's .bashrc
05:19 Tim: dropping old parser cache data from db40, was there since previous pilot

July 10

10:47 Ryan_Lane: restarted the lsearchd daemon on search7. it was dead.

July 9

11:23 mark: IGMP snooping is broken on csw1-sdtpa
10:33 mark: Disabled PIM, multicast routing on csw5-pmtpa, and enabled IGMP snooping on all VLANs
10:25 mark: Disabled PIM, multicast routing on csw1-sdtpa, and enabled IGMP snooping on all VLANs
10:12 mark: Removed static routes on csw1-sdtpa, set default route towards csw5-pmtpa
10:12 mark: Shutdown BGP sessions on csw1-sdtpa
02:45 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 18511 remove hiderevision from oversight group'
02:19 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 27706'
02:18 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27706'
02:18 logmsgbot: reedy synchronized php-1.17/extensions/RSS/RSSHooks.php 'r91775'
02:17 logmsgbot: reedy synchronized php-1.17/extensions/RSS/RSS.php 'r91775'
02:17 logmsgbot: reedy synchronized php-1.17/extensions/RSS/RSS.i18n.php 'r91775'
02:06 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28140'
01:51 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 27078'
01:50 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27078'
01:44 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'Testing showhiddencats for commons'
01:43 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
01:42 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
01:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
00:55 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Lowering ptwikibooks account creation throttle again: event in bug 27172 is since over'
00:43 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Revert that'
00:43 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
00:42 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
00:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Revert that'
00:39 Reedy: bug 22689 Commons (change the default setting for logged-in users)
00:38 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
00:28 Reedy: running namespacedupes on eowikisource finished
00:28 Reedy: running namespacedupes on eowikisource
00:28 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28290 Add eowikisource NSs'
00:23 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28290 Add eowikisource NSs'
00:20 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29641 Appendix namespace for zhwikt'

July 8

23:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Revert bug 28686 per request'
23:32 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29483 add some extra abusefilter lines'
23:23 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Actually sync my brwikimedia changes, turns out I pushed Reedy's changes for bug 29627 (dewiki). Yay conflict resolution'
23:18 Reedy: bug 29267 Namespace aliases for dewiki pushed in Chads last push
23:15 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28686: new namespaces for brwikimedia'
23:03 Reedy: run namespacedupes on mkwikisource
23:02 Reedy: run namespacedupes on rowikisource
22:54 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29190 NSs for rowikisource'
22:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'revert logo, incorrect size'
22:39 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29557, fix sah wikisource logo'
22:32 Reedy: make that ukwiki
22:32 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29740 Enable rollbacker on viwiki'
22:29 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29717 Enable rollbacker on viwiki'
22:07 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29166 Author NS for mkws'
21:59 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28860 botadmin for frwikt'
21:42 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28221 update rowikt namespaces'
21:32 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874'
21:30 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874'
21:27 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874 enable patrol'
21:24 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874, allow mlwiki sysop add/remove patroller, autopatroller, and rollbacker'
21:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874, allow mlwiki sysop add/remove patroller, autopatroller, and rollbacker'
21:16 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28192 patrol stuff for rowiki'
20:48 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29702'
19:45 logmsgbot: reedy synchronized php-1.17/extensions/Collection/Collection.php 'r91753'
19:32 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'kill purgethumbnail logging'
19:31 logmsgbot: reedy synchronized php-1.17/includes/filerepo/LocalFile.php 'r91751'
19:11 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable wikilove on mediawikiwiki'
18:57 logmsgbot: reedy synchronized php-1.17/includes/filerepo/LocalFile.php 'add success logging'
18:51 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php '[purgeThumbnails] = udp://10.0.5.8:8420/purgethumbs'
18:44 logmsgbot: reedy synchronized php-1.17/includes/filerepo/LocalFile.php 'r91746 debugging for testing thumbnail purging'
17:47 mark: Migrated routing to management gateway from csw1-sdtpa to cr1-sdtpa:ae0.402
17:34 mark: Moved transport link between csw1-sdtpa and cr1-eqiad to cr1-sdtpa:xe-0/0/1
17:06 RobH: All ops should be watching eqiad temps and humidity closely over the next few hours/days as the airflow is adjusted to lower relative humidity to acceptable ranges
16:44 mark: Migration completed
16:40 mark: Migrated routing of vlan 100 (squid+lvs) from csw1-sdtpa to cr1-sdtpa
16:28 mark: Migrated routing of vlan 2 (private) from csw1-sdtpa to cr1-sdtpa
16:16 mark: Migrated routing of vlan 101 (public services) from csw1-sdtpa to cr1-sdtpa
16:13 RobH: srv278 shutdown
16:10 mark: Migrated routing of vlan 102 (sandbox) from csw1-sdtpa to cr1-sdtpa
15:51 mark: Migrated routing of vlan 103 (virt) and vlan 104 (pub-svc2) from csw1-sdtpa to cr1-sdtpa
15:30 mark: Migrated routing of vlan 105 (virt-hosts) from csw1-sdtpa to cr1-sdtpa
15:29 mark: Moved transport link between csw1-sdtpa to cr2-eqiad, to cr1-sdtpa port xe-1/1/0
15:01 RobH: db1023 drac fixed, had typo in ip allocation, its ready for os install, rt#1105
14:42 RobH: confirmed that high humidity readings in eqiad are valid, reopened ticket with eq support
14:35 mark: Powercycled asw-a5-sdtpa
07:25 Tim: removed the part of scap that copied skins to /mnt/upload6. This is no longer required, I added a redirect to bits instead.
07:17 Tim: adding redirect from http://upload.wikimedia.org/skins/ to http://bits.wikimedia.org/skins/ using ms7 obj.conf
07:14 Tim: adding http://bits.wikimedia.org/skins/ alias to the latest version of skins, for backwards-compatibility with hot linkers
06:26 apergos: updated snapshot hosts and search hosts to use sudoer file that allows non root syncs
06:25 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 91659: non-root scap test
06:21 logmsgbot: tstarling synchronized php-1.17/README 'non-root sync-file test'
05:58 logmsgbot: tstarling synchronized php-1.17/README 'non-root sync-file test'
05:57 logmsgbot: tstarling synchronized php-1.17/README 'non-root sync-file test'
03:56 Tim: in puppet, removed the subscribe from the rsyncd service for /home, since the init script is buggy and causes the service to go down (bind() failed: Address already in use). Rsyncd doesn't need it anyway, it re-reads the configuration file at each request.
03:34 Tim: on nfs1: changed rsyncd.conf to allow client IPs from a subnet that includes mobile1, since it looks like patrick tried to set up mediawiki on it. Ran sync-common on mobile1.
03:00 Tim: deleting and recreating /usr/local/apache/common-local on all servers, to get mwdeploy ownership
02:39 Tim: updated sudoers.appserver and /h/w/b/sync-common-file to support the sync script changes
02:33 Tim: updated wikimedia-task-appserver manually on searchidx1, searchidx2, fenari, hume. Manually removed the php.ini-dist diversion on searchidx1, see CR r76095
02:00 Tim: built wikimedia-task-appserver 2.2 for lucid, uploading to brewster, since we have some lucid boxes in production
01:35 Tim: uploading wikimedia-task-appserver 1.48 to brewster. Waiting for puppet to immediately wreak havoc in response.
01:25 Tim: re-enabled apache on srv255. Note that sync-file pushes to it will fail until I finish updating everything
01:17 Tim: did a test install of wikimedia-task-appserver version 1.48 on srv255
01:06 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28221 rowikt sitename'
00:56 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29217 Add Portal namespace to Azerbaijani Wikisource'
00:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29381 Change title of Kannada wiktionary'
00:34 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'revert'
00:34 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29381 Change title of Kannada wiktionary'
00:30 logmsgbot: reedy synchronized closed.dblist
00:22 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'fixing sitemap for fawikinews'
00:21 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'fixing sitemap for fawikinews'

July 7

23:26 Tim: temporarily stopping apache on srv255 for sync script testing
22:59 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 26732'
22:29 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29483'
21:45 Ryan_Lane: changing the spammer blocks from blocking post to blocking all requests
21:22 Ryan_Lane: dropping the weight of ms1004 to 5
21:15 Ryan_Lane: upping weight of sq79 and sq82
21:14 Ryan_Lane: repooling sq79 sq82
20:59 Ryan_Lane: depooling sq79 and sq82 in lvs
20:47 Ryan_Lane: lowering carp weight of sq79 and 82
20:45 Ryan_Lane: repooling sq79 and sq82 in lvs
20:44 Ryan_Lane: temporarily depooling sq79 and sq82
20:00 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29360'
18:47 Ryan_Lane: adding another IP to that search block
18:38 Ryan_Lane: deploying new squid conf with a blocked ip, to block a search spammer
18:04 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'More GNSM: had to set comment namespace and fallback category'
17:59 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'GNSM for fawikinews: bug 29563'
17:52 Ryan_Lane: that's powering
17:52 Ryan_Lane: powing off srv154
17:51 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap.alias.php
16:39 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27840, moving wiki'
15:45 mark: Established full iBGP mesh between cr1-sdtpa, csw5-pmtpa, cr1-eqiad, cr2-eqiad (NOT csw1-sdtpa)
00:25 awjr: disabling fundraising-related squid log filtering for udp2log

July 6

23:45 Ryan_Lane: powercycling srv281
23:44 Ryan_Lane: powercycling srv266
23:44 Ryan_Lane: powercycling srv217
23:43 Ryan_Lane: powercycling srv206
23:42 Ryan_Lane: powercycling srv154
23:40 Ryan_Lane: powercycling srv276, it's dead
22:56 mark: Setup cr1-sdtpa with initial config; connected to csw5-pmtpa (via L2 csw1-sdtpa); OSPF up
22:36 RobH: wmf_ops: you can disregard the humidity alarms for eqiad that are spamming alerts to email. eq confirms no humidity issue on site and I will investigate the actual sensors this friday
22:21 Ryan_Lane: repooling srv169, it was missing the wikimedia-lvs-realserver package. fixed in puppet
22:04 Ryan_Lane: rebooting srv169
21:59 Ryan_Lane: depooling srv169
21:59 Ryan_Lane: sync'd srv169, repooling to test
21:54 Ryan_Lane: depooling srv169
21:51 Ryan_Lane: restarted apache on singer
21:46 logmsgbot: laner synchronized php-1.17/wmf-config/db.php 'depooling srv154, repooling srv178'
21:30 logmsgbot: pdhanda ran sync-common-all 'Synced to r91606 for ArticleFeedback'
21:27 Ryan_Lane: setting proxy setting for secure back to original setting. removed ~ files from sites-enabled
21:17 Ryan_Lane: putting retry=3 back, and adding a timeout of 15 seconds to secure
21:16 Ryan_Lane: removed retry=3 from ProxyPass directive for secure. 3 seconds really isn't enough for this service...
21:06 RobH: running puppet on spence, this is going to take forever.
21:05 Ryan_Lane: restarting apache on singer
19:37 mark: Added DNS entries for cr1-sdtpa and cr2-pmtpa
19:25 logmsgbot: hashar: hexmode raised an user issue with blocking. It is a lock wait timeout happening from time to time on enwiki. 30 occurences in dberror.log for Block::purgeExpired. Could not reproduce it so I am assuming it was temporary issue.
19:15 logmsgbot: hashar: srv154 seems unreachable. dberror.log is spammed with "Error connecting to <srv154 IP>"
19:13 RobH: added webmaster@ to other top level domain mail routing to forward to the wikimedia.org webmaster for google securebrowsing stuff per RT#1122
18:08 pdhanda: running maintenance/cleanupTitles.php on commonswiki
17:51 pdhanda: Running maintenances/namespaceDupesWT.php on commonswiki
17:12 RobH: srv169 successfully back in service, tests fine and has all updated files, lvs3 updated to include it in pool
17:11 RobH: returning srv169 into service
15:37 mark: Removed ms5:/etc/cron.d/mdadm
15:37 mark: Stopped MD raid resync on ms5
15:28 RobH: search18 booted back up successfully
15:25 RobH: api lag issues known due to search server failure, being worked presently
15:24 RobH: search18 sas configuration bios confirms both disks are still in a non-degraded (according to it) mirror
15:23 RobH: search18 randomly rebooted after checking disks before the login prompt
15:19 RobH: rebooting search18
15:14 RobH: search18 appears to be completely offline, investigating lom logs before rebooting.
15:12 RobH: search18 offline, logging into mgmt to check it out
15:01 RobH: eqiad humidity levels ticket dispatched for fufillment
14:37 mark: Paused rsyncs on ms5
14:04 mark: Powercycled sq36
13:18 ^demon|away: fixed permissions on svn c/o on ci.tesla, ran svn cleanup. cruise control still not pleased and yelling about locks
13:16 mark: Upgrading firmware of scs-a1-sdtpa
12:51 mark: csw5-pmtpa crashed and reloaded
11:53 mark: Upgrading firmware of scs-c1-pmtpa

July 5

23:53 ^demon: scratch that....ssh just seems to have been rather slow in getting its act together. ci.tesla is just fine now
23:50 ^demon: well now I've locked myself out of ci.tesla. Seems it doesn't start ssh on reboot...what a silly thing to do
23:47 ^demon: rebooting ci.tesla since it was horribly hung up on the latest build--was it really stuck for the past 24hrs?
23:35 logmsgbot: reedy synchronized php-1.17/resources/mediawiki/mediawiki.js 'r91505'
21:28 logmsgbot: pdhanda ran sync-common-all 'Synced to r91494 for WikiLove'
21:10 logmsgbot: pdhanda synchronized php-1.17/resources/jquery.ui/themes/vector/jquery.ui.button.css 'Synced to r 91493.'
21:00 logmsgbot: pdhanda synchronized php-1.17/resources/jquery.ui/themes/vector/jquery.ui.button.css 'Synced to r 91490.'
19:53 logmsgbot: reedy synchronized php-1.17/includes/api/ApiQuery.php 'r91479'
19:07 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.collapsibleTabs.js 'r91476'
19:07 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.simpleSearch.js 'r91476'
16:49 RoanKattouw: Short CPU spike on the Apaches, approx 16:45-16:50 UTC. Things seem to be recovering now
16:04 mark: Put OSPF/OSPFv3 on csw1-sdtpa:e14/1 in active mode again; appears stable so far
15:45 mark: Reenabled csw1-sdtpa:e14/1 with OSPF/OSPFv3 in passive mode and high metric
15:31 mark: Shutdown csw1-sdtpa:e14/1 (l3 wave)
15:24 mark: Disabled IP load-sharing on csw1-esams (out of next-hops), disabled e14/1 (10G wave L3), reenabled it
13:38 mark: Rebooting streber; dpkg hung
08:34 mark: Rerouted traffic AS14907 -> AS43821 via 2828
07:11 Tim: creating 256 tables on db40 for DBOC
07:08 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'config change for the next DBOC trial, unused at present'
07:04 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/SqlBagOStuff.php 'r91442: MySQL parser cache fix, ready for a new trial'
06:54 logmsgbot: root synchronizing Wikimedia installation... Revision: 91441:
06:53 logmsgbot: tstarling synchronized php-1.17/includes/AutoLoader.php
06:53 logmsgbot: tstarling synchronized php-1.17/includes/libs/IEUrlExtension.php
06:48 Tim: doing svn up/scap to get r91440

July 4

19:29 mark: Restarted lsearchd on search7
14:02 apergos: restarted morebots
13:51 mark: Powercycled ms1004
12:52 mark: Started rsync of upload images on ms7 to ms1002 (eqiad) in a screen on ms1002
12:35 mark: OS-installed ms1002
08:43 morebots died because of netsplit
04:58 Tim: setting up lighttpd mod_auth on lily

July 1

20:18 logmsgbot: hashar: ci.tesla: installed exiv2 (required for some Api tests)
20:13 andrewS: mw1-mw22 racked and drac setup
19:49 RobH: typo in my number sequence update for the new 74 mw servers, updating dns to fix
19:09 Ryan_Lane: Changing text backend for https in esams to textsvc IP instead of text.esams.wikimedia.org IP
18:49 Ryan_Lane: restarting pybal on amslvs1 and amslvs3
18:34 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'removing mysql from pcache, need to repartition, current scheme too costly on memory'
18:25 Ryan_Lane: adding textsvc.esams IP to amssq and knsq, and amslvs servers
18:23 logmsgbot: hashar: ci.tesla install wbritish-small package to get a word list in /usr/share/dict
18:08 Ryan_Lane: restarting pybal on lvs2 to add textsvc service
17:40 andrewS: Rob Halsell is a Wizard. Period.
17:39 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
17:29 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding enwiki to the mix'
17:27 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'sending a bit lower cache load to db40, though I guess that was initial shock before'
17:25 RobH: Aware that #wikipedia is down and fixed, in less than a minute, woot
17:25 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'oops, should have started with lower load, opentable cache..'
17:23 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'sending cache load again to db40'
17:16 Ryan_Lane: adding text.svc address to pmtpa text squids and lvs2
17:08 Ryan_Lane: adding text.svc private address to esams and pmtpa DNS
14:56 RobH: pdns crash after rehup on ns1.wikimedia.org, restarting pdns
14:49 RobH: dns updated to remove the mgmt ip addresses for srv302+ and replace them with mw1-mw74
14:33 RobH: updating dns, removing entries for non-existent srv302+ and replacing them with mw1-mw74
11:18 andrewS: motion sensors are being installed today in sdtpa that are going to trip the lights when there is movement. not sure how this will affect cameras. will update once their work is done later today.
07:33 andrewS: the 3 new racks in pmtpa have power plugs running to them now.
05:51 apergos: cron job set up on snapshot1 to dump central auth tables (in subdir by date just like all the private wikis) on weekly basis, runs as backup user
00:01 RobH: disabled the notify requesters on resolve in the hw-decommission queue in RT

June 30

20:43 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/special/SpecialBannerLoader.php 'r91218'
20:42 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/special/SpecialBannerController.php 'r91218'
18:39 RoanKattouw: Running extensions/Vector/switchExperimentPrefs.php on enwiki to disable WikiLove for those with the "Exclude from experiments" preference
18:38 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove by default on enwiki'
18:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on enwiki, non-default initially'
18:18 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 91190:
18:17 RoanKattouw: Deploying updates for ArticleFeedback and WikiLove
18:09 RobH: srv169 still offline, do not repool (set to false in pybal on lvs3)
18:00 RobH: doing reboot testing on srv169 and puppet testing. server is depooled, disregard notices.
17:56 RobH: depooled srv169, its not updating its /var/local/apache via puppet for some reason.
17:53 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add code enabling WikiLove by default if $wmgWikiLoveDefault is true'
17:52 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add wmgWikiLoveDefault variable, set to false'
17:48 RobH: blog.wikimedia.org updated to wp3.1.4
16:48 RobH: srv169 reinstalled and returned to service. (lvs3 is not repooling, citing a key error that has since been removed, perhaps pybal nees to be refreshed?)
16:25 RobH: reinstalling srv169
10:57 mark: Changed preferred path for AS14907 (pmtpa/eqiad) - AS43821 (esams) from 14907 2828 6908 43821 to 14907 3257 1299 43821 (bidirectionally)

June 29

23:39 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 91090:
23:38 awjr: running scap to deploy CentralNotice changes (r91118)
23:25 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/centralnotice.css 'r91118'
23:24 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/centralnotice.js 'r91118'
23:03 RobH: db1001-db1048 (minus a few hosts mentioned in rt#1105) are installed, puppetized, and ready to be placed into replication service.
22:53 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Making CentralNotice config changes for test wiki (page path and CentralDBname)'
21:03 RobH: OS install running on db1033-db1048
20:53 RobH: db1033-db1048 raid setup, ready for OS installs
20:44 mark: Started rsync of ms5 commons thumb dir '1' to ms1004
20:23 RobH: working on db1033-db1048 (setting up hardware raid)
20:10 RobH: db1017-db1031 os installed, puppet updated, ready for replication setup (db1023 & db1032 need further troubleshooting)
19:54 RobH: os install on db1017-db1031 done, running puppet updates
19:45 mark: Started rsync of ms5 commons thumb dir '0' to ms1004
19:36 mark: Deployed ms1004 (eqiad) as caching thumbs server in a similar configuration as ms6.esams; requesting thumbs from ms5 and caching (and purging) them locally
19:35 RobH: db1023 and db1032 drac unresponsive, rt# 1111 abd 1112 to troubleshoot next onsite
19:23 RobH: rob is working installs on db1017-db1032
19:08 RobH: updated netboot.cfg for private1-b-eqiad broadcast
18:46 RobH: raid10 setup on db1017-db1027
18:30 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 91090:
18:27 RoanKattouw: Deploying r91089 (jQuery in the head and position=top support) with scap
18:11 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on testwiki'
17:58 mark: OS-installed ms1004
17:33 RobH: os install running on db1017-db1021
17:22 mark: Deployed DNS entries for ms1001-1004
17:20 RobH: db1015 and db1016 os installed, puppet updated, ready to go into replication service
17:20 RobH: corrected dns, pushed db1017 and upwards into eqiad-b-private ip range
17:18 mark: Caught ms5 doing RAID10 resync check, stopped it using echo idle > /sys/block/md1/md/sync_action
16:43 RobH: updating dns for the all eqiad dbs
16:36 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'In the spirit of bug 29277: also set scriptDirUrl for $wgLocalFileRepo'
15:49 RobH: starting install on db1015-db1018
15:45 RobH: db1011, db1012, db1013, db1014 os installed, puppet updated, ready to go into replication service
14:49 RobH: installing db1011, db1012, db1013, & db1014
14:40 RobH: db1007, db1008, db1009, & db1010 os installed, puppet updated, ready to go into replication service
14:32 RobH: stat1.wikimedia.org server online for Erik Z
13:46 RobH: db1007, db1008, db1009, db1010 OS installed, needs puppet run
13:42 RobH: starting os install on stat1, db1010 and db1009. mislogged before, was finishing install on db1007 and db1008
13:39 RobH: updated dns with stat1 server info
13:31 RobH: db1008 & db1009 finishing OS install
01:50 RobH: db1005 & db1006 os installed, puppet updated, ready to go into replication service
00:42 logmsgbot: demon synchronizing Wikimedia installation... Revision: 91036: Pushing r91036 for Filepage.css fixes

June 28

23:29 RobH: db1003 and db1004 os installed, puppet updated, ready to be put into replication service
22:46 RobH: db1001 installed and updated with puppet, ready to be put into replication service
22:45 RobH: db1002 installed and updated with puppet, ready to be put into replication service
22:44 RobH: db1001 is installed with OS and up (has been for awhile), db1002 and db1003 finishing installs
22:24 RobH: carbon host online as install tftp host in eqiad.
22:10 RobH: syncing docroot and gracefully restarting apaches to take the update
22:01 RobH: carbon rebooting
21:43 RobH: testing tftp function of carbon from cp1001 as its on the same public vlan network
21:13 RobH: updated puppet to point tftp hosts in eqiad to carbon
21:13 RobH: rsyncing over the tftpboot directory from brewster to carbon
21:00 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. Change requested by jamesofur on irc'
20:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. This time without breaking the world'
20:33 RobH: reinstalling carbon
20:27 RobH: updated misc::install-server and brewster/carbon entires in puppet, install server roles are now called on a service by service bais. brewster also needs reboot for kernel update, so doing that now as well.
17:47 RobH: carbon host powered down, the misc::install-server class in puppet needs to be broken out into more specialized roles, carbon reinstalled and updated to just host the tftp roles of the install server
17:36 RobH: new install host carbon os installed, running puppet updates
17:11 RobH: copied db41 sudoers file to db40 since it was overwritten by old raid utils package. now maybe the log spam will stop
15:20 RoanKattouw: NTP has synced clock in prototype, it's got the correct time now
15:18 RoanKattouw: Installed ntp on prototype, its clock was 10 minutes behind. It seems to be slowly catching up now
14:58 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling while my eyes are on other stuff'
14:52 RobH: stat1 racked and remote accessible. has public vlan setup, needs IP allocated and os installed. (also needs to be added to racktables)
13:59 mark: Partly implemented European routing policy on AS14907 as well, and tuned a bit for routing southern routes via Tampa
13:27 logmsgbot midom cleared profiling data
13:27 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
13:25 logmsgbot midom cleared profiling data
13:24 logmsgbot midom cleared profiling data
13:20 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
13:19 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
13:13 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round ended'
13:12 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
11:35 Nemo_bis: !Wikipedia and other !Wikimedia sites problems at 6:15 and 9:30 UTC due to database overload fixed in 30 min, config now tweaked.
10:41 mark: Set 'send-community' on the iBGP peer groups on csw1-sdtpa and csw5-pmtpa
10:41 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
10:39 domas: reenabled full mysql pcache, watching with half of my right eye
10:39 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
10:38 domas: reduced max_connections on db40 to 1000, enabled "connection recycling" (disconnects longest idling connections when max_connections gets reached), set wait_timeout=2
10:30 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding enwiki to the mix'
10:23 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling db-pcache on eswiki for transaction analysis'
09:52 logmsgbot: mark synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable MySQL parser cache'
09:22 mark: Activated BGP sessions to TiNet / AS3257
06:49 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled SubPageList3 on wikiversity'
06:45 logmsgbot: tstarling synchronized php-1.17/extensions/SubPageList3/SubPageList3.php 'r90659, fix fatal error'
06:31 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/SqlBagOStuff.php 'r90942, proper fix'
06:21 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling MySQL parser cache'
06:20 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/SqlBagOStuff.php
06:16 Tim: temporarily disabled db40 parser cache due to connection errors
06:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
06:14 logmsgbot: root restarted all apaches
06:14 logmsgbot: root is doing a hard restart of all apaches (apaches overload)
00:31 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on all wikis'
00:09 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on enwiki'

June 27

23:22 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'no automatic purging'
23:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on en.wikiquote'
21:54 Ryan_Lane: repooling sq69
21:09 RobH: grosley back online
21:00 RobH: grosley reboot in process, disk checking.
20:57 RobH: powercycling grosley
20:56 RobH: grossley entirely unresponsive from serial console
20:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29603 - enabled abusefilter for iawiki. Bug 29279 - turned off Devanagari digits for hiwiki'
20:36 logmsgbot: pdhanda ran sync-common-all 'Enabled abusefilter for iawiki. Bug 29603'
20:18 RobH: allocated 11/41 on csw1-sdtpa to vlan101 for use by stat1
19:32 RobH: raised power warnign threshhold on phases on ps1-d1-sdtpa from 12 to 16
18:13 RobH: rebooting carbon for installation in eqiad.
18:06 RobH: updating dns to have carbon mgmt dns info
17:42 andrewS: 47 r410 servers now await placement @ sdtpa
17:13 andrewS: 36 new dell servers sit outside sdtpa cage and await placement
16:40 Ryan_Lane: depooling sq69 for continued work on XFF for geoiplookup
15:30 JeLuF: increased puppet's timeout for "apt-get update" from 2 to 4 minutes. Regular apt-get update on storage1 takes more than 2 minutes
11:50 mark: Changed ganglia / gmond configuration in puppet to use a different, associative hash structure
03:06 Tim: changed mailman site password

June 26

13:13 rainman-sr: built search index for nycwikimedia

June 25

03:26 andrewS: 8 new servers in sdtpa d1 awaiting labels and power/data hookups
03:25 andrewS: sdtpa feels hotter than normal. will keep an eye on cage temp next few days.
00:30 Ryan_Lane: repooling sq69

June 24

22:26 notpeter: initialized and started repl between nfs1 and 2 for opendj
22:08 Ryan_Lane: depooling sq69 to test geoiplookup changes for https
21:36 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28529 - Change move-rootuserpages right in Portuguese Wikipedia'
21:32 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28398 - Custom namespaces for ku.wiktionary and ku.wikipedia'
21:24 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29527 - Enable AbuseFilter on Catalan Wikipedia'
21:23 logmsgbot: jeluf synchronized php-1.17/wmf-config/abusefilter.php '29527 - Enable AbuseFilter on Catalan Wikipedia'
20:00 andrewS: pmtpa: we should have power and the overhead railings to run data cables by late monday, as per Matt.
19:20 Ryan_Lane: depooled srv301
12:37 Tim-away: finished fixing ExtensionDistributor, re-enabled cron job and xinetd
07:37 Tim: XO appears to be back up, after ~3 hours of downtime
07:30 Tim: fixing ExtensionDistributor snapshot permissions, was owned by mark instead of extdist
06:21 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29168 - Define namespace aliases for Author, Page, Index on all Wikisources'
06:10 Tim: configured csw5-pmtpa to send its syslogs to 10.0.5.8 instead of 10.0.0.1
00:19 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'reapplied r90644, it was fixing an actual problem'

June 23

22:36 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 14407. Added Login icon for incubatorwiki'
22:33 logmsgbot: pdhanda synchronized images/sul/incubatorwiki.png 'Bug 14407. Login icon for incubatorwiki'
22:18 logmsgbot: pdhanda synchronized php-1.17/wmf-config/CommonSettings.php 'Bug 14407. Added incubatorwiki to CentralAuthAutoLoginWikis'
21:30 logmsgbot: catrope synchronized php-1.17/maintenance/update.php 'Sync out update.php too, just because Ariel is being paranoid about it :D'
21:27 ^demon: svn up'd /h/w/c/p/m/update.php for nice warning about not running update.php on the cluster
21:06 logmsgbot: pdhanda ran sync-common-all 'Updating to r90676 of extensions/UploadWizard'
21:03 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Enabling wgNoticeInfrastructure for testwiki per request by Kaldari'
20:36 pdhanda: updated to r90667 of extensions/Narayam
20:33 logmsgbot: pdhanda ran sync-common-all
18:38 JeLuF: #21237 - removed brion's email address from apache config
18:10 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28909 - Change path to Low Saxon Wiktionary logo'
18:04 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29391 - Enable e-mail notification for watchlists on br.wikimedia'
18:03 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29348 - Email notifications for stewardwiki'
18:00 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29360 - Define $wgUploadMissingFileUrl for ptwikibooks'
17:02 binasher: sync-file r90664 includes/db/LoadMonitor.php
17:01 logmsgbot: root synchronized php-1.17/includes/db/LoadMonitor.php 'max threads logging'
16:47 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'attempting to set max threads on testwiki only'
16:32 RoanKattouw: Site came back up instantly after Tim disabled max_threads
16:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'disabled max threads'
16:28 logmsgbot: catrope synchronized php-1.17/includes/db/LoadBalancer.php 'Make the "LB failure with no last connection" DB error log message more verbose'
16:24 binasher: upping max threads in db.php
16:24 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'upping max threads'
16:17 logmsgbot: laner synchronized php-1.17/includes/db/DatabaseMysql.php 'Roll back Tim's getLag() change from 03:18 today (r90644)'
15:58 Ryan_Lane: restarting pybal on lvs3
15:52 logmsgbot: catrope synchronized php-1.17/includes/db/Database.php 'Roll back Tim's getLag() change from 03:18 today (r90644)'
15:50 RoanKattouw: Going to roll this back: 03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644'
15:46 RoanKattouw: Lots of "LB failure with no last connection" errors in dberror.log, suddenly started at 10:53 UTC
15:43 RoanKattouw: Torrus broken, reporting lots of NaNs
15:40 RoanKattouw: Strike that, site not completely down. Intermittent service reported in US/Canada
15:39 RoanKattouw: AMS squids all down in Nagios. Load is approximately halved on app servers. TPA squids look like they should be fine, but the site is reported to be down worldwide
15:37 RoanKattouw: Mark not near a computer, texting Ryan instead
15:36 RoanKattouw: All wikis seem to be down worldwide. Have texted mark
09:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29215 - Add import source for dearbcomwiki'
09:14 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29380 - Change logo of Kannada wiktionary (kn.wiktionary.org)'
08:48 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29515 - install Narayam in all Sanskrit wiki projects'
08:42 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29474 - Search Index and Author namespaces by default on all Wikisources which don't have a different configuration'
08:27 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29172 - Add Page, Author and Index namespaces to $wgContentNamespaces on all Wikisources'
07:53 JeLuF: fixed broken ganglia installation on project2 (reported by puppetmon)
07:51 JeLuF: added new class mobile::disabled in puppet that has apache disabled, assigned mobile1 to it.
07:19 logmsgbot: tstarling synchronized php-1.17/extensions/SubPageList3/SubPageList3.php 'r90654'
07:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'enabling SubPageList3 on testwiki'
07:11 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
07:07 logmsgbot: root synchronizing Wikimedia installation... Revision: 90493:
07:05 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 90493:
07:04 Tim: running scap to push out SubPageList3 source directory (not enabled)
06:36 Tim: killed puppet-monitor on fenari, I'm pretty sure that's the thing that was spamming #wikimedia-tech every 3 minutes
05:34 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled mysql parser cache on testwiki'
05:33 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'mysql parser cache pre-deploy changes'
05:32 Tim: created "parsercache" database on db40 and added suitable wikiuser/wikiadmin grants
04:46 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db33 back'
04:46 Tim: restarted the parser cache hit rate monitor
03:20 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'restored max lag to 30s, now that r90644 is deployed it shouldn't break anything'
03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644'
03:03 Tim: started ntpd on db36, not sure why it was stopped, it has an init script. Clock skew was 1.7 seconds.
02:55 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'increasing max lag to 3000'
00:43 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'removed db33 for TS dump'

June 22

23:08 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabled extension NewUserMessage on rowikisource. Bug 28307.'
22:47 Ryan_Lane: repooling sq69
22:07 Ryan_Lane: depooling sq69 to test vcl changes for https geoiplookup
21:23 notpeter: resized db10:/a to 220G
19:10 Reedy: Wikimedia repo post commit hook fixed for CR
18:43 Reedy: Wikimedia repo set up as a CodeReview repo on mw.org
18:04 JeLuF: fixed puppetmon, log format has changed
13:29 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db21 back'
10:41 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db34 back'
02:14 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'remove db34 to dump for TS'
02:08 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'remove db21 to dump s6 for TS'
01:59 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db18 back'

June 21

23:33 logmsgbot: pdhanda ran sync-common-all 'Creating nyc.wikimedia.org Bug 29273'
23:29 pdhanda: creating nyc.wikimedia.org
22:41 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'Attempt to fix fatal'
22:14 binasher: pulling srv301 from api lvs pool to prep mobile testing
21:40 binasher: returned mobile4 to lvs
20:32 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js 'r90541'
20:32 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'r90541'
20:31 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.i18n.php 'r90541'
20:31 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'r90541'
19:25 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'taking db18 out of rotation to dump s7 for TS'
19:26 river: taking db18 out of rotation to dump s7 for TS
18:56 Ryan_Lane: moving preilly from restricted to mortals, for deploy access
15:01 apergos: deleted the old ssh key for snap2.wm.o from the puppet db also, as fenari was whining about it similarly, hope that gets is. spence puppet run *finally* done, sheesh
13:17 apergos: after changing snapshot2 from external to internal subnet (oops forgot to log that huh), puppet on spence was complaining about the external resource Nagios_host[snapshot2] not being able to override the local resource, so in puppet db on db9 I deleted the old snapshot2 nagios resources, leaving only the new ones, puppet run is in process now on spence and is bleeping sloooow.
09:14 apergos: taking down snapshot2 in prep for os upgrade (reinstall)
01:52 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Fix checkuserwiki permissions back to how they were'
00:49 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Permission changes for checkuserwiki. Bug 28781.'

June 20

23:57 binasher: reconfiguring mobile1 for testing the new php mobile mediawiki extension
23:03 notpeter: added checks for google safe browsing for lots of MWF sites to nagios
22:53 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Missed Comment 1 in bug 28785.'
22:42 logmsgbot: reedy synchronized php-1.17/includes/api/ApiEditPage.php 'bug 29278 r90943'
21:57 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Changed language, sitename and namespace for checkuserwiki. Bug 28781.'
21:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Changed icons on checkuserwiki. Bug 28785.'
19:20 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r90480'
18:51 binasher: drop db29 (s6 master) weight to 0
18:51 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'drop db29 weight to 0'
18:21 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'adding db43 to s6 with a weight of 20'
18:21 binasher: increased db43 weight to 1500
18:07 binasher: just put db43 into rotation in the s6 cluster
18:04 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'adding db43 to s6 with a weight of 20'
16:53 pdhanda: Upgrading wikitech.wikimedia.org to 1.17
15:43 apergos: started en pedia dumps run on snapshot 4, from root screen session as backup user. This will run as 32 jobs in parallel; we should keep a half an eye on the db in about 16-17 hours (previous runs had only 15 jobs going at once).
14:18 mark: Configured TiNet transit on cr2-eqiad (BGP sessions deactivated)
14:18 mark: Configured border ACL filtering on the eqiad border
00:46 logmsgbot: tstarling synchronized php-1.17/includes/db/LoadBalancer.php 'r90423'

June 17

20:39 RobH: email is working again, but address in question is not filtered
20:38 RobH: pushed a small exim change to try to sort out a sender address, but it didnt work, rolled back changes
16:44 RobH: upgraded blog.wikimedia.org to wordpress 3.1.3
16:44 RobH: restarted morebots.
00:33 Ryan_Lane: restarting pybal on amslvs4
00:32 Ryan_Lane: restarting pybal on amslvs2
00:31 Ryan_Lane: restarting pybal on amslvs3
00:30 Ryan_Lane: restarting pybal on amslvs1

June 16

23:51 Ryan_Lane: restarting pybal on amslvs3
23:51 Ryan_Lane: restarting pybal on amslvs1
23:50 Ryan_Lane: adding uploadsvc and uploadsecure services to amslvs1 and amslvs3
22:25 Ryan_Lane: restarting pybal on lvs1
21:47 RobH: puppet now works on eqiad hosts.
21:38 Ryan_Lane: enabled https for bits. still needs varnish changes.
21:38 RobH: updating dns with puppet cname in eqiad to sockpuppet
21:37 RobH: dispatched gilman to peter for puppetize testing of grosley
21:21 RobH: dns update for puppetmaster cname in eqiad
21:13 RobH: updated dns with carbon install server and cp server ranges
21:10 mark: Added all eqiad subnets to dhcpd.conf
21:10 mark: Prepared puppet's base.pp for eqiad
21:05 Ryan_Lane: restarting pybal on amslvs3
21:03 Ryan_Lane: killing and starting pybal on amslvs1
21:03 Ryan_Lane: restarting pybal on amslvs1
21:01 Ryan_Lane: adding bitssvc address to amslvs systems
20:52 notpeter: restarting drbd replication on nfs2
20:52 RobH: db1001 is our first wmf proper installed host in eqiad, huzzah
20:44 mark: Added static route for 10.2.1.23/32 to lvs2 on csw1-sdtpa
20:09 mark: Added 10.64.0.0/12 to squid http_access on brewster
18:51 Ryan_Lane: bound bitssvc address to lvs2, restarting pybal
18:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php
18:24 Ryan_Lane: restarting pybal on lvs2
18:24 Ryan_Lane: adding bitssvc and bitssecure services to pmtpa lvs
18:01 notpeter: reimaging nfs2
16:36 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php
14:54 andrewS: srv169 plugged in and powered on
14:26 RobH: updating again, there are db1001-1016 in a single rack, doh
14:24 RobH: updating dns with new dbs in eqiad
12:00 RoanKattouw: Set $wgInstantCommons back to true on prototype
11:59 RoanKattouw: Enabled WikiLove on prototype
11:15 andrewS: will be working on srv278 later today
11:12 logmsgbot: root synchronized php-1.17/wmf-config/mc.php
09:50 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account creation throttle for el pedia workshop'
01:38 Tim: success, my SIP connection now works again
01:38 Tim: attempting to remove the SIP connection rate limit from sfoasterisk

June 15

21:32 awjr: killing epicly long running query in civicrm db on db9
21:28 logmsgbot: reedy synchronized php-1.17/wmf-config/codereview.php 'Couple more auto defers'
19:07 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/WikiLove.api.php 'r90138'
18:47 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.core.js 'r90135'
18:45 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/WikiLove.hooks.php 'r90135'
18:18 logmsgbot: catrope synchronized php-1.17/resources/mediawiki.util/mediawiki.util.js 'r90128'
18:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on officewiki'
18:02 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add switch for WikiLove'
18:01 mark: Upgraded msw1-eqiad to JUNOS 10.4R3.4
17:59 RoanKattouw: Adding WikiLove table on officewiki (extensions/WikiLove/patches/WikiLoveLog.sql)
17:58 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add $wmgUseWikiLove, false by default'
17:53 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 90124:
17:52 RoanKattouw: Syncing WikiLove code. Code sync only, not enabled anywhere yet
16:19 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Revert raised account creation throttle for Lennart on testwiki'
12:37 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29351 Set timezone to America New_York for pcdwiki'
05:17 apergos: phase 2 of integrity checks on dump files (bz2s) running from screen on sn1 as root

June 14

19:41 RobH: chad is crazy it seems.
19:41 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29215: import sources for arbcom_dewiki'
19:40 logmsgbot: catrope synchronized php-1.17/README 'Test sync to confirm whether Chad is crazy'
18:29 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'bug 29331 - Set e-mail capture sender to improve@wikimedia.org'
17:50 notpeter: restarting udp2log on locke after upgrade
15:10 mark: Upgraded asw-a-eqiad to JUNOS 10.4R3.4
14:42 mark: Upgraded asw-b-eqiad to JUNOS 10.4R3.4 (jtac recommended version)
14:26 apergos: running a bunch of rudimentary integrity checks on xml dumps, out of screen as root on snapshot1
06:09 logmsgbot: tstarling synchronized php-1.17/includes/HistoryBlob.php 'r90036'

June 13

23:40 notpeter: taking down nfs-home on nfs2
23:03 Ryan_Lane: ran sync-docroot - how does this not get logged?
19:16 RobH: snapshot4 properly boots to serial console display on bootloader and os
18:31 RobH: snapshot4 rebooting to troubleshoot console redirection
13:27 apergos: snapshot4 installed and *almost* correct (stray apache started up but the rest looks ok)

June 12

19:47 JeLuF: Software RAID on fenari was broken, mirror sdb1 was failed. Re-added sdb1. If the error is persistent, sdb needs to be replaced.
19:19 JeLuF: cron.{hourly,daily,...} on hume was not running due to a broken record in /etc/crontab, so logrotate was not running
12:58 logmsgbot: ariel synchronized php-1.17/includes/HistoryBlob.php 'check for existence of mhash function, not extension (need for lucid php build) (take 2, after svn up :-P)'
12:51 logmsgbot: ariel synchronized php-1.17/includes/HistoryBlob.php 'check for existence of mhash function, not extension (need for lucid php build)'

June 11

07:37 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Move section disabling minordefault out of the if($wmgUseUsabilityInitiative) conditional and document it'
07:12 RoanKattouw: Running script to fix preferences of users that can't turn off minordefault on enwiki. See bug 24313 comment 55. Script is in maintenance/fixMinorPrefs.php

June 10

21:26 binasher: mobile2 goes natty
18:08 Ryan_Lane: added ip and dns entry for internproxy.wikimedia.org in the sandbox vlan
17:28 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'See whether this fixes bug 29314'
15:21 RobH: copying old data from ms4 for solaris specific info. do not disrupt ms4.
02:24 RobH: ms4 repaired and back online, ready for reinstall per ticket 885
01:15 Ryan_Lane: starting services on virt1, since iptables rules are in

June 9

23:54 binasher: upgrading mobile2 to maverick (pre natty)
23:04 Ryan_Lane: added preilly account as a restricted user. Added him to mobile1-5
22:30 binasher: upgrading mobile3 to natty ice
21:38 binasher: upgrading mobile3 to maverick (then natty)
20:02 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 89789:
20:02 RoanKattouw: Running scap to deploy ArticleFeedback changes
18:59 binasher: mobile5 natty
18:42 binasher: mobile5 upgrade to maverick on the way to natty
15:28 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Raise account creation throttle for Lennart's IP on testwiki'
15:10 RobH: dns updated, all nameservers dig correctly.
15:10 RobH: pushing dns updates to add mgmt of all eqiad devices
01:22 Ryan_Lane: shutting off services on virt1 (including puppet) until I get a chance to firewall them properly

June 8

22:35 Ryan_Lane: rebooting virt1
21:58 hcatlin: deploying to mobile cluster
21:34 Ryan_Lane: changing controller.labs.wikimedia.org to a CNAME for virt1.wikimedia.org
21:27 mark: Started ircd on ekrem
21:20 mark: Moved routing of vlan 101 from csw1-sdtpa to csw5-pmtpa
21:06 binasher: changing pybal healthcheck url for mobile
20:47 Ryan_Lane: rebooting controller.labs.wikimedia.org
20:44 Ryan_Lane: restarting pdns on ns0, it's locked up
20:33 Ryan_Lane: make that deleting virt1.pmtpa.wmnet, and changing virt1.mgmt.pmtpa.wmnet to controller.labs.mgmt.pmtpa.wmnet
20:32 Ryan_Lane: changing virt1.pmtpa.wmnet to controller.labs.pmtpa.wmnet
20:27 Ryan_Lane: changing ip of controller.labs.wikimedia.org
20:17 mark: Rebooting ekrem
20:16 mark: Changed ekrem's ip
20:10 mark: Added 2nd ip to ekrem (208.80.152.178)
19:56 binasher: upgrade mobile4 to natty
19:53 mark: Shutdown ekrem's switch port and null routed the ip for a while
19:53 Ryan_Lane: deleting vmdev1 and vmdev2, and moving the controller.labs.wikimedia.org entry into the public services subnet
19:45 binasher: upgrading mobile4 to maverick on its way to natty
19:31 Ryan_Lane: reclaiming sandbox ips for owa.tesla, owa1.tesla, owa2.tesla, lvs1.tesla, and grid.tesla
18:10 Ryan_Lane: depooling mobile5
18:05 Ryan_Lane: changing weight for mobile5
17:42 Ryan_Lane: upping the weight of mobile5 to 24
17:27 Ryan_Lane: upping weight of mobile5 to 16
17:18 Ryan_Lane: depooling mobile4
17:16 Ryan_Lane: reducing weight of mobile4/5 to 8
17:13 Ryan_Lane: repooling mobile4/5 with a weight of 16
16:29 ^demon|away: irc feeds are down, somebody who has access should investigate.
15:39 mark: Rebooting sq71 to remove old temp eqiad tunnel config
14:49 mark: Setup PIM-SM on eqiad core routers and joined the pmtpa domain
14:49 mark: Changed pmtpa PIM-DM to PIM-SM
11:31 mark: Added A records for cr1-eqiad and cr2-eqiad loopback interfaces
10:47 mark: Set local-pref value for private peers on cr1-eqiad and cr2-eqiad
10:43 mark: Removed cr1-eqiad:ae1.666 configuration to remove the temporary tunneled link
10:39 mark: Configured OSPF and OSPFv3 on csw1-sdtpa:e16/1 and cr2-eqiad:xe-5/2/1
10:39 mark: Assigned IP addresses to csw1-sdtpa:e16/1 and cr2-eqiad:xe-5/2/1 to bring up the second 10G wave
00:37 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling CustomUserSignup on testwiki'

June 7

23:51 Ryan_Lane: rebooting virt1/2
23:21 Ryan_Lane: rebooting virt1
23:21 Ryan_Lane: moving nova-network to virt2, as it was causing issues accessing the public IP on virt1
23:12 Ryan_Lane: rebooting virt1
22:33 Ryan_Lane: removed nova-network from virt2-4, since multiple network nodes isn't supported. Added to virt1. Rebooting virt2-4.
22:11 hcatlin: deploying to mobile cluster
21:55 Ryan_Lane: restarting opendj on nfs2
19:56 logmsgbot: tfinc synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling collections extension on wikinews'
19:39 andrewS: ms4 ticket 729 updated, system is in testing
17:46 Ryan_Lane: rebooting virt1 to enable virtualization in the bios
17:42 Ryan_Lane: rebooting virt2 to enable virtualization in the bios
17:39 Ryan_Lane: rebooting virt3 to enable virtualization in the bios
17:35 Ryan_Lane: rebooting virt4 to enable virtualization in the bios
17:01 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Disabling Tidy on collabwiki per bug 29295'
16:50 mark: Adjusted BGP filters on cr1-eqiad to allow advertisement of pmtpa aggregates to BGP peers
16:29 mark: Removed static routes from cr1-eqiad and cr2-eqiad
16:18 mark: Setup full mesh of iBGP sessions between csw5-pmtpa, csw1-sdtpa, cr1-eqiad, cr2-eqiad (separate v4/v6 sessions)
15:17 mark: Removed IPv6-only iBGP session between cr1-eqiad and cr2-eqiad and converted the v4 iBGP session into multi-address-family
14:22 mark: Setup OSPFv3 between csw1-sdtpa and csw5-pmtpa
14:08 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28292'
14:04 logmsgbot: reedy synchronized php-1.17/includes/api/ApiPageSet.php 'r89646'
13:58 mark: Setup OSPF and OSPFv3 on the first 10G wave circuit between csw1-sdtpa and cr1-eqiad
13:39 RobH: updaed puppet giving andrew shields basic restricted shell access to fenari
11:08 mark: Setup quick log rotation of ipv6and4 on maerlant
04:51 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Let normal users test/signoff on revisions, just like we let them post comments. Restricting it to "coders" kind of defeats the purpose'

June 6

20:22 logmsgbot: hashar: ci.tesla: cleaned up backlog and /tmp (I really need to fix this)
17:00 logmsgbot: hashar: ran namespaceDupes.php frwikiversity --fix (bug 29015)
15:45 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/IncubatorTest.php 'r89571'
15:45 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialViewUserLang.php 'r89571'
14:54 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialRandomByTest.php 'r89565'
12:23 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/IncubatorTest.php 'r89561'
11:41 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialViewUserLang.php 'Attempted fix for fatal error'
11:36 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikimediaIncubator extension on incubatorwiki'
11:28 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add incubator loading code'
11:24 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add $wmgUseIncubator, set to false'
11:14 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 89556:
11:13 RoanKattouw: Running scap to deploy WikimediaIncubator extension (code push only, not yet enabled anywhere)
09:46 logmsgbot: catrope synchronized php-1.17/includes/json/Services_JSON.php 'r89555'
06:18 Ryan_Lane: rebooted virt2/3, so that nova-network would reconfigure itself properly
05:26 Ryan_Lane: depooled mobile4/5

June 5

23:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29269 Enable wgRawHtml on collabwiki - it is locked down wiki'
18:06 logmsgbot: reedy synchronized php-1.17/includes/api/ApiPageSet.php 'r89514 for bug 25734'
15:32 logmsgbot: reedy synchronized php-1.17/wmf-config/codereview.php 'Reduce semantic autodefer regex'
09:18 mark: killed old stray lsearchd process, started new one on search7 (search pool 3)

June 4

11:17 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29270'

June 3

22:44 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php
22:43 logmsgbot: pdhanda synchronized closed.dblist
22:10 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29007. Permission changed per comment 2'
21:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29007. Added autopatrol to fawiki'
21:34 Ryan_Lane: upping weights of mobile4/5 to 24
21:28 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r89444, r89445'
21:28 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r89444, r89445'
21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.css 'r89444, r89445'
21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r89444, r89445'
21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r89444, r89445'
20:43 Ryan_Lane: added temporary dns entry controller.labs.wikimedia.org until we figure out a proper DNS scheme for labs
19:43 Ryan_Lane: repooling mobile4/5 with a weight of 8
19:09 notpeter: pushing mible 4 and 5 into pool
19:00 Ryan_Lane: reinstalling opendj on virt1 with a new domain; using labs instead of tesla
17:59 awjr: killing eternally running query on civicrm db on db9
17:28 pdhanda: running maintenance/namespaceDupes.php on frwikiversity

June 2

20:00 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'Live hack to disable join CTA'
19:03 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump AFT tracking version to 8'
19:02 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r89355'
19:02 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.startup.js 'r89355'
19:01 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.css

June 1

21:46 logmsgbot: reedy synchronized php-1.17/includes/User.php 'bug 27514 Make unblockself available as a global permission'
21:24 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 26319 Enable collection on fawiki. Testing on rtl'
21:13 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 27323 WMF Logo/Button not served via Bits'
19:07 Ryan_Lane: switching test.wikipedia.org back to text cname, since it is using a fake cert. (testing was successful)
18:47 Ryan_Lane: changing cname for test.wikipedia.org to wikipedia-lb, to test https
18:41 Ryan_Lane: changing scheduler for https services from wrr to sh on pmtpa lvs servers. restarting pybal on lvs1
18:38 Ryan_Lane: make that ns1
18:38 Ryan_Lane: restarting pdns on ns2, it locked up after authdns-update
18:36 Ryan_Lane: adding wikipedia-lb geodns entry
18:30 Ryan_Lane: restarting pybal on amslvs1
18:29 Ryan_Lane: adding wikipedia-lb service to amslvs1
16:11 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29228'
15:43 logmsgbot: reedy synchronized php-1.17/extensions/CentralAuth/SpecialCentralAuth.php 'bug 28767'
15:40 logmsgbot: reedy synchronized php-1.17/extensions/CentralAuth/SpecialCentralAuth.php 'bug 28767'
14:39 Reedy: Poking at CentralAuth (bug 28767) on testwiki. If things get more broken, it's probably me
09:32 apergos: there's a hung tcpdump on amssq37 tht won't die. it doesn't seem to be doing much but it won't die either (that was me looking at purge issues)
08:18 thedj: Cache invalidations issues continue, but squid source unknown atm.

May 31

23:28 logmsgbot: asher synchronized php-1.17/wmf-config/db.php 'removing db7 from rotation'
23:19 Ryan_Lane: restarting pybal on lvs1
23:07 Ryan_Lane: adding forward entry for wikipedia-lb, for https testing
22:44 Ryan_Lane: restarting pdns on ns0, it's locked up
22:43 Ryan_Lane: changing cname for wikimania2005 to wikimedia-lb, for testing https
22:37 Ryan_Lane: adding geodns entry for wikimedia-lb
22:26 Ryan_Lane: adding wikimedia-lb forward entry for esams
21:56 Ryan_Lane: switching wikimania2005 back to text
21:45 Ryan_Lane: changing cname of wikimania2005 from text to wikimedia-lb to test https
20:51 hashar: wikitech 'ci.tesla: setup a cron for user ci to delete /tmp/mwParser* directories'
20:26 mark: Extended LVS prefix lists on csw1-esams and csw2-esams with 91.198.174.224/28
20:25 Ryan_Lane: restarting pybal on lvs1
19:52 Ryan_Lane: updating yvon and gurvin in dns, moving them to the squid-lvs address range
19:40 Ryan_Lane: adding wikimedia https service to pybal, and restarting pybal
18:47 mark: Disabled (redundant) PIM on csw5-pmtpa:ve3, so only csw1-sdtpa is running PIM now
18:20 Ryan_Lane: and restarting apache on it
18:20 Ryan_Lane: running sync common on srv186
16:07 logmsgbot: hashar: ci.tesla: was actually out of inodes on /. /tmp filled with mwParser-...-images directories
16:02 logmsgbot: hashar: ci.tesla: restarted cruisecontrol (no space left on device) error. It still has space though!

May 29

21:19 logmsgbot: demon synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'r89136'
19:17 logmsgbot: hashar: ci.tesla installed imagemagick php-gd php5-gd
18:42 logmsgbot: hashar: resumed continuous integration on ci.tesla
18:37 logmsgbot: hashar: ci.tesla out of disk, cleaning up /dev/sda1 wich is / (a good idea would be to use separate partitions
17:12 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29119 Allow nlwiki crats to remove rollbacker group'
05:57 Ryan_Lane: restarted pdns on dobson, it wasn't responding to requests
03:07 Ryan_Lane: killed some long running civi queries on db9 that have been running for over 200,000 seconds.
02:54 mark: Killed oprofile daemon and unloaded oprofile module on db32
02:38 mark: Restarted squid-frontend on sq76

May 28

19:41 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'Set wgCodeReviewMaxDiffPaths = 30'
17:52 Andrew: Loaded board vote configuration from extensions/SecurePoll/cli/wm-setup/bv2011-*-final3-jump.xml
17:48 logmsgbot: andrew synchronized php-1.17/wmf-config/CommonSettings.php 'Hooks for SecurePoll'
15:01 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29119'
08:15 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/includes/user/Auth.php 'deploy r89026'
07:49 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 89008:
07:48 Andrew: Running scap to deploy SecurePoll updates

May 27

22:17 logmsgbot: demon synchronized closed.dblist 'bug 29064: cleanup wikimania wiki closures (2005-2010)'
22:17 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29064: cleanup wikimania wiki closures (2005-2010)'
22:00 logmsgbot: catrope synchronized php-1.17/includes/GlobalFunctions.php 'Remove wfMsgExt profiling again'
21:53 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Prospective fix for the UW bug'
21:38 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'More profiling'
21:35 logmsgbot: catrope synchronized php-1.17/includes/GlobalFunctions.php 'Add profiling points in wfMsgExt()'
21:29 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Add more profiling'
21:15 logmsgbot: catrope synchronized php-1.17/StartProfiler.php 'Live hack for debugging UploadWizard RL weirdness'
21:13 logmsgbot: catrope synchronized php-1.17/load.php 'Revert live hack'
21:04 logmsgbot: catrope synchronized php-1.17/load.php 'Add live hack for profiling'
20:48 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29129. Added the new namespace to be search by default for metawiki.'
20:30 logmsgbot: pdhanda synchronized php/wmf-config/InitialiseSettings.php
12:10 Andrew: Installing SecurePoll DB tables on wikis in ~/centralauth_wikis_missing_securepoll
07:57 apergos: the images Rl2-quality-full-003.jpg and Picture_602.11.85.5.jpg have been removed from our image server permanently, office action (legal request).
06:37 apergos: thanks tim
06:37 apergos: restarted irc bot on ekrem
06:09 apergos: er... on ekrem :-P
06:09 apergos: restarted irc daemon, hopefully properly
06:02 apergos: attempting to reboot ekrem (was not pingable)

May 26

23:20 notpeter: powercycling knsq5
21:34 Ryan_Lane: adding en.prototype, de.prototype and test.prototype cnames for prototype.wikimedia.org
18:58 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Raise AFT tracking percentage from 2% to 10% (restores pre-ramp-up volume) and put all users in the show-expertise bucket. Also bump tracking version to 7'
18:54 Ryan_Lane: adding wikimedialbsecure service to amslvs1, for testing https
18:52 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r88909'
18:47 Ryan_Lane: adding wikimedia-lb.esams address to ams and kn text squids, also adding lvs configuration for this address
18:31 Ryan_Lane: removing mediawiki-lb.esams and foundation-lb.esams reverse addresses, they were assigned to the same ips as text.esams and bits.esams respectively
17:48 Ryan_Lane: adding forward entry for wikimedia-lb.pmtpa.wikimedia.org
15:34 mark: Moved AS13030 connection configuration from port 2/11 to port 2/11 on br1-knams

May 25

22:39 awjr: svn up'd on civicrm.wikimedia.org to r189 - deploying further optimizations to contribution searching
21:22 Ryan_Lane: adding wikimedia-lb svc address to squids
21:22 Ryan_Lane: adding wikimedialbsecure service to lvs2, for testing https access to wikimedia.org sites.
21:21 Ryan_Lane: modified pybal configuration to add wikimedia-lb ip to text backend
19:52 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'r88825'
18:32 Ryan_Lane: turning off the selenium grid virtual machines on tesla
17:04 RobH: updated bugzilla per rt 764 and 768. whine feature should work now
16:44 awjr: deployed patch to civicrm.wikimedia.org to optimize contribution search query (r185, wikimedia repo)
14:57 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Don't allow users to create accounts on closed wikis either'
14:54 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Cleanup wikimania200[58] entries: bug 29064'
14:53 logmsgbot: demon synchronized closed.dblist 'Add wikimania200[58] to closed.dblist: bug 29064'
14:45 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29064: close wikimania2009wiki'
14:43 logmsgbot: demon synchronized closed.dblist 'bug 29064: close wikimania2009wiki'
12:26 Tim: marked the deletion jobs in OTRS as "invalid" to avoid confusing error messages
12:25 Tim: re-enabled anti-deletion security patch in OTRS, disabled "temporarily" by Fred on June 4, 2009
11:58 domas: rebooting db9, looks unhealthy
11:06 apergos: so that was me on srv225: stopped apache, waited for puppet restart: it now syncs, then does the restart. oh btw srv225 isn't in the pybal apache conf file, weirdly... anyone know why?
09:15 apergos: restarted front and backend squids on sq75
08:47 apergos: restarted front and back end squids on sq81, front end was 20gb and back end was out to lunch
08:35 apergos: ekrem's root partition was full; cleared out some older logs from /var/log/apache2, compressed the most recently rotated (it had failed from disk full), reloaded puppet
07:20 Tim: set read_only=1 on db10, it's a slave so it should be read only. Nagios says it was broken for 6 months.
07:07 logmsgbot: midom synchronized php-1.17/wmf-config/db.php
05:09 Tim: restarted nagios IRC bot
01:05 Tim: created an account for myself in watchmouse
00:48 Andrew: Running populateBv2011EditCount.php in a screen on hume
00:32 Ryan_Lane: pruning binlogs on db9

May 24

23:12 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29129. Added a namespace alias for metawiki.'
20:36 notpeter_: resyncing drbd nfs1 to nfs2
20:28 RobH: mobile4 awaiting final configurations for service, mobile5 installation done, not yet run puppet updates
20:20 RobH: mobile4 is running puppet updates, mobile5 is mid-install
19:46 RobH: mobile4 & mobile5 racked and accessible for mgmt. installing OS on mobile4 now
18:00 Ryan_Lane: added new reverse zone for svc ips in esams; added entries for bits.svc.esams.wmnet and upload.svc.esams.wmnet
17:31 Ryan_Lane: removing bits-lb and upload-lb public ip enties, and adding bits.svc.pmtpa.wmnet and upload.svc.pmtpa.wmnet entries for https testing
16:46 mark: Upgraded cr2-eqiad to JUNOS 10.4R4.5
16:44 mark: PIM configuration was missing on some virtual interfaces on csw1-sdtpa, which broke purging to esams. Put back in place
16:09 mark: Reloading re1.cr2-eqiad
16:01 mark: Fixed tunnel hack between pmtpa and eqiad
15:01 RobH: db20 networking restarted, back online
14:58 RobH: Miguel is rebooting db18
13:55 mark: reloading csw5
13:52 domas: ns2 has been down for 3 hours, will try restarting pdns
13:47 mark: reloading asw-a4
13:45 mark: reloading asw-d3
13:39 domas: db18/20/28 are still down, may need power cycle. Took them out of rotation.
13:37 RobH: restarted networking on db33-39.
13:30 mark: restarted networking on db30-32.
13:25 domas: DB server issue appears to be a linux kernel issue, forcedeth-related, can be fixed with /etc/init.d/networking restart. Doing so on db10-30, asking mark to do the rest.
13:23 mark: restarted slot 3 since that's where the problematic database servers are
13:12 mark: csw1-sdtpa back up, but a whole lot of database servers are unreachable
13:07 mark: upgraded and reloaded csw1-sdtpa
12:05 mark: Upgraded bootrom and software on asw-d1-sdtpa and asw-a5-sdtpa
10:53 mark: Copied existing 2.7.0.2 image to secondary flash, and installed 2.7.0.3a image on primary flash on csw5-pmtpa
10:42 mark: Copied existing 2.6.0 image to secondary flash, and installed 2.7.0.3a image on primary flash on csw1-sdtpa
04:20 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r88707'
04:03 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
04:03 Tim: enabling GoogleNewsSitemap on enwikinews

May 23

21:18 Ryan_Lane: added upload-lb.pmtpa.wm.o and bits-lb.pmtpa.wm.o addresses to reverse dns for https preparation
19:32 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29003'
18:45 apergos: reloading puppet on all hosts, with a minute sleep in between, running in screen on fenari as root. needed for enabling pluginsync on clients
18:20 Ryan_Lane: moved ipv6 proxy from iris to maerlant
17:35 RobH: db20 back online, needs mysql setup for replication on another cluster. created rt#836
17:29 RobH: db20 was offline from earlier hardware issues that appear to have been resolved. it is in no current deployment group. booting it online to see if it requires reinstallation
14:31 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/SitemapFeed.php
14:31 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/FeedSMItem.php 'r88640'
14:30 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r88640'
14:23 hcatlin: Deploying code update to Mobile wikipedia
05:08 Tim: in watchmouse: updated subversion monitoring, was broken since May 7

May 22

17:47 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeStatusChangeListView.php 'r88585'
17:46 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeCommentsListView.php 'r88585'
00:09 Reedy: Commons is back up and editable

May 21

23:38 Reedy: Intermittant MySQL errors occuring on hits to 10.0.6.41, it is being looked into
23:35 Ryan_Lane: powercycling db31, it's dead
10:27 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '25963 - Enable of Recent changes patrol for Serbian Wikisource (sr.wikisource.org)'

May 20

22:30 Reedy: StewardWiki security issues fixed
22:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Stewardwiki'
21:23 awjr: running contribution auditing scripts on civicrm.wikimedia.org
21:21 awjr: enabling log audit module on civicrm.wikimedia.org
21:18 awjr: svn up'ing to r183 of wikimedia branch on production instance of civicrm on grosley
20:57 logmsgbot: pdhanda ran sync-common-all 'Bug 28717. Group changes and flagged revs enabled for bnwiki'
20:50 Ryan_Lane: maerlant too
20:50 Ryan_Lane: rebooting maerlent
20:13 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28773. more rights and namespaces for stewardwiki'
17:16 RobH: updated dns for new mobile servers mgmt
17:12 RoanKattouw: Applying index changes on click_tracking_user_properties from r88462 to the cluster. Using /home/catrope/patch-click_tracking_user_properties-index.sql
15:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/abusefilter.php 'Bug 28461. Custom abuse filter settings for be_x_oldwiki'
15:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28461. Custom abuse filter settings for be_x_oldwiki'
14:41 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28614. Added suppressredirect to autopatrolled on svwikisource'
14:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28576. Changed upload settings for huwiki'
03:12 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Touch config'
03:12 logmsgbot: demon synchronized closed.dblist 'bug 29044: closure of tkwikibooks'
02:48 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29058: import sources for arwikinews'
02:04 logmsgbot: LocalisationUpdate failed
00:06 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29057'

May 19

22:26 ^demon: add new 'tools' group to public svn authz
21:02 logmsgbot: hashar: cruisecontrol: tests should now run in roughly 1min30! Still need attention though
20:16 logmsgbot: hashar: cruisecontrol: archived build logs for 2010 and 2011 from 01 till 04
18:48 logmsgbot: hashar: relaunched CruiseControl service on ci
02:01 logmsgbot: LocalisationUpdate failed

May 18

19:44 Ryan_Lane: restarting squid processes on sq71
18:11 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Turning GNSM back off for now -- apache spike'
17:50 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Deploying GNSM for enwikinews'
17:29 logmsgbot: demon synchronizing Wikimedia installation... Revision: 88303:
16:06 ^demon: didn't work, had errors with ArticleAssessmentPilot, FlaggedRevs, LQT_alpha, and MessagesTp. Plus lots of rsync permission errors. Saved output to /home/demon/logs/l10nupdatefail.log
15:53 ^demon: doing manual run of l10nupdate
15:24 logmsgbot: demon synchronized php-1.17/wmf-config/extension-list
02:05 logmsgbot: LocalisationUpdate failed

May 17

18:33 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'fix capitalisation fail'
18:30 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29017 again doesn't seem to have pushed out'
13:24 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enable CLDR'
13:23 logmsgbot: catrope synchronized php-1.17/languages/Language.php 'r88303'
13:09 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable CLDR because of fatal'
13:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable CLDR extension on all wikis'
12:52 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 88299:
12:51 RoanKattouw: Running scap to deploy CLDR extension. Only enabled on test for now, will go site-wide in a minute
11:57 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesEn.php 'r88295'
11:34 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/resources/jquery/jquery.validate.wmCommonsBlacklist.js 'r88293'
09:52 Tim: creating an account on wikitech for priyanka
09:51 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable UploadWizard on Commons, adding note about how to switch it back off if it explodes'
09:39 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'Fix fatal in live hack'
09:37 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'Live hack to debug UploadWizard issue'
09:17 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enabling UploadWizard on test to hopefully reproduce ResourceLoader issue'
05:21 mark: Reduced passenger pool size to 200 on all mobile servers
04:46 mark: Depooled mobile2 and rebalanced LVS weights on m.wikipedia

May 16

21:56 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28504 - Need to remove all tokipona settings from InitialiseSettings.php'
21:52 logmsgbot: hashar: cleaning up settings files for tokipona stuff (bug 28504)
19:03 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29017'
09:37 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account throttle for el wiki for workshop (maybe I can talk the community into upping the threshhold generally??)'

May 15

21:13 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Closing 4 wikis: thwikinews (bug 28341), huwikinews (bug 28342), cowikibooks (bug 28644), astwikiquote (bug 28964)'
21:13 logmsgbot: demon synchronized closed.dblist 'Closing 4 wikis: thwikinews (bug 28341), huwikinews (bug 28342), cowikibooks (bug 28644), astwikiquote (bug 28964)'
21:02 logmsgbot: demon synchronized php-1.17/includes/DefaultSettings.php 'r88203'
21:01 logmsgbot: demon synchronized php-1.17/includes/User.php 'r88203'
12:27 logmsgbot: pdhanda ran sync-common-all
12:12 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
11:48 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28987'
11:32 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28727 - please create Portal namespace on am.wikipedia'
11:05 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '25172 - Enable DynamicPageList extension on it.wikisource'
10:35 logmsgbot: hashar: had to manually delete a row from the ptwiki.langlink table (ll_from: 162503, ll_lang: zh-classic, ll_title: Template:Link A (bug 28805)
10:27 logmsgbot: hashar: running update ignore query on ptwiki for bug28805
10:23 logmsgbot: hashar: batch for bug 28805 finished.
10:17 logmsgbot: hashar: running batch queries on all wikis for bug 28805
08:44 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28609 - Disable uploads for eu.wikipedia.org'
01:14 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28914 - Adding 'ipblock-exempt' userright to the Bots usergroup in enwiki'
00:52 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28959 - si.wikipedia $wgSitename'

May 14

22:46 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27434'
22:08 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'wgMetanamespace for amwiki (bug 28727)'
22:05 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28727 - please create Portal namespace on am.wikipedia'
17:11 mark: Fixed IPv6 routing on csw5-pmtpa
16:34 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 28153'
16:30 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 28153'
16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28141'
16:14 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28141'
15:57 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27078'
15:52 RobH: added dns entry for wikimania2012.w.o
15:23 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28773'
15:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php
14:50 awjr: cleaned out hudson job files for 'Donations queue consume' on grosley and archived them to storage3
14:22 mark: Allocated service IPs for all projects for HTTPS in DNS
14:06 logmsgbot: tstarling synchronized php-1.17/lib/geshi
13:53 mark: Upgraded kernel on streber, rebooting it
13:41 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
13:31 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
13:17 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
13:13 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
13:11 logmsgbot: reedy synchronized php-1.17/lib/geshinew 'Test pushing symlink'
12:35 Ryan_Lane: shutting down owa1-3.tesla.usability.wikimedia.org
12:24 Ryan_Lane: adding maerlant.esams.wikimedia.org as a https/ipv6 proxy node
10:06 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled enotif for user talk pages on all wikis'
09:54 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28895'
09:43 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28612. Added the autopatrolled group to elwiktionary'
09:28 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Fill in template for bug 28727 (Portal namespace creation for am.wikipedia)'
09:16 logmsgbot: tstarling synchronized php-1.17/includes/MessageCache.php 'removed message stats patch'
08:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28479. Added the patroller group to plwikiquote'

May 13

17:33 Ryan_Lane: yvon and gurvin configured as https servers
17:26 RobH: gurvin os installed pupppet not run
17:14 RobH: moved gurvin vlan form 2 to 101
17:06 RoanKattouw: Uncommented srv196 in the node list and ran sync-common on it. It was giving me colorless diffs on CodeReview
17:03 RobH: yvon reinstalled, wont run puppet initial run due to error from new https service entry
16:51 RobH: updating dns for hosts that didnt have public ips (gurvin, some corrections for yvon)
16:28 logmsgbot: catrope ran sync-common-all
16:22 RoanKattouw: Running sync-common-all because sync-docroot didn't work
16:12 RoanKattouw: Syncing docroot
16:12 RoanKattouw: Adding xml/api directory to mediawiki.org's docroot, with an index.html similar to the one in the export-0.4 directory
16:02 logmsgbot: reedy synchronized php-1.17/RELEASE-NOTES
12:14 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28347'
10:48 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 26037'
10:31 Tim: deployed a patch to gather i18n usage stats for siebrand

May 11

22:09 mark: Reloaded csw5-pmtpa on existing code base
01:16 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Temporarily disabling UploadWizard per Neil K while a bug is being worked out.'
01:15 logmsgbot: demon synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'r87865'
01:14 logmsgbot: demon synchronized php-1.17/extensions/UploadWizard/UploadWizard.i18n.php 'r87865'
00:10 notpeter: puppet back to normal on all boxes that are up except for knsq5.

May 10

23:36 Ryan_Lane: fixing torrus. why oh why must you corrupt yourself constantly, torrus?
23:06 notpeter: distributing new puppet certs to all boxes managed by puppet
22:51 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'ED 1.17'
21:34 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling Article Feedback dashboard on enwiki'
21:30 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.php 'r87844'
21:29 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r87844'
21:28 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87844'
21:28 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r87844'
21:02 Ryan_Lane: powercycling srv281 since it's dead
21:01 Ryan_Lane: powercycling srv247 since it's dead
20:56 Ryan_Lane: mobile has stablized, moving weights back to normal
20:50 Ryan_Lane: lowering weight on mobile1 and mobile2
20:45 Ryan_Lane: restarted apache on mobile2.
19:16 mark: Moved all inter-subnet routing from csw5-pmtpa to csw1-pmtpa
19:16 mark: Reconfigured pybal on lvs3 to BGP connect to csw1-sdtpa instead of csw5-pmtpa, and restarted it
17:53 mark: Moved vlan 2 routing from csw5-pmtpa:ve2 to csw1-pmtpa:ve2
16:27 apergos: or at least srv254.
16:25 apergos: nagios is whining about some services that are actually in operation (eg srv250)
16:07 Ryan_Lane: or not, mgmt interface isn't letting me in
16:06 Ryan_Lane: powercycling srv250
13:56 Tim: removed myself from SMS notification in nagios, since the configuration is so broken that it sends me a text every 10 minutes day or night
13:49 RobH: nagios sms pages seem to be delayed from the actual service outage by a long, long delay
13:10 Tim: did the above in both puppet and nagios directly since puppet seems to be still broken
13:07 Tim: increased the number of retries of payments.wikimedia.org before sending an SMS from 1 to 20
02:02 notpeter: accidentally deleted sockpuppet:/var/lib/puppet/ssl am regnerating keys and will begin signing soon

May 9

23:36 notpeter: so annoying. doubling puppet freshness check timing to 4 hours.
22:01 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabling dashboard on enwiki pending fixes to data collection script'
21:20 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87774:
21:20 RoanKattouw: Running scap to deploy UploadWizard changes
20:54 RoanKattouw: Running extensions/ArticleFeedback/populateAFStatistics.php on enwiki
20:49 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87770'
20:49 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r87770'
20:30 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling ArticleFeedback dashboard on testwiki'
20:28 RoanKattouw: Running populateAFStatistics.php on testwiki
19:39 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Dummy sync to push config change through in ResourceLoader'
19:39 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable AFT click tracking at 2%, and bump tracking version'
19:32 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Force config changes to update in startup module'
19:21 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to the target value of 2.7%'
19:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 2%'
18:41 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 1%'
18:39 RobH: upgraded Reedy's cluster access to allow him to deploy and do shell requests, will take awhile to push out to the cluster
18:25 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 0.5% for real this time'
18:21 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump AFT tracking version, needed to actually change the click tracking percentage'
18:16 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28495'
18:14 Ryan_Lane: adding a shell account for asher (Asher Feldman), and adding him to roots
18:13 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Temporarily disable click tracking for AFT'
18:11 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bump AFT lottery odds to 1%'
18:10 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable AFT dashboard on enwiki'
18:09 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump ArticleFeedback tracking version'
18:01 RoanKattouw: Running schema change (adding index of aa_timestamp field on enwiki.article_feedback table) on s1 by directly running it on the master. Table size is ~275k rows
17:56 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wmgArticleFeedbackLotteryOdds to 0.005 on enwiki. This enables AFT on 0.5% of all enwiki main namespace pages'
17:44 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87742:
17:43 RoanKattouw: Running scap to deploy ArticleFeedback changes
16:49 RoanKattouw: Deploying ArticleFeedback changes to testwiki
14:21 apergos: restarting amssq39 squid back-end (memory hog)
13:41 apergos: restarting squid back-end on amssq40 (memory hog)
13:27 logmsgbot: catrope synchronized php-1.17/resources/mediawiki/mediawiki.js 'r87712'
13:27 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php 'r87712'
13:26 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderContext.php 'r87712'
13:26 RoanKattouw: Deploying r87712, which should fix the ResourceLoader breakage in IE
13:12 apergos: restarted squid back-end on amssq37 (hogging memory)
08:06 apergos: rebooting srv198 for testing
07:52 apergos: rebooting srv281 for testing

May 8

22:46 RobH: fixed deep linking issue for uploads from old techblog. merge did not copy, had to manually pull from singer and also add additional rewrite rule to apache via puppet
22:40 RobH: copied over techblog upload data, which upon further review was not copied by the merge plugin as expected
18:29 RobH: updated categories on blog to remove testblog url entry
18:20 RobH: updating dns to revert bad change on blog propogation, split wikimediafoundation template files to themselves, were soft linked to wiktionary.org
18:05 RobH: blog redirects are in place for projects, not sure why or what added them, as they do not belong. fixed hooper via puppet to handle blog.wikimediafoundation.org
15:46 RobH: added exim simple mail sender to puppet entry for hooper

May 7

20:31 RobH: hooper running back on puppet, with all config files being updated normally
18:11 RobH: tinkering with redirects on hooper. disabled puppet daemon while doing local apache config changes, once tested will push up to puppet and reenable it on hooper
18:08 RobH: blog dns updated enough for both myself and guillom to test basic blog function. all basic use seems fine. still testing more advanced use cases
16:58 RobH: updated dns for blog and techblog. may take an hour or so to propogate, cannot update new blog until it resolves for authors to it. (old blogs will slowly cycle out of dns)
16:40 RobH: blog migration moving along, all data migrated, basic rewrites and such are in place, final review and cleanup in progress before changing testblog.w.o to blog.w.o in database and updating DNS
16:03 RobH: testblog.w.o currently now running 'master' database for the wmf blog. online as testblog, can do quick db replace when its ready to go online
15:57 RobH: starting the copy of the main blog, any changes to blog.w.o after this time will be lost in the migration to the new host
15:53 RobH: updating blog software on hooper, installing all extensions before connecting to database copy of newsblog (not yet copied)
15:37 RobH: wiping out testblog on hooper and getting basic software installed for new blog deployment. database not yet copied, old blogs still 'primary'
12:55 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account throttle for el wiki for workshop'
09:20 rainman-sr: deleted some old logs on search11 to free up space

May 6

22:46 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28773. Modified wgAddGroups and enabled subpages for stewardwiki'
20:39 Ryan_Lane: added apaches::service to imagescalers class in puppet to ensure that sync-common will run before apache is started on the imagescalers
18:10 awjr: enabling drupal/civicrm logging on civicrm.wikimedia.org (forgot to turn this back on post-upgrade)
17:41 notpeter: killing the shit out of puppet to clear its cache.
17:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28791. Fixed logo for piwiki'
17:19 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28689 - enabled NewUserMessage on thwiki'
13:44 RobH: installed and activated wordpress importer on testblog
08:58 RoanKattouw: Ran sync-common on srv281 and restarted Apache. It was complaining about missing directories in docroot
08:55 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87486:
08:55 RoanKattouw: Uncommenting srv281 in /etc/dsh/group/mediawiki-installation and scapping again
08:53 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87486:
08:52 RoanKattouw: Running scap because srv281 is out of sync. I thought we had measures to prevent out-of-sync servers from being pooled?
05:03 apergos: powercycled srv281 (again)
01:55 Tim: stopped xinetd on fenari, running svn up to add REL1_17 branch to extdist snapshot

May 5

17:32 notpeter: taking down nfs1 to upgrade to lucid and upgrade udplog
05:47 logmsgbot: tstarling synchronized php-1.17/img_auth.php 'r87486'
05:47 logmsgbot: tstarling synchronized php-1.17/includes/WebRequest.php 'r87486'
05:46 logmsgbot: tstarling synchronized php-1.17/includes/User.php 'r87486'

May 4

23:11 awjr: all CiviCRM-related cron and hudson jobs re-enabled
22:19 awjr: CiviCRM upgrade complete, putting civicrm.wikimedia.org back on-line
22:17 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r87462'
22:17 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap.php 'r87462'
22:16 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap.alias.php 'r87462'
21:33 awjr: performing CiviCRM upgrade (3.1.6->3.4.0) on grosley
21:24 awjr: upgrade to drupal 6.20 complete for civicrm.wikimedia.org
21:10 awjr: replacing /srv/org.wikimedia.civicrm dir on grosley with svn co from new deployment branch in wikimedia repo (/branches/deployment/fundraising-civicrm/d620c34)
20:51 awjr: disabling civicrm and drupal-related cron/hudson jobs
20:49 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87452:
20:43 logmsgbot: catrope synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'Fix GNSM, r87488'
20:37 awjr: backing up 'civicrm' database from db9
20:34 awjr: backing up 'drupal' database from db9
20:30 awjr: taking civicrm.wikimedia.org offline for drupal and civicrm upgrade
20:14 RoanKattouw: Deploying new UploadWizard code to testwiki
18:42 Ryan_Lane: created a restricted account for neilk
15:33 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php
14:29 hcatlin: deploying mobile language update

May 3

22:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/abusefilter.php 'Bug 28502 - custom config for thwiki'
22:37 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28502 - enabled abusefilter for thwiki'
21:21 Ryan_Lane: powercycling srv281
21:12 logmsgbot: catrope synchronized php-1.17/README 'Test sync to see which hosts are borked'
21:04 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28257. Added requested NamespaceAlias to orwiki'
20:32 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87355:
20:31 RoanKattouw: Running scap to deploy 1.17wmf1 merges (r87353)
16:24 awjr: re-upgrading dev.civicrm.wikimedia.org from civicrm 3.1.6 to 3.4.0
15:59 RobH: testblog ops work done, no other services impacted
15:13 RobH: working on testblog on hooper and db9, no other services should be affected
03:30 RobH: issue of singer network spiking and plummeting, along with swap death, at ~3am gmt
03:25 RobH: singer blocked on db9, flushed hosts to allow to reconnect
01:30 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r87311'
01:29 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'r87311'
00:44 tomaszf: killing long running CiviReport query on db9 due to watchdog being off for upgrade

May 2

20:02 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Adding Research namespace to meta, per bug #28742'
17:52 Ryan_Lane: powercycling srv281
17:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabled LabeledSectionTransclusion for mediawiki - requested by guillom'
17:28 RobH: fixing issue with bad serial assignment leading to bad network config on ps1-a1/2-eqiad, restarting both power strips
15:47 mark: Powercycled amssq35
04:21 logmsgbot: tstarling synchronized php-1.17/includes/Wiki.php 'r87235'
03:58 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
03:34 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php
03:31 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php
02:13 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27891 log'
02:00 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'r87234'
01:22 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'r87233'
01:21 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'r87233'
00:09 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'wgBlockDisablesLogin on checkuserwiki'

May 1

21:14 Ryan_Lane: ran flush host on db9 to bring blog.wikimedia.org back

April 30

11:38 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryAllUsers.php 'Attempt to fix auprop=blockinfo bug'
11:15 Ryan_Lane: (make that blog.wikimedia.org ;) )
11:14 Ryan_Lane: ran flush hosts on db9 to bring blogs.wikimedia.org back up

April 29

23:27 ^demon: all of that was to deploy GoogleNewsSitemap to testwiki for the enwikinews folks.
23:23 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php
23:22 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php
23:18 logmsgbot: demon synchronizing Wikimedia installation... Revision: 87149: r87149
19:43 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Fix config bug that caused tracking to fail for AFT, and bump tracking version'
19:18 mark: powercycled ersch
15:40 RobH: typo asw-b8
15:39 RobH: swapping msw1-eqiad fan tray with asw-a8-eqiad fan tray
05:58 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'adding srv170 after hw reset'
05:55 logmsgbot: midom synchronized php-1.17/wmf-config/db.php

April 28

23:46 Tim: doing hard reset of alsted, it's not sending ssh version strings and not allowing logins by serial
22:17 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabled option to allow blocked users to edit their talk pages on itwiki. See Bug 9073'
21:02 Ryan_Lane: created Ambassador-announce-l list
20:52 richcole1: replacing memory in srv281
20:49 RobH: srv281 shutdown for memory replacement
20:41 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable by default, option to enable blocked users to edit their own talk page'
20:34 richcole1: replacing bad drive in srv284
20:08 Ryan_Lane: powercycling srv281
20:03 RobH: ms4 is powered down, was already offline and unresponsive to console.
18:19 logmsgbot: demon synchronizing Wikimedia installation... Revision: 87091: r87090, r87091
18:02 mark: Restarted apache on srv235, probably corrupted apc cache
17:57 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enabling EmailCapture on enwiki'
17:54 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 87084:
17:47 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Preparing conf for ArticleFeedback with way less click tracking to prevent cluster breakage'
17:40 RobH: etherpad.proxy to 000-etherpad.proxy on hooper apache2 enabled, should fix
17:34 RobH: hooper vhost for testblog back online
17:17 RobH: rolled back testblog vhost from hooper
17:01 RobH: testblog enabled for initial setup on hooper without disrupting etherpad service (on either of its url redirects)
16:56 RobH: updating etherpad.proxy apache config to include eiximenis serveralias and enabling the testblog vhost
16:48 Reedy: Manually pinged CR update script, had got stuck - 2 revisions. Let me know if it does it again
16:21 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87085:
16:21 RoanKattouw: Rolling back ArticleFeedback deployment
16:11 RoanKattouw: API cluster was briefly dead just now after enabling EmailCapture, disabling it seems to be bringing it back
16:09 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable EmailCapture, maybe this will bring our API Apaches back'
16:04 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling EmailCapture on enwiki'
16:01 RoanKattouw: (that was for the ArticleFeedback deployment)
15:58 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 87083:
14:08 nelson: Upped the max_connections for rsyncd from 2 to 5 across the test cluster.
13:56 nelson: Upped the concurrency of swift-object-replicator from default of 1 to 5
13:38 mark: Restored I/O scheduler deadline for device /dev/sda on alsted
13:34 mark: Temporarily set I/O scheduler cfq for device /dev/sda on alsted
12:29 mark: powercycled harmon
01:00 awjr: upgraded dev.civicrm.wikimedia.org to CiviCRM 3.4.0

April 27

20:24 nimish_g: nimishg enabling CustomUserSignup extension on enwiki
20:20 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling'
01:44 logmsgbot: nimishg synchronized php-1.17/extensions/ClickTracking/modules/ext.UserBuckets.js 'setting cookie expiry to 90 days'
01:44 logmsgbot: nimishg synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php 'setting cookie expiry to 90 days'
01:43 logmsgbot: nimishg synchronized php-1.17/extensions/CustomUserSignup/CustomUserSignup.hooks.php 'setting cooky expiry to 90 days'
01:03 Tim: updated sudoers in puppet for r83000
00:25 Tim: reniced sshd on sockpuppet to -2 so that I can log in while catalog updates are running

April 26

23:40 logmsgbot: nimishg synchronizing Wikimedia installation... Revision: 86895:
23:33 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php
23:26 logmsgbot: root synchronizing Wikimedia installation... Revision: 86895:
23:15 Tim: disabled CustomUserSignup, was causing fatal error on mediawiki.org due to incorrect path
23:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php
23:13 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'turning'
23:00 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'turning'
22:55 logmsgbot: nimishg synchronized php-1.17/extensions/CustomUserSignup/CustomUserSignup.hooks.php 'r86994'
22:49 notpeter: stopping nagios for a wee sec and then restarting.
22:09 logmsgbot: nimishg synchronizing Wikimedia installation... Revision: 86895:
22:08 nimish_g: nimishg pushing CustomUserSignup code to cluster, running on mediawikiwiki only
21:28 logmsgbot: awjrichards synchronized php-1.17/includes/CategoryPage.php 'r86988'
20:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled CustomUserSignup extension for testwiki'
20:37 logmsgbot: pdhanda synchronized php-1.17/wmf-config/CommonSettings.php 'enabled CustomUserSignup extension for testwiki'
20:32 awjr: disabled wmf_premiums module on civicrm.wikimedia.org - it is currently not being used and potentially causes queue consumption to choke with certain contribution messages
19:51 logmsgbot: catrope ran sync-common-all
19:50 RoanKattouw: Running sync-common-all to deploy UploadWizard changse
17:52 pdhanda: Running maintenance/populateParentId.php on all wikis
08:21 Andrew: sync-common-all worked. scap still broken
08:21 logmsgbot: andrew ran sync-common-all
08:21 Andrew: trying sync-common-all
08:19 Andrew: syncs are broken, log littered with XXX: [sudo] password for andrew:
08:12 Andrew: re-scapping, typo in extension-list
08:12 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895:
08:11 Andrew: Scapping to enable DisableAccount extension
08:11 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895:
08:02 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895:
08:02 Andrew: running scap to deploy the code itself
08:01 Andrew: deploying DisableAccount extension to checkuserwiki, stewardwiki, arbcom_enwiki since the special page was removed without consulting Philippe
02:15 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding settings for checkuser and steward wikis'

April 25

23:33 Ryan_Lane: added python-mwclient to lucid repo
21:36 RobH: storage2 still offline, wont boot into os, but is remotely accessible
21:20 RobH: trying to fix storage2
20:16 notpeter: actually adding everyone on ops to watchmouse service... didn't know this had not already been done.
20:02 RobH: updated csw1 to removed labels and move to default vlan ports 11/12, 11/14, 11/19, & 11/21. old connection ports for dataset2, tridge, ms1, and ms5
19:53 RobH: the datacenter is looking awesome.
19:45 RobH: ms1 moved from temp network to permanent home, no downtime, responding fine
19:42 RobH: ms5 connection moved, no downtime, responds fine, less than 4 seconds
19:40 RobH: updated csw1-sdtpa 15/1,15/2 from vlan 105 to vlan 2, 15/3 and 15/4 from vlan 105 to 101
18:52 RobH: snapshot4 relocated to new home, ready for os install
18:42 RobH: db19 and db20 back online (not in services as they have other issues)
18:39 RobH: db19 and db20 powering back up
18:25 RobH: virt4 experienced an accidental reboot when rebalancing power in the rack, my fault, not the hardware
18:12 RobH: rack b2 power rebalanced
18:01 RobH: db19 set to slave, depooled in db.php, no other services evident, shutting down (mysql stopped cleanly)
18:00 RobH: db20 shutdown
18:00 RobH: didnt log that i setup ports 11/38-40 for db19, db20, and snapshot4 on csw1-sdtpa. tested out fine and all my major configuration changes on netowrk should be complete
17:56 RobH: ok, db20 and db19 are coming offline to relocate their rack location due to power distro issues
15:47 RobH: delay, not coming down yet, need more cables
15:46 RobH: db19 is coming down as well, it is depooled anyhow
15:46 RobH: db20 is coming down, ganglia aggregation for those hosts may be delayed until it is back online.
15:21 RobH: relocating snapshot4 into rack c2, it will be offline during this process
15:20 RobH: db43-db47 network setup, sites not down, yay me
15:10 RobH: being on csw1 makes robh nervous.
15:09 RobH: labeling and setting up ports on 11/33 through 11/37 on csw1-sdtpa for db43 through db47
14:47 RobH: fixed storage2 serial console (set it to higher rate, magically works, or it just fears me) and also confirmed its remote power control is functioning
14:42 RobH: stealing dataset1's known good scs connection to test storage2. dataset1 service will remain unaffected.

April 24

21:30 Ryan_Lane: restarting apache on mobile1
15:35 RobH: swapping bad disk in db30, hotswap, should be fine
14:36 RobH: swapping out the management switch in c1-sdtpa. msw-c1-sdtpa will be offline, so the mgmt interfaces of servers in that rack will be offline. all normal services will remain unaffected.

April 23

22:31 RobH: required even.
22:31 RobH: no drives display error leds, futher investigation requried
22:27 RobH: ms2 is having bad drive investigated. if we do this right, it wont go down. if we don't it will. is a slave es server.
22:00 RobH: singer returned to operation, blog, techblog, survey, and secure returned to normal operation
21:52 RobH: singer is once again coming back down for drive replacement. This will take offline blog.wikimedia.org, techblog.wikimedia.org, survey.wikimedia.org, and secure.wikipedia.org. Service will be returned as soon as possible.
21:19 RobH: singer back online, for awhile, will come back down for further repair shortly.
21:05 RobH: singer going down, blogs will be offline, so will secure, system will return to service as soon as possible
21:00 RobH: preparing to fix the dead drive in singer, this will offline secure, blog, techblog, and survey during the drive replacement process
19:49 mark: Upgrading mr1-pmtpa to junos 10.4R3.4
17:49 RobH: migrating searchidx1 & search1-search10 to new ports in same rack. moving one at a time and ensuring link lights between moves. (already tested with search10)
14:11 RobH: db19 is back online, seems to not have any mysql setup done.
14:02 RobH: restarting db19
14:02 RobH: arcconf checks out all drives on db19 are indeed working as rich found earlier
12:47 mark: Added (x121Address=1) condition to the LDAP query of the ldap_aliases router on mchenry's exim
00:32 hcatlin: Mobile: Deploying fix to an issue that kept the standard-style Main_Page from displaying on mobile
00:25 Ryan_Lane: restarting memcached on all of the mobile servers
00:23 Ryan_Lane: repooling mobile3, since mobile will die without it (fun!!)
00:17 Ryan_Lane: depooling mobile3
00:13 Ryan_Lane: restarting apache on mobile3
00:10 Ryan_Lane: puppet was broken on mobile1, reinstalled it

April 22

23:56 domas: detached gdb from srv193 apache, apparently it was used for something
23:14 notpeter: restarting nagios (again)wq
22:43 notpeter: restarting nagios
19:23 apergos: shot all stopped rsyncs on ms5 (that were copying from ms4 about two weeks ago), changed all perms on the directories they had reached so thumbs can be served/read from them.. oh. not me, someone else must have done it, I'm not here :-P
19:02 RobH: ms4 shutting down for memory troubleshooting
18:52 RobH: ms4 troubleshooting, disragrd bounces]
18:51 notpeter: restarting nagios
12:41 hcatlin: Restarting mobile cluster with April code update.
00:49 notpeter: restarting nagios. hopefully now with more sms!

April 21

23:32 logmsgbot: midom synchronized php-1.17/includes/ImagePage.php
20:54 logmsgbot: pdhanda synchronized live-1.5/404.php
19:02 domas: bumped up maxclients/serverlimit on singer to 350 (up from 150), set maxrequestsperchild to 30 to avoid heap blowup (down from 0), all governed via apache2/conf.d/maxrequests
18:25 Ryan_Lane: restarting apache on singer
18:19 Ryan_Lane: applying system patches to raskin
17:47 Ryan_Lane: restarting apache on singer
17:04 logmsgbot: pdhanda synchronized live-1.5/404.php
16:52 rainman-sr: copying stuff from /home/ariel/searchidx/ to searchidx2 as it didn't get copied
15:32 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Fixing typo to properly check for wgUseEmailCapture'
15:29 logmsgbot: awjrichards ran sync-common-all
15:06 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Added section for EmailCapture, enabling on testwiki'
15:05 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Added section for EmailCapture'
14:50 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 86464:
14:49 awjr: running scap on fenari to dark-deploy EmailCapture extension
03:44 Ryan_Lane: uninstalled python-json on wikitech, and installed python-simplejson
01:45 Ryan_Lane: testing statusnet library again
01:45 Ryan_Lane: testing statusnet library again
01:44 Ryan_Lane: testing statusnet library again
01:35 Ryan_Lane: testing a statusnet library for morebots that has a sane license
01:20 Ryan_Lane: installing ca-certificates on wikitech
01:11 Ryan_Lane: installing curl on wikitech
01:04 Ryan_Lane: installing python-json on wikitech
00:11 Ryan_Lane: added a sweet, special message to morebots
00:03 notpeter: moving some searchidx files into puppet's sphere of influnece.

April 20

23:10 rainman-sr: search indexing is up and running again, might take a couple of days for things to catch up though
23:07 rainman-sr: doing a slow restart of the whole search cluster (2 minute intervals between host restarts)
22:52 rainman-sr: starting search indexer on searchidx2
16:29 mark: Squids finished rebuilding, removed iptables rules
16:21 mark: Removed iptables rules on sq77/78
16:09 mark: Downgraded sq65
16:06 mark: removed iptables rule on sq66
16:05 Ryan_Lane: downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24
16:02 Ryan_Lane: downgrading squid on amssq31
16:01 Ryan_Lane: downgrading squid on amssq38
16:01 Ryan_Lane: downgrading squid on amssq39
16:01 Ryan_Lane: downgrading squid on amssq40
16:01 Ryan_Lane: downgrading squid on amssq41
16:00 Ryan_Lane: downgrading squid on amssq37
15:59 Ryan_Lane: downgrading squid on amssq36
15:59 Ryan_Lane: downgrading squid on amssq32
15:59 Ryan_Lane: downgrading squid on amssq33
15:58 Ryan_Lane: downgrading squid on amssq34
15:58 Ryan_Lane: downgrading squid on amssq35
15:57 mark: Downgrading Squid on sq71-78
15:57 Ryan_Lane: downgrading squid on amssq42
15:56 Ryan_Lane: downgrading squid on amssq43
15:56 Ryan_Lane: downgrading squid on amssq45
15:54 Ryan_Lane: downgrading squid onamssq45
15:53 Ryan_Lane: downgrading squid on knsq23
15:33 RobH: lowered maxclients more on singer, its going to slow down secure, but hopefully keep it online longer.
15:04 Ryan_Lane: downgraded sq71 to squid-2.7.7
14:51 mark: Pulled squid 2.7.9 packages from the lucid-wikimedia reprepro APT repository, reinstated 2.7.7
14:34 mark: Downgrading squid on sq66
14:15 mark: Stopping squid on sq72
14:14 RobH: reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes
14:09 RobH: singer died, rebooting.
14:07 RobH: secure, blogs on singer, bouncing due to singer cpu max
13:53 RobH: singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing
11:34 logmsgbot: catrope ran sync-common-all
11:33 RoanKattouw: Running sync-common-all to deploy r86464
11:11 logmsgbot: catrope synchronized php-1.17/includes/Exif.php 'Fix for bug 28615 (fatal on image description page)'
00:41 awjr: deployment of r86441 to payments servers complete
00:37 awjr: deploying changes to send full donor address information to PayflowPro (r86441) to all payments servers
00:31 awjr: re-deploying changes to send full donor address information to PayflowPro on payments2.wikimedia.org (suspected cause of previous issue: local browser cacheing)
00:14 logmsgbot: midom synchronized php-1.17/wmf-config/db.php

April 19

23:24 richcole1: rebooting db11 to check raid
22:57 awjr: reverting to r84367 of deployment branch on payments2.wikimedia.org due to bad address information appearing in civicrm
22:53 awjr: deploying changes to send full donor address information to PayflowPro payments2.wikimedia.org
22:20 Ryan_Lane: virt1-4 installed
20:54 Ryan_Lane: powercycling singer
20:53 Ryan_Lane: restarting apache on spence
20:52 mark: powercycled singer
20:35 Ryan_Lane: powercycling srv196
17:31 Ryan_Lane: finished ams and kn text squids
17:17 RobH: change live, etherpad still online and functional
17:16 RobH: pushing change to hooper so it doesnt kill etherpad when other apache vhosts exist
17:16 RobH: updated puppet entry for etherpad vhost
16:54 Ryan_Lane: upgrading squid on all squid systems, slowly, over the course of the day
15:09 mark: Killed all etherpad-user processes and started etherpad
15:09 RoanKattouw: Removing data for the section edit link experiment from the clicktracking table. Data is backed up in /home/catrope/sel . This will delete 8.4M of the 9.6M rows in the clicktracking table
15:05 RoanKattouw: Etherpad is broken, serving 500s. Have e-mailed Peter
14:06 logmsgbot: demon synchronized php-1.17/includes/diff/DifferenceEngine.php 'r86395'
12:50 mark: NICEd bayes processing cron job on mchenry
10:14 RoanKattouw: <logmsgbot> !log catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump $wgArticleFeedbackTrackingVersion to 1'
10:14 RoanKattouw: <logmsgbot> !log catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r86384'
10:14 RoanKattouw: <logmsgbot> !log catrope synchronized php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r86299'
10:14 RoanKattouw: Restarted morebots
01:05 Tim: added an entry to /etc/sudoers on fenari and hume allowing wikidev users to run commands as apache

April 18

21:28 mark: Setup PIM-DM on cr1-eqiad:ae1.666 (pseudowire port to pmtpa) and csw5-pmtpa:ve10 (counterpart) to get multicast routing from tampa
21:24 mark: Setup PIM-SM on cr1-eqiad and cr2-eqiad, with static RPs and a local RP on cr1-eqiad. PIM is running on all production subnets.
16:16 pdhanda: running maintenance/CleanupImages on all.dblist
15:58 mark: Rolling restart of pmtpa text frontend squids; one every two minutes
14:27 mark: Powercycled db11
14:13 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'depool db11'
14:12 Reedy: DB11 related issues. Ops are looking at the issue
10:30 logmsgbot: root synchronizing Wikimedia installation... Revision: 86086:
10:05 logmsgbot: root synchronizing Wikimedia installation... Revision: 86086:
08:43 Tim: running /home/wikipedia/bin/l10nupdate to get r86294

April 17

20:27 mark: Setup dirty "pseudowire" between pmtpa and eqiad; a GRETAP tunnel inside a v4-in-v6 tunnel bridged to eth1/eth0.666 on 2 linux boxes (sq71 on pmtpa side). Now running OSPF on it to get full v4 interconnectivity

April 16

08:33 apergos: disabled snaps on searchidx on in rainman's crontab, moving some en.wiki.* files in /a/search/indexes/snapshot off to fenari:/home/ariel/searchidx/....

April 15

18:41 mark: Rsynced private thumbs dir from ms4 to ms5
15:45 mark: Upgraded re0.cr1-eqiad to JUNOS 10.4R3.4
15:27 mark: Switching over routing engines from re0 to re1 on cr1-eqiad
15:23 mark: Upgraded re1.cr1-eqiad to JUNOS 10.4R3.4
14:44 mark: Upgraded re0.cr2-eqiad to JUNOS 10.4R3.4
14:34 mark: Upgraded re1.cr2-eqiad to JUNOS 10.4R3.4
09:57 apergos: moving some old (from 2009) dirs from /a/search/indexes/import on searchidx1 to /home/ariel/searchidx1/... on fenari, if I'm told we can toss em outright I'll do that later.
06:01 apergos: cleared out space on searchidx1 again. we only have 4.5gb free and it's still at 100%. out of tricks...
00:54 notpeter: py making some changes to nagios. subscribing the service to lots of files. also, lots more files being pushed out by puppet.

April 14

23:41 Ryan_Lane: adding reverse mappings for virt1-4
23:28 Ryan_Lane: adding dns entries for virt1-4
22:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enable ArticleFeedback on enwiki, upgrade is complete'
22:09 RoanKattouw: Running populateAFRevisions.php on enwiki
22:06 RoanKattouw: Creating article_feedback_revisions table on enwiki
22:04 RoanKattouw: Clearing message blobs
22:03 logmsgbot: catrope ran sync-common-all
22:03 RoanKattouw: Deploying ArticleFeedback changes to the cluster with sync-common-all
22:02 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable ArticleFeedback on enwiki for upgrade'
21:42 mark: Added vlan 103 to ports e 15/5 to 15/8 (virt1-4 eth1) (tagged) on csw1-sdtpa
21:42 mark: Put ports e 15/1 to 15/4 (virt1-4 eth0) in vlan 105 (untagged) on csw1-sdtpa
21:40 mark: Created VLAN 105 ("virt-hosts") on csw1-sdtpa, assigned virtual interface ve 9, assigned ip 10.4.16.0/24
21:39 mark: Added virtual interface ve 7 ("virt") on csw1-sdtpa, assigned ip 10.4.0.1/24
21:39 mark: Removed virtual interface ve 7 ("virt") on csw5-pmtpa
21:24 Ryan_Lane: cleaning up searchidx1
21:11 RoanKattouw: Running extensions/ArticleFeedback/populateAFRevisions.php on testwiki
21:11 RoanKattouw: Creating article_feedback_revisions table on testwiki
21:08 RoanKattouw: Updating ArticleFeedback code on test
16:18 RobH: singer back online, blogs, survey, and secure should all be back in service
16:15 RobH: blogs offline due to singer outage, as well as survey, working on resolution
16:14 RobH: powercycling singer, unresponsive to ssh and to serial console
16:13 RobH: secure issue is being investigated
16:13 pdhanda: running cleanupTitles.php for all wikis in all.dblist
16:13 RobH: investigating singer cpu spike
15:36 mark: Implemented LDAP lookups on mail relay mchenry
14:13 Reedy: Delayed message. Ran repopulateCodePaths on mediawikiwiki
14:13 Reedy: Delayed message. Ran populateFollowupRevisions on mediawikiwiki
14:04 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/modules/ext.codereview.js 'Disable comment field autofocus'
07:54 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/CodeReview.php 'Fix CodeReview fatals'
07:50 logmsgbot: catrope ran sync-common-all
07:47 RoanKattouw: Running sync-common-all and clearMessageBlobs.php to deploy UploadWizard changes
07:35 RoanKattouw: Updating MediaWiki on fenari to r86034 for UploadWizard deployment. Only pushing changes to testwiki for now
07:14 logmsgbot: tstarling synchronized php-1.17/includes/WebRequest.php 'r86031'
07:14 logmsgbot: tstarling synchronized php-1.17/img_auth.php 'r86031'

April 13

23:18 notpeter: restarting nagios
21:15 notpeter: restarting nagios
19:54 awjr: disabling drupal/civicrm cron on contacts.wikimedia.org (singer) as part of an interim solution to civimail problems in CiviCRM 3.4beta1
18:31 ^demon: ran code review updates for mediawikiwiki
12:54 mark: Restarted pdns on linne
12:54 mark: Defined /64 neighbor blocks in 1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa in DNS
12:31 mark: Defined /31 neighbor blocks in 154.80.208.in-addr.arpa in DNS
09:15 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (soon we have a two week break from these :-P)'
01:55 Ryan_Lane: upgrading squid to 2.7.9-5 on sq58 and sq73
01:35 Ryan_Lane: upgrading squid to 2.7.9-4 on sq58 and sq73
00:42 Ryan_Lane|food: also removing sheepdog from nova-*.tesla
00:39 Ryan_Lane|food: removing natty versions of libvirt and friends and re-installing the lucid versions

April 12

20:18 RobH: killed apache vhost for outreachcivi since it appears defunct in favor of contactscivi on singer
19:08 Ryan_Lane: upgrading nova-* on nova-*.tesla
19:05 Ryan_Lane: updating squid on sq73 to test new squid and patches
05:51 Ryan_Lane: purging nova-ajax-console-proxy from nova-compute1/2.tesla and adding to nova-controller.tesla
04:57 Ryan_Lane: purging nova-ajax-console-proxy from nova-controller.tesla and adding to nova-compute1/2.tesla
02:11 logmsgbot: tstarling synchronized php-1.17/includes/Sanitizer.php 'r85859'
01:30 logmsgbot: root synchronizing Wikimedia installation... Revision: 85853:
01:28 Tim: svn up/scap to deploy r85848
00:31 Ryan_Lane: installing glance on nova-controller.tesla

April 11

22:13 Ryan_Lane: installing python-nova.adminclient on nova-controller.tesla
21:19 notpeter: added lag check to storage3. restarting nagios
20:33 Ryan_Lane: pushing squid-2.7.9-1wm1 to wikimedia-lucid repo
05:15 Tim: did squid configuration update for bug 28235
04:40 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'Made categories reviewable on kawiki'
03:23 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php
03:21 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'flaggedrevs for kawiki'
03:13 AaronSchulz: Enabled FlaggedRevs for Georgian Wikipedia
03:11 logmsgbot: aaron ran sync-common-all

April 10

05:22 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (can't wait for local sysadmins to be able to do this :-P)'

April 9

17:07 logmsgbot: ariel ran sync-common-all 'sync for elwikinews round 2, let's get the import right this time folks cause this is too nerve-wracking'
16:40 logmsgbot: ariel synchronized all.dblist 'remove elwikinews, need to drop and recreate after borked import'
16:29 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'logo updates for a couple wikis for phillipe'
10:10 mark: Changed dataset1's clock source to HPET, synced it with ntpdate and restarted ntpd
07:58 apergos: power cycling searchidx1, load was at 60, unresponsive to commands after login from mgmt console
02:46 RobH: troubleshooting a couple new wikis, had to sync-apaches and restart them gracefully
01:01 notpeter: changed my.cnf on storage3 to replicated-do-db= drupal,mysql,civicrm
00:50 Ryan_Lane: installing nova-ajax-console-proxy on nova-controller.tesla

April 8

23:19 logmsgbot: laner ran sync-common-all
23:18 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php
22:50 Ryan_Lane: that graceful was me
22:28 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php
22:16 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php
22:14 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php
22:07 logmsgbot: laner ran sync-common-all
22:06 Ryan_Lane: gave myself deploy access in svn
17:10 notpeter: pushing out new dns zones. forgot to change ptr record for yvon...
15:09 RobH: updating dns with testblog info
13:36 mark: Added swap on /dev/sdc1 and /dev/sdd1 on ms5
13:34 mark: Stopped RAID10 array /dev/md2 again, sync takes too long
13:30 mark: Created RAID10 array for swap across first partition of 46 drives on ms5
13:21 mark: Stopped all rsyncs to investigate ms5's sudden kswapd system cpu load
07:57 apergos: assigned snapshot1 internal ip in dns
06:11 apergos: moving snapshot1 to internal vlan etc
04:15 notpeter: pushing new dns w/ eixamanius as cname for hooper and yvon as new name for box that was previously eixamanius
04:15 notpeter: stopping etherpad

April 7

21:43 notpeter: removed a silly check for hooper that I made and restarted nagios
19:06 Ryan_Lane: switching openstack deb repo back to trunk, and upgrading packages on nova-controller, since we are likely to target cactus now
15:40 mark: Restarted rsyncs
15:26 mark: Created a test btrfs snapshot of /export on ms6
15:12 mark: Temporarily stopped the rsyncs on ms5 to test zfs send performance
13:56 mark: Reenabled ms6 as backend on esams.upload squids
13:11 apergos: replaced ms4 in fstab on fenari with ms5 so we have thumbs mounted there
12:08 mark: Restarted rsyncs on ms5
12:07 apergos: nginx conf file change to "thumb" added to puppet
12:00 mark: Removed the test snapshot on ms5
11:47 apergos: edited in place /etc/nginx/sites-available/thumbs and /export/thumbs/scripts/thumb-handler.php to make thumbs generated on the fly return 200. they were returning 404
10:25 logmsgbot: catrope synchronized php-1.17/thumb.php 'Attempted fix for wrong temp/thumb paths'
10:20 apergos: (after reports from en vp that the search index has not been updated for over 4 days)
10:18 apergos: restarting search indexer on searchidx1
09:35 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUploadStash.php 'debugging'
09:08 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUploadStash.php 'r85612'
08:23 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/UploadWizard.php 'Fix fatal due to missing API module'
08:17 logmsgbot: catrope ran sync-common-all
08:16 RoanKattouw: I meant srv196, not srv193
08:15 RoanKattouw: Deploying UploadWizard for real this time, forgot to svn up first. sync-common-all then clearMessageBlobs.php
08:14 RoanKattouw: Commenting out srv193 in mediawiki-installation node list because its timeouts take forever
08:10 RoanKattouw: srv196 is not responding to SSH or syncs from fenari (they time out after a looong time) but Nagios says SSH is fine. Should be fixed or temporarily depooled
08:08 RoanKattouw: Clearing message blobs
08:07 logmsgbot: catrope ran sync-common-all
08:04 RoanKattouw: Scap broke with sudo stuff AGAIN, running sync-common-all
08:01 RoanKattouw: Running scap to deploy UploadWizard changes
07:11 apergos: turned em off again, started seeing timeouts. bah
06:39 apergos: and two more...
06:31 apergos: restarted two of the 8 rsyncs on ms5, keeping an eye on them
01:31 domas: added nobarrier to xfs mount options on db32 and db37

April 6

20:38 RobH: updated puppet with a svn::client class (rt#721)
20:18 RobH: pulled wm09schols, wm10schols, and wm10reg out of enabled sites on singer
20:05 apergos: suspended all rsyncs on ms5, we were seeing nfs timeouts on the renderers all of a sudden
18:50 apergos: killed morebots and let the restart script start it up again

April 5

23:00 Ryan_Lane: restarting search indexer on searchidx to free space held by deleted logs
22:58 Ryan_Lane: clearing up some space on searchidx1
22:20 notpeter: crammed an etherpad db into db9's mysql hole.
17:57 Ryan_Lane: restarting llsearchd on all search boxes
17:45 RoanKattouw: Restarted morebots, running on wikitech as catrope
17:45 Ryan_Lane: changing the udp log location for search to emery
12:16 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Undo $wgForceUIMsgAsContentMsg change on incubator from last night per DannyB'

April 4

23:28 Ryan_Lane: uploading ircecho package to lucid-wikimedia repo, for nagios irc bot
22:22 Ryan_Lane: upgrading wikimedia-task-appserver package on srv281
22:22 Ryan_Lane: uploading new version of wikimedia-task-appserver to lucid-wikimedia repo; merges back in 1.17 changes that were missing
22:04 RobH: updated noc robots entry in its apache config on fenari
21:58 Ryan_Lane: srv281 is acting as a temporary scaling server for testing of lucid imagescalers, and to help with thumbs load.
21:27 Ryan_Lane: depooling srv281 from appservers
21:21 Ryan_Lane: syncing apaches to get configuration pushed to srv281
21:16 Ryan_Lane: rebooting srv281
21:01 Ryan_Lane: adding srv281 to rendering cluster in pybal via fenari
20:32 Ryan_Lane: uploading a new version of wikimedia-task-appserver fixing a problem with sync-common
20:13 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add mainpage to $wgForceUIMsgAsContentMsg for incubatorwiki'
19:55 Ryan_Lane: srv281 successfully ran imagescaler puppet class. ready for testing.
19:47 Ryan_Lane: adding php5-fss to lucid-wikimedia repo
19:11 Ryan_Lane: adding wikimedia-task-appserver to lucid-wikimedia repo
18:58 RobH: bugzilla updates complete
18:50 RobH: updating bugzilla per rt#718 bz#28409 bz#28402
18:42 notpeter: added cname etherpad for hooper.wikimedia.org
18:00 Ryan_Lane: added the wikimedia-fonts package to lucid-wikimedia repo
17:29 notpeter: adding self to nagios group. rebooterizing nagios.
05:58 apergos: cleaned up perms on commons/thumb/a/af, left over from interrupted rsync test last night
05:50 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling pool counter on all wikis'
04:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on testwiki and test2wiki'
01:22 Tim: apache CPU overload lasted ~10 mins, v. high backend request rate, don't know cause, seems to have stopped now

April 3

18:42 apergos: 8 rsyncs of ms4 thumbs restarted with better perms so scalers can write... in screen as root on ms5. If we start seeing nfs timesouts in the scaler logs please shoot a couple
17:14 mark: Deployed max-connections on all cache peers for esams.upload squids to their florida parents (current limit 200)
17:00 mark: Removed the carp weights on the esams backends again, as the weighting was completely screwed up
16:59 mark: Started knsq13 backend
14:27 logmsgbot: catrope ran sync-common-all
14:26 RoanKattouw: Running sync-common-all to deploy r85256
13:03 apergos: shot rsyncs on ms5, setting 777 dir perms on all thumbnail dirs (eg e/ef/blablah.jpg) so scalers can write into them
12:53 apergos: did same for rest of projects and subdirs (777 on hash dirs)
12:46 apergos: chmod 777 on commons/thumb/*/* on ms5 so that scalers can create directories in there (mismatch of uid apache vs www-data)
11:12 mark: Raised per-squid connection limit to ms5 of 200 to 400 connections
11:05 mark: Raised per-squid connection limit to ms5 of 100 to 200 connections
10:55 mark: Fixed squid loop, the pmtpa.upload squids were using the esams squids as "CARP parents for distant content"
10:29 mark: Fixed puppet on sq42/43
09:44 mark: Lowered FCGI thumb handlers from 90 to 60 again, to reduce concurrency
08:08 mark: Started 4 more rsyncs (8 total now)
07:49 mark: Removed mlocate from ms5, puppetising
07:42 mark: Started 4 rsyncs from ms4 to ms5 (--ignore-existing)
07:32 mark: increased thumb handler count from 60 to 90
07:11 mark: Doubled the amount of fcgi thumb handlers
07:08 mark: Turned off logging of 404s to nginx error.log
06:50 mark: Restarted Apache on the image scalers
06:49 mark: Reconfigured ms5 to use the 404 thumb handler
06:48 Ryan_Lane: disabling nfs on ms4
06:33 mark: Running puppet on all apaches to fix fstab and mount ms5.pmtpa.wmnet:/export/thumbs
06:32 mark: Unmounting /mnt/thumbs on all mediawiki-installation servers
06:30 mark: Remounted NFS /mnt/thumbs on the scalers to ms5
06:28 Ryan_Lane: bring nfs back up
06:28 Ryan_Lane: brought ms4 back up. stopping the web server service and nfs
06:20 mark: Setup NFS kernel server on ms5
06:18 Ryan_Lane: powercycling ms4
05:29 Ryan_Lane: rebooting ms4 with -d to get a coredump
05:14 apergos: reanbling webserver on ms4 for testing
04:45 apergos: stopping web service on ms4 for the moment
04:29 apergos: shot webserver again
04:26 apergos: turned off hourly snaps on ms4, turned back on webserver and nfs
04:09 apergos: rebooted ms4, shut down webserver and nfsd temporarily for testing
02:58 apergos: still looking at kernel memory issues, still rebooting, ryan should be here in a few minutes to help out
02:03 apergos: a solaris advisor... also have zfs arch cache max to 2g which is ridiculously low but wtf right?
02:02 apergos: set tcp_time_wait_interval to 10000 at suggestion of
01:37 apergos: lowered zfs arch max to 2g (someone should reset this later)... will take effect on next reboot
00:29 apergos: rebooting with the new zfs arc cache max value, which will reduce the min value as well... dunno if this will give us enough breathing room or not
00:24 apergos: set zfs arc cache to ridiculously low value of 4gb, since when it's healthy it's using much less than that (1gb), this will take effect on reboot
00:22 Reedy: Still experiencing MS4 issues, thumb service is likely to be problematic for most users

April 2

23:47 apergos: rebooting ms4 from serial console, out to lunch and took the renderers down too
18:42 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Per NeilK, change Category:Uploaded_by_UploadWizard to Category:Uploaded_with_UploadWizard'
17:59 mark: Upgrading varnish to 2.1.5
17:14 logmsgbot: demon synchronized php-1.17/includes/filerepo/LocalFile.php 'r85200'
14:19 mark: Implemented CARP weights for distant CARP parents on squid configurator (used to be all equal before)
11:36 mark: Created btrfs filesystem on ms6, striped (raid10 style) over 46 devices - very experimental
09:50 mark: Reinstalling ms6 with Ubuntu 10.04
09:50 mark: Fixed torrus again
06:02 mark: !wikipedia The image thumbnail servers appear stable now
04:59 mark: Increased nginx worker processes from 1 to 4, set file limit to 30k
04:40 mark: !wikipedia Image Thumbnail server outage, it's being worked on
04:34 mark: Power cycling ms4 again
04:06 mark: Power cycled ms4 again
04:02 mark: Removed ms4 from pmtpa.upload config, sending all thumbs to ms5
03:47 mark: Restarted rsyncs ms4->ms5
03:25 Ryan_Lane: powercycling ms4 again
02:59 Ryan_Lane: rebooting ms4
02:46 Ryan_Lane: seems ms4 is totally dead, powercycling it
01:09 Ryan_Lane: installing python-pyinotify on spence for an updated ircecho

April 1

21:35 Ryan_Lane: purging some binlogs on db9 to free up space
21:35 RobH: bugzilla now version 4
21:31 RobH: taking down bugzilla for a quick upgrade
18:48 Ryan_Lane: added ctwoo, brion, py, and reedy to the engineering alias
18:36 mark: Deployed ms5.pmtpa.wmnet as a special 'apache' for pmtpa squid uploads... now serving a small portion of commons thumbs
18:11 RobH: bugzilla back online, CRproxy was affected, and repaired
17:30 RobH: bugzilla.wikimedia.org going offline for database backup and upgrade
17:13 RobH: beginning upgrade process for bugzilla, it's availability will be in question during this time
16:59 mark: Turned off Etag in the webserver7 configuration (/opt/webserver7/https-ms4/config/obj.conf) on ms4
16:50 notpeter: rm-ing old binlogs on db9 after confirming that there is no slave lag on db10
15:53 mark: Puppetised nginx and htcp purger setup on ms5
11:36 apergos: restarted lighty on dataset2 (but why did it die?)
00:06 logmsgbot: tstarling synchronized php-1.17/includes/specials/SpecialImport.php 'r85099'