Server admin log

February 27

22:46 mutante: rebooting bast1001
22:36 mutante: installing package upgrades on bast1001
22:16 logmsgbot: reedy synchronized wmf-config/
21:44 RobH: rdb1001 & rdb1002 allocated, network setup, and ready for handoff, per rt4608
21:36 RobH: authdns-update for rdb1001-1002
21:36 RobH: authdns-update for sdb1001-1002
21:31 Tim: running resetUserTokens.php on all wikis, in screen on hume, to fix user_token field corruption (bug 41586)
21:26 logmsgbot: tstarling synchronized php-1.21wmf10/includes/User.php 'user_token fixes'
20:05 K4-713: updated payments cluster to fcbf9211c9836b
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf10
18:55 LeslieCarr: rebooting arsenic
18:42 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for EducationProgram
18:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for EducationProgram
17:57 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
17:51 K4-713: updated payments cluster to eab489d0889a
17:13 mutante: rebooting fenari for kernel upgrade
08:33 apergos: restarted lucene search on search1015
08:04 logmsgbot: reedy synchronized live-1.5/static-1.21wmf4/
08:04 logmsgbot: reedy synchronized docroot/bits/static-1.21wmf4/
07:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
07:05 Krinkle: done adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
07:04 logmsgbot: krinkle Finished syncing Wikimedia installation... :
06:29 apergos: snapshot1002 down for reinstallation
06:27 logmsgbot: krinkle Started syncing Wikimedia installation... :
06:15 Krinkle: adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
06:15 Krinkle: added 1.21wmf1, 1.21wmf2 (bug 44570)
05:35 apergos: restarted lucene search on search1016
04:57 logmsgbot: krinkle Finished syncing Wikimedia installation... :
04:37 logmsgbot: krinkle Started syncing Wikimedia installation... :
02:57 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 27 02:57:22 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 27 02:31:06 UTC 2013
01:35 Krinkle: Bringing back checkouts of MediaWiki 1.21wmf5 - 1.21wmf1 on fenari for bug 44570
01:35 Jeff_Green: dist-upgrade and reboot grosley
00:25 logmsgbot: tstarling cleared profiling data

February 26

23:29 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
23:27 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
23:04 K4-713: synchronized payments cluster to 596b854787132c730
22:35 mutante: installing package upgrades on fenari
22:31 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:31 LeslieCarr: nagios is dead! spence.wikimedia.org if you want to access it directly
22:16 mutante: gracefulling eqiad Apaches to push redirect for wikivoyage.eu
22:15 logmsgbot: dzahn gracefulled all apaches
22:15 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:58 RobH: stat1 puppet runs fail due to dan account error, i am working on it, but i also want to run out and get something to eat, so leaving in broken state for 10 minutes until i return to work on it
20:21 mutante: DNS update - add wikiovyage.eu
19:34 Jeff_Green: package updates & reboots of payments hosts
19:13 paravoid: freezing all exim4 bounces to restore normal email delivery
18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:54 paravoid: restarting opendj on sanger
17:45 AaronSchulz: Copying math files into ceph
15:14 mark: suspended i-000001d7 and i-0000039e for sending out malicious emails
15:06 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php
15:05 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php
12:26 apergos: temp blacklist live hack on sodium for exim4 to prevent some spam, see /etc/exim4/blacklist for that
11:20 logmsgbot: mlitn synchronized php-1.21wmf10/extensions/ArticleFeedback/api/ApiArticleFeedback.php 'Update ArticleFeedback to master'
06:53 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
06:43 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:40 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:28 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:26 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:21 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:10 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:06 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:04 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:03 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 26 02:52:01 UTC 2013
02:46 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:44 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:41 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:37 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:29 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 26 02:28:06 UTC 2013
02:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
01:59 Ryan_Lane: test
01:59 Ryan_Lane: test
01:58 Ryan_Lane: test
01:40 K4-713: updated payments cluster to ce28b210374d8
00:20 mutante: mc1001,ms1004,mw1029,mw1059,mw1070,mw1157 - fixed puppet freshness
00:03 mutante: amssq33,amssq44 - fix more puppet runs

February 25

23:54 mutante: knsq26,knsq28 - deleting puppet lock file, fix puppet runs
23:45 logmsgbot: reedy synchronized wmf-config/
23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable SearchExtraNS on commonswiki'
23:34 mutante: re-enabling disabled puppet on srv253,srv235
22:16 Ryan_Lane: restarting opendj/pdns on virt0
22:02 Reedy: Ran namespaceDupes.php on huwikt
21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:28 logmsgbot: reedy synchronized docroot
20:27 logmsgbot: reedy synchronized wmf-config/
19:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
19:40 mutante: powercycling frozen niobium
19:32 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf10
18:54 mutante: wikivoyager.de and wikivoyager.org transferred and both redirect to wikivoyage.org
18:50 mutante: deleting foundation-news-l (deprecated, replaced by WikimediaAnnounce-l)
18:49 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
18:31 AaronSchulz: Copying all captchas into ceph
18:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
06:38 apergos: restarted lucene search on search1016
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 25 02:52:32 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Mon Feb 25 02:28:55 UTC 2013
01:15 mark: Powercycled strontium
01:07 paravoid: powercycling niobium (both unresponsive from the load)
01:06 paravoid: powercycling arsenic

February 24

06:28 apergos: restarted search1016 lucene
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 24 02:54:30 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Feb 24 02:29:29 UTC 2013

February 23

21:45 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb
21:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb
21:43 Reedy: mw110 is asking for a password
06:13 apergos: restarted lucene search on search1016
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 23 02:52:10 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Feb 23 02:27:17 UTC 2013
00:43 RobH: lvs1003 had a ton of hung puppet processes (no output on them when tracing, so dunno whats up). killed them and kicked manual run

February 22

23:59 RobH: argon is shut down now
23:55 RobH: argon is going offline, killing our limesurvey install
23:29 paravoid: deploying squid config for upload's new transcoded container
23:26 paravoid: deploying squid
22:55 Ryan_Lane: changing dns entry for wikitech-static
22:41 RobH: mw110 throwing odd puppet errors, will come back to it later
22:41 RobH: sq41 reinstalled, throws endless puppet errors during redeployment
22:07 MaxSem: reindexing complete
21:39 RobH: harmon and copper online with OS installed for datadumps misc host support
21:39 MaxSem: Nuking GeoData Solr index and reindexing
21:38 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
21:35 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
21:05 RobH: installing harmon and copper into dump generation misc use
20:17 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GuidedTour/modules/tours/gettingstarted.js 'Resume split test (3/3)'
20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GettingStarted/resources/ext.gettingstarted.js 'Resume split test (2/3)'
20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments/experiments/openTask.js 'Resume split test (1/3)'
20:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
20:13 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
20:09 RobH: authdns-update
19:58 RobH: sq41 boot order was fubar, fixed, reinstalling (as reboot into installer wiped mbr)
19:56 RobH: wtp1002-1004 os installed
19:34 Jeff_Green: start mysqldump of otrs database on db49
19:18 RobH: kicked ntpd manually on mw1045
19:09 RobH: sq41 reinstalling
18:57 RobH: mw112-125 back into pybal api pool
18:56 RobH: mw86-99 & mw111 back into pybal pool
18:49 logmsgbot: robh Finished syncing Wikimedia installation... :
18:37 logmsgbot: robh Started syncing Wikimedia installation... :
17:53 sbernardin: taking down sq41 to replace controller card.... https://rt.wikimedia.org/Ticket/Display.html?id=4550
16:24 mutante: tartupeedia.ee now redirects to et.wp portal page (bug 44893)
16:12 mutante: gracefulling eqiad Apaches via dsh to push tartupeedia.ee redirect (bug 44893)
16:11 logmsgbot: dzahn gracefulled all apaches
16:07 mutante: mw110 - sits at partioning screen in installer
15:47 hashar: gallium removed live hack in /srv/org/wikimedia/doc/puppetsource/manifests/role/labsmediawiki.pp
14:52 ^demon: restarted gerrit on manganese, jetty flapping again
14:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:14 mark: Halted ms-be3001-3003 for controller replacement
13:11 mark: Halted ms-be3004 for controller replacement
13:07 mark: Halted cp3009 for controller replacement
12:20 mark: Halted cp3010 for controller replacement
11:23 mutante: DNS update - add indiawikipedia.com
07:33 apergos: restarted lucene search on search1016
06:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:33 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:33 logmsgbot: tstarling synchronized docroot/bits/favicon/wikipedia.ico
05:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:18 logmsgbot: tstarling synchronized docroot/bits
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 22 02:53:05 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Fri Feb 22 02:29:09 UTC 2013
01:57 binasher: migrating wikivoyage and wikidata external blob stores to innodb
01:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed f699c5a9ad2f780dc18863592dcae4a51ba6d841'

February 21

23:21 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/CodeEditor 'Syncing patch that disables background linting'
23:18 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
23:02 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
22:35 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/
22:33 logmsgbot: awjrichards synchronized php-1.21wmf10/extensions/MobileFrontend
22:12 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Add photo upload schema for event logging'
21:56 notpeter: running do-release-upgrade on dbs 52, 39, 51, and 35
21:49 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/EventLogging
21:48 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing 1 slave for s2-5 from db-secondary for upgrades and maria'
21:48 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
21:47 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/CodeEditor
21:37 notpeter: restarting mysql on db1047 to pick up conf changes
21:31 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
21:23 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
21:05 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php Enabling CodeEditor for Schema namespace on test2 and metawiki.
20:02 binasher: some wikis have myisam page_props tables. converting all to innodb via osc
18:39 Ryan_Lane: enabled Extension:OpenID as a provider on labsconsole
11:04 hashar: Ariel saved Gerrit by restarting it!
10:51 hashar: huge CPU spike on manganese (gerrit host)
10:06 hashar: Jenkins: reseting git branch specifier :-/
09:32 hashar: Jenkins: removing git branch specifier from all mediawiki-core jobs.
05:50 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'shell cgroup'
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 21 02:54:17 UTC 2013
02:30 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
02:30 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 21 02:28:20 UTC 2013
00:53 RobH: issues with new mw servers 86+, took back out of pybal until i can troubleshoot
00:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
00:48 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance/
00:34 binasher: moved wikidatawiki tables on s3 to wikidatawiki_old to keep around for a time just in case; then "drop database wikidatawiki"
00:20 logmsgbot: asher synchronized wmf-config 'enabling wikidatawiki on shard s5'
00:18 logmsgbot: asher synchronized s5.dblist 'adding wikidatawiki'
00:17 logmsgbot: asher synchronized s3.dblist 'removing wikidatawiki'

February 20

22:48 RobH: authdns-update
22:35 binasher: importing wikidatawiki with schema changes into the s5 master, with sql_log_bin disabled
22:24 binasher: importing wikidatawiki with schema changes into the inactive pmtpa side of the s5 shard, with replication of the import just in pmtpa. ignore repl warnings for db{35,44,45,55}
21:48 RobH: authdns-update for wtp1002-1004
21:45 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Take pmtpa Solr servers out of rotation, increase load on master: https://gerrit.wikimedia.org/r/#/c/50056/'
21:41 RobH: ignore db66 errors, asher is working on it
21:25 binasher: running wikidata wb_items / wb_items_per_site schema migrations in parallel on the db66 slave (creating primary keys, cannot be done via online-schema-change)
21:24 RobH: kicked knsq23 to clear puppet freshness alarm
21:24 RobH: maerlant still offline (200 days?). dropped esams ticket for onsite troubleshooting, as mgmt is offline.
21:20 logmsgbot: asher synchronized wmf-config/InitialiseSettings.php 'setting wgReadOnly for wikidatawiki durring maintenance'
21:18 RobH: forced puppet run on lardner, was tired of it showing error for no runs
21:13 RobH: mw112-125 now in service as api apaches in tampa
20:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:38 RobH: mw86-mw111 tested ok, pushing into apache service in pmtpa
20:32 RobH: mw86-mw111 being added to nodelists, ignore errors on them for next few minutes
19:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks to 1.21wmf10
19:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikisource and wikiquote to 1.21wmf10
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikinews and wiktionary to 1.21wmf10
19:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special, wikimedia, private and fishbowl to 1.21wmf10
05:34 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable Scribunto profiler'
04:32 Tim: updating luasandbox to version 1.2, with profiler bug fixes
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 20 02:53:32 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 20 02:29:11 UTC 2013
01:28 RobH: mw1161-1208 have wrong power setting, changing all to system.power.hotspare.enable 0
01:00 notpeter: do-release-upgrade-ing on db58
00:57 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db58 from db-secondary for upgrades and maria, replacing db43'

February 19

23:47 notpeter: do-release-upgrade-ing on db43
23:38 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db43 from db-secondary for upgrades and maria'
23:21 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/49946/'
23:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
23:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/
22:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:41 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix MAX call with GREATEST'
22:29 RobH: sq41 offline, rt 4550
22:20 RobH: sq41 puppet freshness bad, wont accept ssh connections, serial redirection results in no output. will need to reboot it
22:17 RobH: sq41 not allowing ssh, drac resetting
22:15 sbernardin: taking down ms-be12 for hardware replacement per: https://rt.wikimedia.org/Ticket/Display.html?id=4546
22:15 RobH: sq79/80 puppet in locked state, manually restarted puppet, removed lock file, ran manually
22:11 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix mysql 5.5 / ansi sql incompatibility'
21:54 RobH: mw48 was set to puppet agent disabled, re-enabled and fired puppet run
20:04 logmsgbot: mflaschen synchronized . 'Syncing GuidedTour submodule omitted when new branch cut was first cloned.'
19:59 Ryan_Lane: disabling project storage volumes for all labs projects with no instances
15:02 hashar: restarting jenkins to enable some startup setting
14:58 hashar: manganese CPU spike ongoing. Started roughly at 14:40 UTC
13:29 mutante: wikimedia.jp.net now redirects to wikimedia.org | wikimediacommons.jp.net now redirects to commons.wm
13:27 mutante: DNS update - adding wikimedia.jp.net
13:25 logmsgbot: dzahn gracefulled all apaches
13:20 mutante: gracefulling eqiad Apaches via dsh to push redirect change for .jp.net domains
12:39 apergos: powercycling ms-be7 (unresponsive to ping or from console)
11:29 mutante: installing package upgrads on neon/icinga
02:49 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 19 02:49:32 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 19 02:28:25 UTC 2013
01:12 logmsgbot: anomie synchronized php-1.21wmf10/includes/api/ApiQuerySiteinfo.php 'Backport fix for ugly API bug'
00:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
00:30 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php

February 18

23:51 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
23:51 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
23:46 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
23:25 Tim: on snapshot1002 -- fixed incorrect owner on some MW files, killed broken apt-get and upgraded wikimedia-task-appserver
23:24 Tim: fixed rsyncd.conf on scap proxies to allow connections from new apaches
23:14 logmsgbot: tstarling Finished syncing Wikimedia installation... :
23:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:45 logmsgbot: tstarling Finished syncing Wikimedia installation... :
22:42 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:37 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
22:37 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
22:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf10
20:37 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/ 'Scap is a liar'
20:31 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
20:15 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Diff
19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/DataValues
19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
19:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/
19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf10
19:03 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
18:45 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
17:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache
17:53 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache
17:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf10
17:28 logmsgbot: reedy synchronized php-1.21wmf10 'Initial sync out'
17:03 logmsgbot: reedy synchronized wmf-config/
17:02 logmsgbot: reedy synchronized docroot
17:01 logmsgbot: reedy synchronized live-1.5/
12:33 mutante: wikiartpedia.(biz|co|info|me|mobi|net) activated - redirect to wikipedia.org (RT-4240)
12:30 mutante: adding wikiartpedia domains to DNS / authdns-update
12:28 logmsgbot: dzahn gracefulled all apaches
12:24 mutante: gracefulling eqiad Apaches via dsh for redirect.conf change
12:22 mutante: mw1045 - eth0 NO-CARRIER - check cable (RT-4545)
12:13 mutante: mw1041 - down - CPU Machine Chk error on mgmt - removing from dsh groups (RT-4517)
05:45 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
05:16 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
05:10 Tim: on db1043: running alter table to convert the relevant field to signed, since I figure this is less disruptive than sql_mode = 'NO_UNSIGNED_SUBTRACTION' and less likely to break again than skipping the query
04:58 Tim: replication broken on db1043, possibly others, due to update of article_feedback_revisions with negative value for afr_total
03:40 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 18 02:29:47 UTC 2013

February 17

02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 17 02:29:31 UTC 2013

February 16

13:38 apergos: restarted lucene-search on search1015 about 20 minutes ago
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 16 02:28:26 UTC 2013
01:47 logmsgbot: reedy synchronized php-1.21wmf9/
01:11 logmsgbot: reedy synchronized wmf-config/filebackend.php

February 15

23:56 RobH: authdns-update to correct mw110 reverse entry
23:28 Ryan_Lane: *modified them in LDAP
23:28 Ryan_Lane: modified project objects to indicate whether or not they are using gluster volumes for project and/or home storage
23:04 RobH: disregard mw1161 flaps, im working on it
20:15 Ryan_Lane: restarting opendj and pdns on virt0 and virt1000
20:15 Ryan_Lane: removed opendj from nfs1/2
19:07 RobH: no one sign mw86-mw125 puppet certs, im still working on them
18:23 RobH: mw87-mw125 installing
17:33 binasher: reimaging db1029-1031
16:55 mutante: removing mw1085 from dsh groups - broken hardware (RT-4453)
16:00 hashar: Enabled a linter for operations/apache-config . Any apache typo in operations/apache-config will publicly write your name on the typo wall of shame. See gerrit 49251 for implementation details and https://integration.mediawiki.org/ci/job/operations-apache-config-lint/ for some build examples.
14:31 mutante: wikimediafoundation.info activated to redirect .org
14:28 mutante: gracefulling Apaches in eqiad using dsh
14:25 logmsgbot: dzahn gracefulled all apaches
14:15 ^demon: [gerrit] updated replication plugin to 20c6fc7e (1.1-SNAPSHOT) which is back in line with upstream
08:02 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Echo 'Syncing Iab077a77aa75a6ff0cf03acecca38403595dda93 to fix exception 640c60c4'
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 15 02:30:05 UTC 2013
01:19 paravoid: deploying squid changes for s/knams/esams/ for knsq* squids (see git log for rationale)
01:09 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling Schema NS on test2wiki'

February 14

22:49 RobH: mw1194 online and now in service as api apache
22:48 RobH: mw1200 added to api pool, online and in service
22:44 RobH: mw1189-1199 (sans 1194) now online as eqiad api apaches
22:37 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
22:29 logmsgbot: mflaschen Started syncing Wikimedia installation... :
22:21 sbernardin: taking storage1 offline for decommissioning per: https://rt.wikimedia.org/Ticket/Display.html?id=4529
22:18 RobH: pushing mw1161-1188 into apache service (sans 1161/1164) from false to true in pybal, otherwise all online
22:16 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating for test2wiki'
22:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math
22:03 Ryan_Lane: replacing gluster upstarts on all labstore nodes with a regular init script
22:01 logmsgbot: reedy synchronized docroot
22:00 logmsgbot: olivneh synchronized wmf-config
21:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on {be|mediawiki|outreach}wikis'
21:45 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
21:39 sbernardin: taking down srv266 and srv268 for decommissioning ... https://rt.wikimedia.org/Ticket/Display.html?id=4534
21:31 ottomata1: restarted solr jetty on solr2
21:30 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
21:25 ottomata: restarted solr jetty on solr3 and solr1003 for MaxSem
21:03 logmsgbot: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
20:24 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating CommonSettings.php for Echo'
20:15 apergos: rebooting mw1010 after some issues with OOM
19:22 RobH: new servers are NOT in pybal yet, and thus not getting traffic
19:22 RobH: just added mw1161-1200 into node lists for eventual service, some may throw sync script errors as I track them down
19:15 Ryan_Lane: rebooting labstore3
19:08 Ryan_Lane: rebooting labstore4
19:01 Ryan_Lane: rebooting labstore2
18:52 Ryan_Lane: rebooting labstore1
15:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Updating wikibase sort order'
15:22 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/client/includes/SortUtils.php 'Updating wikibase sort order'
14:52 mutante: re-enabled cp1044 in pybal - mobile varnish cache servers all on precise now
14:39 hashar: Jenkins: adding in several extension to Zuul configuration gerrit 49041
14:30 hashar: jenkins : updated all jobs using latest version jjb-config
13:35 mark: Fixed fixing
13:27 mutante: disabling cp1044 in pybal, reinstalling with precise
13:22 mutante: re-enabling cp1043 in pybal
13:11 mark: Upgraded varnish to latest version with updated range patch on cp1021-1028
12:16 mark: Inserted varnish 3.0.3plus~rc1-wm7 into the precise-wikimedia APT repository
12:14 mutante: stopping varnish backend on cp1043 and starting reinstall
12:10 mutante: disabling cp1043 in pybal
11:31 mutante: ms1004 - full disk, deleted large HTCPpurger.log file, fixed puppet runs
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 14 02:29:05 UTC 2013
01:17 andrewbogott: updated OpenStackManager to git head on virt0. Previous state is preserved (temporarily) in the local 'premerge' branch.
01:13 paravoid: removing mysql & apache from emery(!)
00:03 RobH: db29 os installed, handed off to peter

February 13

23:50 RobH: db29 reinstalling
23:48 paravoid: reinstalling sq48
23:22 RobH: db27 and db29 will be up and down for reinstallations
23:17 sbernardin: rebooting db29 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
23:17 sbernardin: rebooting db27 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
23:11 RobH: db1013 online and puppet updated, needs to be pushed into some db cluster service
22:45 RobH: mw27 locked up, powercycling
22:36 notpeter: powercycling all eqiad jobrunners
22:13 RobH: adding mw1201-1208 to eqiad apache pool
21:44 Reedy: Running sync-common on mw1203
21:43 logmsgbot: reedy synchronized php-1.21wmf9 'resync to fix file permission errors'
21:32 Reedy: Running sync-common on mw1204
21:30 Reedy: Running sync-common on mw1202
21:27 RobH: db27 rebooting for reinstallation
21:27 RobH: db29 offline due to hardware issue, related ticket 4526
21:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:11 logmsgbot: csteipp synchronized wmf-config
21:09 logmsgbot: reedy cleared profiling data
20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Uncommenting enwiki from localClientDatabases'
19:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on enwiki'
19:47 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase/
19:30 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaning wikis to 1.21wmf9
19:21 logmsgbot: aaron synchronized php-1.21wmf9/maintenance/runJobs.php 'deployed b03384f0ebd95e7f79638fb14ccd55da9c186d97'
19:12 logmsgbot: py gracefulled all apaches
18:59 cmjohnson1: mw1041 is borked and will be powered down and removed from network for h/w checks
18:27 logmsgbot: aaron synchronized php-1.21wmf9/extensions/TranslationNotifications 'deployed d7bf60b57f472e5f75101f58505f61842848f961'
16:26 Nemo_bis: Job queue weirdness and TranslationNotifications duplicates again noticed on Meta (bug 42715)
13:33 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
09:17 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
09:10 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
08:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48797/'
08:25 hashar: jenkins: changing encryption key and regenerating secrets. See bug 44592
05:50 logmsgbot: tstarling synchronized multiversion/MWScript.php
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 13 02:52:39 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 13 02:28:06 UTC 2013
00:42 notpeter: restarting varnishncsa on cp1043
00:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'Touchhhhh'

February 12

23:54 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/specials/donateimage.js 'touch file'
22:58 logmsgbot: maxsem Finished syncing Wikimedia installation... :
22:46 notpeter: disabling innodb-flush-log-at-trx-commit=2 on db1047 w/asher's blessing. will schedule downtime to apply
22:38 logmsgbot: maxsem Started syncing Wikimedia installation... :
22:04 notpeter: creating metrics user on all s1-7 masters
20:25 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
20:22 binasher: flushed mobile varnish
20:21 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
20:19 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
20:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:05 binasher: flushed mobile varnish cache �:(
19:59 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
19:57 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable WikibaseClient on enwiki'
19:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable WikibaseClient on enwiki'
18:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:41 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:36 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:25 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:20 cmjohnson1: nas1001-a swapping mainboard
18:10 logmsgbot: aaron synchronized php-1.21wmf9/extensions/UploadWizard 'deployed 4548f9817d3d8025f4c19e279e9fb1f05b178087'
18:09 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Un-rollback Wikibase'
18:06 RoanKattouw: Reverted Wikibase update earlier, now reapplying
17:53 paravoid: powercycling niobium, SSH & console unresponsive
17:50 mark: Blocking modules=mobile.startup requests with 503 on bits varnish (earlier)
17:50 mark: Forced mw1149-1150 to healthy in bits.eqiad varnish
17:30 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Rolling back Wikibase'
16:18 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 16:18:31 UTC 2013
16:13 Jeff_Green: restarted apache on mw1152
16:01 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
16:00 Reedy: mw1161 and mw1165 are both missing mediawiki deployment files
15:58 logmsgbot: reedy synchronized php-1.21wmf9/includes/site/
15:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 15:54:11 UTC 2013
14:25 mutante: ran puppet on professor
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 02:52:42 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 02:28:53 UTC 2013
01:01 ^demon: stopped puppet and gerrit on manganese and formey
00:28 logmsgbot: aaron synchronized php-1.21wmf9/includes/db/Database.php 'deployed d8705542627f006a7ec9f81a9fb488fcc9a367bd'

February 11

23:19 RobH: unpooled mw1161-1188 from pybal config
22:03 logmsgbot: demon synchronized php-1.21wmf9/extensions/VisualEditor/modules/ve/init/mw/ve.init.mw.Platform.js 'Touching file'
21:08 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Wikibase client throwing fatals, disabling on enwiki pending fix'
21:04 cmjohnson1: auth-dns update
21:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'wikidata client on enwiki'
21:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'wikidata client on enwiki'
20:30 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: huwiki and itwiki to 1.21wmf9
20:27 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: hewiki and itwiki to 1.21wmf9
20:22 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf9
20:11 logmsgbot: demon synchronized php-1.21wmf9/includes/site 'Updating sites code to pull in I8964a03c, I42221832'
18:39 RobH: disregard, pushing one to test, mw1161
18:36 RobH: mw1161-mw1188 being pushed into pybal pool as general apaches
16:58 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/lib/includes/modules/SitesModule.php 'Touching file'
16:50 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'TorBlock to master, now with less annoying logs'
16:41 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Enable logging group for torblock'
16:40 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'Updating TorBlock to master'
16:27 cmjohnson1: commenting out srv226-234 on pybal/pmtpa
14:59 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'bug 44251 hardcode $wgDBuser = "wikiuser" gerrit 47074'
13:55 hashar: jenkins: regenerating all phpcs-HEAD jobs so they stop failing when no files need to be processed bug 44567 gerrit 48450
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 11 02:52:39 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 11 02:27:44 UTC 2013

February 10

22:16 Tim: fixed syslog.eqiad.wmnet DNS entry, restarting rsyslogd on the eqiad apaches (RT 4500)
19:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/48015/'
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 10 02:53:56 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 10 02:28:46 UTC 2013

February 9

12:52 Reedy: mw1161 is missing /usr/local/apache/common-local
12:51 logmsgbot: reedy synchronized php-1.21wmf8/resources/
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 9 02:52:45 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 9 02:28:28 UTC 2013

February 8

23:48 logmsgbot: anomie synchronized php-1.21wmf8/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
23:48 logmsgbot: anomie synchronized php-1.21wmf9/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
23:09 RoanKattouw: and on cerium too
23:08 RoanKattouw: Running puppet on titanium
20:20 RobH: disregard errors for mw1161-1164
20:13 RobH: mw1161-1164 offline for reallocation from job to general apache
20:10 cmjohnson1: decommissioning srvs226-234
19:43 RobH: mw1165-1188 os installed for apache use, initial puppet runs in progress
19:33 RobH: celsus back online
19:23 RobH: celsus rebooting to fix power mgmt cpu spike issue
19:02 RobH: mw1161-1164 in service as eqiad jobrunners
16:32 ottomata: updated vumi packages in apt to vumi_0.5.0~a+143-0~ppa3 and vumi-wikipedia_0.1~a+14-0~ppa3
16:09 mark: Upgraded and rebooted cr2-eqiad (both REs)
15:11 mark: Set cr2-eqiad VRRP prio to 50 to transition to backup role
15:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
15:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: srwikibooks back to 1.21wmf9 to test variant fixes
14:39 hashar: gallium installed php-luasandbox manually. Gerrit change is gerrit 48127.
05:47 logmsgbot: tstarling synchronized php-1.21wmf9/includes/context/RequestContext.php
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 8 02:54:49 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 8 02:30:42 UTC 2013
02:02 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
01:48 LeslieCarr: cr2-knams is back up on the new code
01:36 LeslieCarr: cr2-knams reloading after upgrade
01:34 LeslieCarr: deactivating european bgp sessions - may cause some connectivity issues as connections reset
01:24 LeslieCarr: disabling notifications on icinga and nagios
01:03 logmsgbot: reedy synchronized php-1.21wmf9/languages/classes/
00:25 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/GettingStarted

February 7

23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling GuidedTour on additional wikis'
23:25 LeslieCarr: manually cleared arp and turned managmeent interface off and on of asw-c-eqiad to solve reachability issue
23:02 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy GuidedTour
22:47 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy GuidedTour
22:39 RobH: mw82 pooled into tampa apaches
22:04 ottomata: pulled power and rebooted amaranth toolserver for multichill. Its back online. multichill will notify whomever needs to do more work.
21:58 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/RelatedArticles/ 'Fixes 44761'
21:58 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/RelatedArticles/ 'Fixes 44761'
21:50 RobH: mw81-mw85 in pmtpa apache service
21:39 cmjohnson1: performing a hard reboot on amaranth
21:34 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/E3Experiments
21:33 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments
21:12 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/PostEdit
21:11 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/PostEdit
21:10 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
21:09 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
20:29 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Being paranoid, node lists were in flux during last sync
20:23 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: All ku/sr wikis back to 1.21wmf8 - lang converter breakage
20:20 Nemo_bis: !Wikipedia and !Wikimedia projects are/were down, issue being investigated
20:14 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: kuwiktionary back to 1.21wmf8
20:07 RobH: removing srv219-224 from rendering config in pmtpa
20:06 mark: Rebooting cr1-sdtpa
19:56 RobH: adding mw75-mw80 into tampa rendering pool
18:42 mark: Rebooted re0.cr1-eqiad
18:05 mark: Rebooting re1.cr1-eqiad
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:44 mark: Rebooting cr1-esams
15:40 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
15:39 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
14:54 mutante: granting RT group 'toolserver admins' the 'modify ticket' permission in queue esams
14:38 mutante: planet now updating hourly instead of daily
14:13 mutante: planet speaks SSL now for all languages. (it does not force http->https but it can if there is a demand)
12:00 Nemo_bis: amaranth dead/unreachable from Europe, ping dies at xe-5-3-1.cr2-eqiad.wikimedia.org (173.241.131.218)
10:08 hashar: jenkins : regenerating jobs to make phpcs to show the sniff codes being used. gerrit 47847
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Feb 7 02:52:26 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 7 02:27:52 UTC 2013
02:23 logmsgbot: spage synchronized php-1.21wmf9/extensions/GuidedTour/modules/externals/mediawiki.libs.guiders/mediawiki.libs.guiders.submodule 'sync GuidedTour submodule omitted from 1.21wmf9'
01:57 RoanKattouw: Force-running puppet on wtp1 to figure out why it's broken
01:53 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I9fd98309'
00:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:20 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance
00:00 logmsgbot: reedy synchronized langlist

February 6

23:59 RobH: authdns-update to add min.wikipedia.org via addwiki automation
23:56 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
23:55 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
23:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:43 logmsgbot: reedy synchronized langlist
23:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:28 RobH: srv190-srv223 & srv225 should be offline, they are being decommissioned, can ignore nagios errors
23:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:13 RobH: specifically (i can spell, maybe)
23:13 RobH: removed from pybal specifially
23:12 cmjohnson1: removing srv192-218 and srv225 per rt 4438
23:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:05 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
21:36 cmjohnson1: srv193 is back online
21:21 cmjohnson1: srv193 power down to relocate to different rack
19:47 RobH: authdns-update
19:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.21wmf9
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.21wmf9
19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity to 1.21wmf9
19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia wikis to 1.21mwf9
19:11 cmjohnson1: powerdowing mw1 to reseat DIMM B1
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl wikis to 1.21wmf9
18:43 ^demon: running puppet on manganese, restarting gerrit
18:41 RobH: authdns-update
17:48 LeslieCarr: restarted morebots
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 6 02:51:36 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 6 02:28:05 UTC 2013
02:05 logmsgbot: reedy synchronized wmf-config/
00:42 RobH: mailing lists returned to normal
00:41 RobH: done with data removal on mailing list server, rebuilding mbox
00:37 RobH: stopping mailman to scrub an entry out of archives, will restart shortly

February 5

23:29 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
23:26 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
22:44 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
22:28 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
22:08 RobH: analytics1007 coming down for troubleshooting, disregard nagios errors
22:01 RobH: analytics1001 reinstalled, not puppet signed, back online
22:00 RobH: analytics1001 respects my authority.
21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 44688 - Please disable MoodBar extension on English Wikipedia'
21:28 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
21:12 RobH: analytics1001 going down for troubleshooting/reinstallation
20:21 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:12 RobH: blog update sucessful, caching plugin update nonsuccessful, rolling it back to older plugin
20:07 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:04 RobH: updating the blog software, here goes nothing
19:17 hashar: gallium : Zuul updated!
19:13 hashar: gallium : updating Zuul by cherry-picking a set of changes that let us fetch tags. ff79197..1854e32
17:05 cmjohnson1: authdns update
16:37 LeslieCarr: copied all equinix ashburn IPv4 peering sessions to their new /22 on cr2-eqiad
16:37 Jeff_Green: dist-upgrades for payments systems
16:36 Jeff_Green: restarted morebots :-P
05:37 Ryan_Lane: restarting opendj on all hosts
05:36 Ryan_Lane: remove virt1 replication agreement from all ldap servers
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 5 02:51:10 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 5 02:29:02 UTC 2013
01:07 logmsgbot: reedy synchronized php-1.21wmf9/extensions/TimedMediaHandler
00:51 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
00:39 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
00:29 logmsgbot: reedy synchronized php-1.21wmf9/extensions/ZeroRatedMobileAccess

February 4

23:49 logmsgbot: reedy synchronized multiversion/
23:46 logmsgbot: reedy synchronized multiversion/
23:45 logmsgbot: reedy synchronized wmf-config/
23:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:02 hashar: gallium moved a git extension : mv /var/lib/git/rt-authen-externalauth.git /var/lib/git/operations/debs/
19:44 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
19:39 ottomata: manually gracefully reloaded eqiad apaches. See RT 4449
19:37 logmsgbot: root gracefulled all apaches
19:20 logmsgbot: reedy synchronized wmf-config/
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and wikidatawiki to 1.21wmf9
18:35 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
17:56 logmsgbot: reedy Finished syncing Wikimedia installation... : Build messages for 1.21wmf9
17:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:24 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
17:14 logmsgbot: reedy synchronized php-1.21wmf9/extensions/GuidedTour/
17:09 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
17:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
17:06 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
16:45 logmsgbot: reedy synchronized php-1.21wmf9/ 'Initial sync'
16:22 logmsgbot: reedy synchronized wmf-config/
16:06 Ryan_Lane: force starting any glusterfs volume that has a downed process
15:12 Ryan_Lane: rebooting labstore1
15:11 Ryan_Lane: upgrading labstore1 to precise
14:33 apergos: powercycled ms-be3
14:32 Ryan_Lane: make that rebooting labstore3
14:32 Ryan_Lane: rebooting labstore2
14:07 Ryan_Lane: upgrading labstore3 to precise
11:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cbd432f..82ed797 gerrit 47393 Vector micro design on mlwiki, gerrit 29792 wikipedia translation on pawiki'
07:37 Ryan_Lane: upgrading labstore2 to precise
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 4 02:25:58 UTC 2013
00:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

February 3

20:58 hashar: jenkins : generating mwext-**-phpcs-HEAD jobs
17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:42 logmsgbot: reedy synchronized wmf-config/
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 3 02:28:29 UTC 2013
02:03 logmsgbot: reedy synchronized wmf-config/
02:02 logmsgbot: reedy synchronized images/sul/foundation.png
01:14 logmsgbot: reedy synchronized wmf-config/
00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

February 2

23:59 logmsgbot: reedy synchronized wmf-config/
23:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:34 logmsgbot: reedy synchronized wmf-config/
23:30 logmsgbot: reedy synchronized php-1.21wmf8/extensions/SearchExtraNS
23:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
09:23 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/PageImages/initImageData.php 'https://gerrit.wikimedia.org/r/#/c/44556/'
08:45 MaxSem: Running foreachwikiindblist wikipedia.dblist extensions/PageImages/initImageData.php --earlier-than=20121229000000
03:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 2 02:28:18 UTC 2013

February 1

23:02 K4-713: synchronized payments cluster to b421871b43822
20:00 RobH: authdns-update
19:43 RobH: authdns-update
14:45 cmjohnson1: authdns update "adding mw1161-1200 to eqiad mgmt and production zone files
12:04 cmjohnson1: authdns update adding db1051-60 to zone files
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 1 02:27:09 UTC 2013
01:50 K4-713: updated payments cluster to 45f8a6c6009a4bd7

January 31

23:14 andrewbogott: rebooting virt0 in a fit of optimism and/or desperation
22:44 logmsgbot: spage Finished syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:37 logmsgbot: spage Started syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:26 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:20 ottomata: deployed tab delimiter log format change on squids, varnishes and nginxes
21:40 ottomata: reverted tab separartor change to squid
21:37 ottomata: deloyed tab separator log format changes to squids, merged the corresponding puppet changes to varnish and nginx
20:47 paravoid: deploying patched ganglia and removing .htaccess. Ganglia is public again!
14:59 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
14:58 notpeter: taking down db1047 for upgrade to precise
13:21 hashar: gallium: installed mercurial manually (puppet change is gerrit 46931 PS2)
13:19 hashar: gallium manually removed bzr: apt-get remove bzr python-keyring python-httplib2 python-launchpadlib python-zope.interface python-oauth python-bzrlib bzr python-simplejson python-configobj python-lazr.uri python-lazr.restfulclient python-wadllib
13:19 hashar: gallium: manually installed "bzr"
12:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
10:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remove strategyappswiki
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 31 02:29:47 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 31 02:28:18 UTC 2013

January 30

22:40 binasher: deployed new text/upload squid confs with eqiad in proxySubnets
22:38 logmsgbot: reedy synchronized wmf-config/
22:23 logmsgbot: tstarling cleared profiling data
22:12 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/LocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
22:11 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/OldLocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
21:59 LeslieCarr: killing varnishhtcpd on cp3009 for realzies this time
21:58 RobH: mw23 offline, reinstalled with precise and signed into puppet, puppet throwing errors on initial run.
21:52 binasher: started udplog with correct mw config on fluorine
21:49 LeslieCarr: restarting varnishhtcpd on cp3010 - it appears to be locked up , netstat -l shows a huge Recv-Q for port 4827 and appears to be the previously discovered issue
21:43 binasher: restarting udp2log on fluorine
20:59 logmsgbot: reedy cleared profiling data
20:50 RobH: authdns-update
20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:20 RobH: mw23 was stuck in installer, restarting and setting up.
20:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:45 logmsgbot: reedy synchronized wmf-config/ 'Enabling wikidata client on itwiki and hewiki'
19:42 ottomata: added updated udplog_1.8-5 .debs to apt.wikimedia.org. No real changes, these now include the packet-loss program in the package.
19:22 logmsgbot: reedy synchronized wmf-config/
19:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase/
19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf8
18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:48 Ryan_Lane: downgrading memcached to ubuntu version on virt0
17:18 Reedy: Created wb_changes_dispatch on wikidatawiki
16:55 Reedy: Created sites/site_identifier tables on hewiki and itwiki
16:34 cmjohnson1: mw1072 going to down to check h/w state currently failed rt4381
15:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 44395) Allow bureaucrats to remove the translateadmin group on wikidatawiki gerrit 46011'
15:37 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 44506) raise throttle for an Israel editthon gerrit 46747'
12:54 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/GeoData
12:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/GeoData
05:55 Tim: rebooted kaulen, was not responding on network or serial console
03:11 mutante: wikimania.asia redirect now works, it did not before because of a bug in apache-graceful-all not actually restarting Apaches in eqiad
03:07 mutante: reloading Apaches in eqiad using manual dsh (because apache-graceful-all sanity check has a bug)
02:55 logmsgbot: catrope gracefulled all apaches
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 30 02:52:03 UTC 2013
02:39 logmsgbot: dzahn gracefulled all apaches
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Jan 30 02:28:22 UTC 2013
00:44 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:43 mutante: sync InitialiseSettings.php for bug 44473
00:19 binasher: restarted pdns-recursor on dobson
00:13 logmsgbot: reedy synchronized wmf-config/

January 29

23:54 logmsgbot: reedy synchronized wmf-config/
23:21 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
23:17 logmsgbot: pgehres synchronized php-1.21wmf7/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
23:00 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable Special:Nearby everywhere'
22:51 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
22:48 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
22:47 Tim: rotated SAL wikitech page - test logging

Server admin log

February 27

February 26

February 25

February 24

February 23

February 22

February 21

February 20

February 19

February 18

February 17

February 16

February 15

February 14

February 13

February 12

February 11

February 10

February 9

February 8

February 7

February 6

February 5

February 4

February 3

February 2

February 1

January 31

January 30

January 29

Archives

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Toolbox