Server admin log/Archive 20

March 24

19:38 mark: Brought esams Sun V20z servers back up - restored rr.esams and upload.esams LVS
19:24 mark: Bringing lily back up
19:07 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22924 - Enable the Collection extension at el.wikipedia'
18:40 logmsgbot: jeluf synchronized live-1.5/robots.php
18:39 domas: disabled keepalive on singer
18:37 river: delegated 1.24.10.in-addr.arpa to ha-dns-auth.toolserver.org
18:17 Rob: techdetail: lowered maxclients on singer apache2 and disabled secure site in apache configuration, do not rollback changes to normal until after esams is back online
18:16 Rob: techlbog back up, secure is NOT up, and is not expected to come back up, until the esams outage is resolved
18:01 Rob: disabled secure on singer, as everyone trying to use it overloaded both secure and blogs.
18:01 Rob: Secure is offline
17:51 mark: http://techblog.wikimedia.org/2010/03/global-outage-cooling-failure-and-dns/
17:50 mark: Reduced Negative TTL setting in SOA record of wikimedia.org from 1H to 10M
17:40 domas: NXDOMAINs with 1h TTL served on all records for short moment
17:40 Rob: globally (robs bad)
17:40 Rob: pending techblog post regarding the issue with accessibility in EU
17:25 mark: DNS scenario dir esams-down was empty, all zone files missing. Fixed.
17:18 mark: esams down due to overheating issues; moving traffic to Florida
14:18 mark: csw2-oe12-esams connected to iris's serial console (temp)
04:24 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php
04:21 Andrew: Running scap to update LiquidThreads beta
04:17 Andrew: Preparing to update LiquidThreads beta (production) to alpha version.

March 23

22:16 apergos: latest patches and reboot for ms5
22:13 Rob: sq40 back online, sq41 reinstalling
21:50 Rob: reinstalling sq40 & sq41
21:27 Rob: sq38 & sq39 reinstalled and online
20:43 Rob: taking down sq38 & sq39 for reinstallation
20:43 Rob: sq36 & sq37 back online and in rotation
19:39 Rob: sq36 & sq37 being reinstalled
19:35 logmsgbot: jeluf synchronized live-1.5/robots.php 'change charset to utf8'
19:32 Rob: sq34 and sq35 (not 25)
19:32 Rob: sq34 and sq25 back online and in rotation
19:27 logmsgbot: jeluf synchronized all.dblist
19:24 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.hooks.php 'r64086'
19:24 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.php 'r64086'
19:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Add vector-collapsiblenav as a hiddenpref'
19:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed more group permission for closed wikis'
18:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove all simplewikibooks configs from Initialisesettings'
18:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'REVERT Test case for simplewikibooks'
18:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'Test case for simplewikibooks'
18:28 Rob: sq34 & sq35 coming down for reinstallation
18:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'REVERT Test case for simplewikibooks'
18:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'Test case for simplewikibooks'
18:19 Rob: sq33 is reinstalled and doing quite fine.
16:02 Rob: puppet stopped on sq34-sq50, DO NOT RESTART PUPPET. These hosts need reinstallation before puppet can be restarted, or squid changes can be deployed to them.
15:52 Rob: changed the setting for pmtpa to change partitioning schema and up the cache size. Do not deploy old squids with this information until Rob finishes reinstalling them
15:52 Rob: changed squid config in text squids for hard disks update on reinstalls
12:01 logmsgbot: jeluf ran sync-common-all
11:59 logmsgbot: jeluf synchronized closed.dblist

March 22

21:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#21213 - logo for ace wikipedia'
18:16 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22657 updating logo'

March 21

20:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22882 - Add a patroller group to fr.wikisource'
20:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22454 - Patroller and autopatrolled flags on es.wikipedia'
16:48 domas: db28 has more memory errors: http://p.defau.lt/?IvvVKaNdi05v1yfwLiHWVw
16:44 domas: dist-upgraded db28 to karmic
12:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'bgwiki group permissions fixed'
11:37 logmsgbot: jeluf synchronized closed.dblist
11:04 JeLuF: removed tokipona wikis from all.dblist
11:02 logmsgbot: jeluf ran sync-common-all
10:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary'
10:03 logmsgbot: jeluf ran sync-common-all
09:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'UNDO: 22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary - pages still exist in these namespaces'
09:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#16857 - Request to switch over logo for sanskrit language wikipedia localisation'
09:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22283 - Add "Author" namespace to Telugu Wikisource'
09:26 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22329 - Redirect WV to Wikiversity space for English Wikiversity'
09:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary'
09:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22481 - Namespace alias at mwl.wikipedia.org'
09:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22481 - Namespace alias at mwl.wikipedia.org'
06:57 JeLuF: bugzilla.wikimedia.org
06:57 JeLuF: Changed kaulen config to include ServerAlias bugzilla.wikipedia.org for

March 20

21:21 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed sysop access to stabilization except on wikis already have config rows. This page should not be enabled by default.'
19:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22549 - Enable email on watchlist change on fr.WV'
19:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22699 - Autopatroler at cs.wikipedia'
18:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/mc.php 'srv98 down'
18:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/mc.php 'srv119 down'
14:53 apergos: tossed a few files from tridge to make space, a very short-term solution
14:13 JeLuF: Bug 22681: migrated users on bgwiki dawiki enwiktionary hewiki hrwiki itwiktionary from groups autopatrol and autopatroller to group autopatrolled
14:12 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22681 - Clean up/unify autopatrol/autopatrolled/autopatroller group names'
13:38 JeLuF: created Italian planet at planet.it.wikimedia.org
12:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22909 revert'
12:15 JeLuF: restarted morebots
12:15 JeLuF: added it.planet.wikimedia.org to DNS
12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22669 - Logo change for az.wikiquote.org'
12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/abusefilter.php
12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22909 Liquidthreads for huwiki'
12:15 JeLuF: jeluf synchronized php-1.5/languages/messages/MessagesKrc.php 'MessagesKrc from trunk, for bug:22657'

March 19

17:53 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'iframe display bugfix'
17:53 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.js 'iframe display bugfix'
17:52 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.js 'iframe display bugfix'
17:44 Fred: getting BUFFER IO ERRORS on srv169. JFS refuses to fix itself.
17:39 Fred: srv169 is having problems with sda6 (/a)... working on it and trying to salvage.
07:36 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'bug 22624 hack'
07:05 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 're-adding cluster1+2 for bug 22624'

March 18

23:36 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/language/FlaggedRevs.i18n.php 'deployed r63938'
23:35 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 'deployed r63938'
22:27 logmsgbot: root ran sync-common-all
21:11 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 'deployed r63929'
20:34 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'update flaggedrevs_labs config'
20:31 AaronSchulz: deployed r63927 (flaggedrevs_alpha)
16:14 Fred: security updates of apaches in progress
06:22 logmsgbot: tstarling synchronized php-1.5/includes/StubObject.php 'reverted profiling hack, I assume whatever information you wanted to get out of this, you've got now'
06:19 logmsgbot: tstarling synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.css 'revert whitespace-only change'
05:57 logmsgbot: tstarling synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'revert whitespace-only change'
05:55 logmsgbot: tstarling synchronized php-1.5/extensions/LiquidThreads_alpha/pages/ThreadPermalinkView.php 'reverted debugging hack'
00:33 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding override for strategy and usability wiki for wmgNoticeProject'
00:05 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Changing mapping of commosn under wmgNoticeProject to just the string commons'
00:03 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Commenting out commons from wmgNoticeProject to fix loading'

March 17

22:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version'
22:51 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:27 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63888'
22:20 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63887'
22:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/OptIn.i18n.php 'r63885'
22:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bump style version appendix'
22:00 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63883'
21:41 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 21005'
21:30 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for usability deployment'
21:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable experimental Usability features but add them to $wgHiddenPrefs'
21:08 RoanKattouw: Running scap
21:05 RoanKattouw: Updating UsabilityInitiative to trunk state cluster-wide
20:39 RoanKattouw: Pushing UsabilityInitiative software update to testwiki

March 16

20:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'removed ex-fedora clusters, should all be unreferenced now due to RCT completion'
19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22650 - Enable the AbuseFilter on the French Wikipedia'
19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php ' Bug 22650 - Enable the AbuseFilter on the French Wikipedia'
17:19 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed redundant tab settings'
08:05 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
08:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Reactivate LiquidThreads live preview'
05:01 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22691 - some fixes on de_wikiquote'

March 15

23:49 Fred: upgraded puppet on ms7 to the recent version that doesn't have scary stupid bugs in it!
21:58 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22663 - Import sources for frwiktionary'
21:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22694 - definition of import sources for mg.wiktionary'
21:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22817 - Create Artikeldata namespace on dawiki'
21:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22825 - Set $wgNamespacesWithSubpages[NS_MAIN] = true on sewikimedia'
21:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22833 - Enable multipage upload form for English Wikiversity'
21:01 logmsgbot: jeluf synchronized live-1.5/robots.php
20:48 logmsgbot: jeluf synchronized docroot/meta/favicon.ico
20:22 logmsgbot: jeluf synchronized docroot/incubator/favicon.ico
20:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22648 - Add portal namespace to kw wikipedia'
20:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22674 - Update Special:Import options at English Wikibooks'
19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22406 - Enable Extension:Quiz on zhwiki'
19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22615 - Import sources for sv.wikibooks'
19:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22787 - TZ for Estonian projets'
19:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22831 - wgBlockAllowsUTEdit for kowiki'
19:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22827 - Logo change for pcdwiki'
17:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Resyncing because I suspect some boxes didn't pick this up'
17:00 hcatlin: [mobile] code push complete!
16:52 hcatlin: [mobile] pushing code update for adding ext, cz, mr, ca, bn, kl, mn, scn, si, and ta langauge codes
16:29 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22834'
15:51 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22473'
15:27 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22791'

March 14

04:52 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Make Vector and usability enhancements default on liquidthreads_labswikimedia'
04:46 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bumping style version'
04:43 Andrew: Scapping to deploy LiquidThreads alpha update
04:40 Andrew: Preparing for LiquidThreads alpha update to trunk state

March 13

18:42 Rob: sanger is back online and working
18:33 Rob: sanger moved, coming back online
18:21 Rob: thats sanger, not singer.
18:21 Rob: taking down singer to move it up in the rack, during this downtime (approx 15 min), wikimedia.org email will be unavailable.
18:20 Rob: mchenry is back online and dealing with its 10 minute mail backlog.
18:14 Rob: mchenry moved, bringing it back online
18:01 Rob: mchenry host coming down to move its rack location. Downtime will be approx. 15 minutes. (all mailing list relays may not work and thus new mailing list entries will back up until this move is completed.)

March 12

19:23 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling .odg extension for uploads'
18:49 Fred: moved stats.w.o from Zwinger to Spence
12:00 RoanKattouw: storage2 (download.wikimedia.org) was down between 11:40 and ~11:53 UTC, coincides with load spike
07:43 apergos: really edited the right copy of worker.py this time (the copy in /backups :-P)
06:19 apergos: editd worker.py in place on snapshot3 to remove --opt and replace all except lock options individually (it was running with that on db12 again, with high lag). shot that worker process, lag and thread count going down now
02:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
02:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'reducing load on db12 again, seems to be lagging again'
02:38 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'increased general load on db12'
02:34 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
02:32 Tim: db12 caught up, repooling
02:15 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'subpages for Grants namespace on meta (bug 22810)'
02:02 tomaszf: killing 20100312 xml snapshot run for enwiki due to unnecessary locking of pagelinks table on db12
01:43 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'moving db12 back into the "dump" query group so we don't kill db26 too'
01:41 Tim: note: mysqldump query for pagelinks table is running on db12, presumably that is the cause of the slowness and network spike
01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
01:32 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'brought db26 into rotation as enwiki watchlist/contribs server, to replace db12 after cache warming'
01:30 Tim: network spike on db12 at 01:00. Going to depool it in case it's going to try the same trick as last time
01:21 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'resyncing for bug 22810 (stupid ssh-agent :-P)'
00:47 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'Grants namespace on meta, bug 22810'
00:17 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Removing foundation wiki from wmgDonationInterface'

March 11

22:03 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22669 rollback'
22:02 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22669'
21:36 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22404'
21:19 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'subpages for templates on foundationwiki (#22484)'
20:37 atglenn: re-enabled replication on media server (ms7 -> ms8), we are back in business
16:43 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22089'
12:27 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
11:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deploy r63591, fixes for LiquidThreads comment digestion'
11:21 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Deploy r63591, fixes for LiquidThreads comment digestion'
01:34 Tim: reinstalled wikimedia-task-appserver on fenari to fix symlinks /apache and /usr/local/apache/common
00:26 atglenn: playing catchup on replication now from ms7-ms8 zfs send/recvs running as root

March 10

22:59 atglenn: patched and rebooted ms8, working on snapshot issue
21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.i18n.php 'r63575'
21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.php 'r63575'
21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r63575'
18:54 Fred: removed duplicate ACL for 208.80.152.157 from text-squids

March 9

23:32 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r63523 into LiquidThreads production'
23:26 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63524 into LiquidThreads alpha'
23:26 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63523 into LiquidThreads production'
23:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63524 into LiquidThreads alpha'
23:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63523 into LiquidThreads production'
23:15 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r63154 into LiquidThreads production'
23:13 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63154 into LiquidThreads alpha'
20:22 rainman-sr: this morning's search api outage seems to be connected with index update on search8 which triggered high GC load and stuck threads, removing enwp spell index from search8
15:37 hcatlin: message before was referring to mobile site.
15:37 hcatlin: deployed new languages. cs homepages.
14:58 domas: forgot to log - yesterday's API fail was repeated today too, all API nodes were blocked on talking to lucene, and search cluster was idle too, and...
14:54 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'force recache to solve l10ncache issue'
14:51 logmsgbot: catrope synchronized php-1.5/includes/LocalisationCache.php 'r63417'
00:41 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.hooks.php 'deployed r63447'
00:40 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.class.php 'deployed r63447'

March 8

23:59 logmsgbot: tstarling synchronized php-1.5/includes/Sanitizer.php 'r63426'
23:57 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
23:52 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed labs config cruft'
23:32 AaronSchulz: deployed updates MFT to FlaggedRevs_alpha (r63444)
23:30 Fred: updated search nodefile to contain search[13..20].
23:10 AaronSchulz: finally got labs msgs working it seems
23:09 logmsgbot: aaron synchronized php-1.5/wmf-config/CommonSettings.php 'do not override flaggedrevs_alpha msg dirs for lab sites'
22:40 logmsgbot: tstarling synchronized php-1.5/thumb.php 'r63432'
22:04 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/language/OldReviewedPages.i18n.php
21:54 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r63423'
21:41 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
21:38 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php
21:36 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'disable LU on labs sites'
21:09 logmsgbot: catrope synchronized php-1.5/includes/LocalisationCache.php 'r63417'
20:57 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'revert, no effect on msg issue'
20:53 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'settings labs cache dir'
20:31 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Switched labs sites to FlaggedRevs_alpha'
20:30 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'custom cache dir config for labs sites'
19:18 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb
18:15 Fred: disabled regular http check for mobile1 (leaving the longer timeout one in place)
13:58 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-et.cache 'Sync file *after* touching it'
13:55 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-et.cache 'Sync LU cache file'
13:52 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'Try again'
13:48 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesNds_nl.php 'Fix message that didn't get picked up'
13:46 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'See if this works'
13:20 RoanKattouw: Rebuilding all l10n caches to prep debugging a LocalisationUpdate issue
13:16 domas: something was wrong with sq31/sq32 - cpu/network shot through roof, can't correlate load
10:42 domas: *sigh*, squid conf: added highPrecedence to defaults, so it is picked up by configuration builder. also, checked in files into RCS (had to take over locks from mr.Starling, sorry!). unbroke test.wikipedia API
09:43 domas: where did test api,php fix go?!
04:38 Fred: removed Bart's ACL in squid cache conf since it was a duplicate ACL.

March 6

22:59 mark: Increased membufs on all backend squids
06:58 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabled FlaggedRevs_alpha due to outdated messages issues'
05:40 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php '$wgFlaggedRevsStylePath for labs'
05:35 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
05:33 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Set labs site to use flaggedrevs_alpha'
05:26 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set labs site to use flaggedrevs_alpha'
05:10 logmsgbot: aaron ran sync-common-all
00:18 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22728 - create new user group on Greek Wiktionary'

March 5

18:48 Fred: removed duplicate ACL from squid backend config.
18:11 Rob: restarted npre daemon on ms1 since it wasnt transmitting to nagios.
00:51 logmsgbot: aaron ran sync-common-all
00:23 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Restore Monobook for nl.wikimedia.org'

March 4

19:44 Tim: installed NRPE and RAID utils on ms1
19:40 Tim: removed db19 from the mysql node group and nagios since it is being used for varnish now
19:29 Tim: fixed prototype.wikimedia.org (nagios critical), was waiting 10s on every request due to memcached being down

March 3

16:16 apergos: reenables gmetric on ms8 a little bit ago, scrub finished with no errors (yeah right(
06:06 apergos: replication off again tonight, zfs nicely borked on ms8, zfs destroy snapshot X gives message "cannot destroy X, dataset already exists" . time for sun contract and call them
05:53 apergos: gmetrics turned off again on ms8, running zpool scrub export
05:40 apergos: reset SYs on ms8 worked (slowly). gmetrics reenabled. (and I thought morebots was just being a bad bot)
05:39 apergos: ( :-P ) starting catchup for the last day's lost replication ms7 -> ms8
05:22 apergos: reboot failed, bootadm hung probably waiting on zfs. stop SYS claims to work but subsequent show SYS shows power state on.
05:14 apergos: rebootiing ms8 to clear zfs deadlock

March 2

23:32 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'r63191'
23:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/OptIn.php 'r63191'
23:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Tweak OptIn settings'
22:36 RoanKattouw: Rebuilding localisation cache
20:15 Tim: on hume: started recompressTracked.php for all wikis, to fix any newly-tracked pointers from fixBug20757.php and to complete enwiki
03:48 atglenn: leaving replication and gmetrics turned off on ms8 for thenight, will go back to looking at scat and mdb output tomorrow.
02:27 atglenn: turning off gmetric on ms8, the dfs are piling up behind a deadlocked zfs list
02:13 atglenn: turn of ms7 replication to ms8 temporarily to work out a zfs deadlock issue on ms8

March 1

12:53 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
12:51 river: removing db17 from rotation for ts dump
12:51 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
11:48 domas: recovered locke, ran log collection with per-process limit of 200MB
11:39 domas: locke was killed by awks using gigabytes of memory, mwahaha.

February 27

02:09 Tim: rewrote domas's api.php fix and deployed my version
01:56 logmsgbot: midom synchronized php-1.5/includes/LocalisationCache.php 'removing profiling hooks'
01:42 domas: fixed api.php on test/srv124

February 26

23:40 Tim: disabled the temporary squid purge script, I'm assuming it's done its job now
19:43 logmsgbot: catrope synchronized php-1.5/includes/api/ApiBase.php 'r63014, r63015'
18:24 Fred: bugzilla now residing on Kaulen... Updating server roles.
18:07 Fred: switching bugzilla cname to new server. (both servers are pointing to the same DB, so switch should be transparent to most)
17:28 logmsgbot: midom synchronized php-1.5/includes/LocalisationCache.php 'profiling'
17:26 logmsgbot: midom synchronized php-1.5/includes/diff/DifferenceInterface.php 'removing profiling hook'
17:23 logmsgbot: midom synchronized php-1.5/includes/StubObject.php 'tracehooks'
17:21 logmsgbot: midom synchronized php-1.5/includes/StubObject.php 'tracehooks'
14:38 RoanKattouw: logmsgbot is missing in action AGAIN, starting it as me on fenari
14:37 RoanKattouw: Synced r63008
12:17 domas: *sigh*, recovered locke after OOM, added few entries to limits.conf

February 25

23:48 Tim: running the new fixBug20757.php on all wikis
21:48 RoanKattouw: Removed empty line from the end of all.dblist, broke account creation ^^
20:18 RoanKattouw: Removed those closed wikis again, db lists back to normal
20:08 RoanKattouw: Temp adding some closed wikis to all.dblist to be able to dump them (again)
19:01 Rob: removed old servers will, bart, etc from DNS
18:32 Rob: pushing more dns changes for LOM on new servers.
00:59 Tim: made the squid purge script on locke also purge redirects from /w/index.php?, for bug 22639
00:32 Tim: installed ack-grep on fenari

February 24

20:48 Rob: IP conflicts on alsted/ersch/formey/kaulen. I must have typo'd something in one of their DRAC configurations because they are all messed up. will go into DC and physically review settings via console.
16:10 Rob: had formey and alsted ip info mixed up in dns, updating dns
13:13 RoanKattouw: Adding optin_survey table to wikis that didn't have it
01:38 atglenn: ganglia 3.1 packages up on ms7 and ms8 (see atg for solaris packages for other boxen)
01:20 domas: that was fun, let's do it again!
01:17 mark: Implemented udp2log filter on locke that pipes redirected URLs to a modified version of purgeList to clean the squid caches
00:59 Rob: added austin to the mortals user category for shell access on puppet, updating singer's account list
00:30 mark: Fixing wikipedia.be vhost ServerAlias setting in redirects.conf
00:14 domas: redefined tiertwo acl in squid to allow fenari/zwinger purges, purged special:random, restarted mobile1 stuff

February 23

23:41 domas: stopped nginx on en.m until we resolve special:random issue
20:53 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22628 - Logo of Chamorro Wikipedia'
17:23 Rob: drac setup on new servers ersch, formey, alsted, and kaulen. still need port allocation.
16:47 Rob: forgot to add mgmt, updating dns
16:43 Rob: updating dns for new servers ersch, formey, alsted, & kaulen
15:37 Rob: FINALLY got in the last RMA drive for storage1, arrays were toast (losing 1/2 the drives in a raid10 and having the unlucky chance of them being paired), reinstall in progress
12:50 RoanKattouw: Cleaned up my dblist voodoo, everything should be back to normal now
12:41 RoanKattouw: Fixed sync-dblist, was broken
12:31 RoanKattouw: Syncing pmtpa.dblist anyway, seems I have to
12:24 RoanKattouw: Temporarily adding wikis listed in bug 20325 to pmtpa.dblist so I can export their contents. Please don't sync this

February 22

23:40 Tim: restarted replication on ms2
23:39 Tim: ms1 is up with a full copy of ms3 data, now replicating cluster22/rc1
21:09 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Revert'
20:30 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22522 - Allow bureaucrats to remove sysop flag on Simple English Wikitionary'
20:09 Rob: finished the setup of srv217 (just needed puppet fixed and files updated)
20:02 logmsgbot: root ran sync-common-all
20:01 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set vector as default skin on nlwikimedia'
19:44 logmsgbot: root ran sync-common-all
18:10 Fred: ganglia is going to hiccup a little for the next few minutes while I restore data... "Don't panic"
07:53 Tim: rebooted srv206 via drac
07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/mc.php 'removing srv206, is down'
06:19 Tim: increased multicast TTL from 1 to 3 in gmond_template.erb, to fix ganglia on ms*
05:54 Tim: started ms2 -> ms1 copy using nc+tarpipe
05:30 Tim: shutting down mysql on ms2 for copy to ms1
05:09 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'depooling ms2 for copy to ms1'

February 21

02:43 Fred: puppettized user gmetric

February 20

22:37 mark: Installed Ubuntu 8.04 on ms1 - now ready for ES deployment

February 19

18:48 Fred: removed memcached node srv193 and replaced with srv233
16:52 Rob: updated InitiliseSettings with bug 22585
01:31 Fred: puppetized gmond.conf generation using templates

February 18

23:51 RoanKattouw: Fixed search script on yongle, removed UA check exemption for yongle
23:51 RoanKattouw: logmsgbot is broken again
04:07 Tim: re-running storageTypeStats.php on enwiki to identify rows with old_flags='object,utf-8', which need to be handled properly in fixBug20757.php

February 17

17:25 Rob: pushed updates to wordpress installations
17:04 RoanKattouw: Synced wmf-config/checkers.php 'Allow missing UA from yongle'
17:04 RoanKattouw: Restarted logmsgbot
16:47 Fred: test
07:24 Fred: there is a problem with ganglia. It is being worked on.
00:15 RoanKattouw: Resynced srv151, was returning empty responses

February 16

23:31 Fred: upgrading gmond to 3.1.2 everywhere. However, due to the newish module structure, there is a potential that ganglia will hickup while puppet does its job...
22:03 logmsgbot_: mark synchronized php-1.5/wmf-config/checkers.php 'Exceptions'
20:30 Rob: srv127 is online, but not in LVS. Its cert was accepted on sockpuppet, but puppetd --test results in a cert failure on retrieval from sockpuppet.
20:17 Rob: rebooting srv127
20:15 mark: Increased membufs to 40 per COSS dir on the pmtpa upload backend squids
19:51 mark: Increased membufs per COSS dir from 10 to 20 on the new pmtpa squids
18:10 apergos: but documentation can save people precious time when things are on fire
18:02 mark: Documentation is not a substitution for thinking
16:44 mark: Fixed puppet on most servers
16:23 domas: anyone knows why mysqldump on snapshot3 is locking tables? maybe --single-transaction could work better?!!!?
16:22 RoanKattouw: Strike my last, I hear it'll fix itself in a day
16:21 RoanKattouw: Oops, meant to type srv187-189
16:21 RoanKattouw: Ganglia not picking up data for srv1987-189, 193, 214-218, 250-253, 257 even though the boxes are up and gmond is running; been like this for 3 days
15:21 mark: Removing all puppet certs/private keys on all machines
14:55 mark: Puppetmaster is screwed up by a wrong command that deleted all files under /var/lib/puppet
14:25 logmsgbot_: midom synchronized php-1.5/includes/diff/DifferenceInterface.php 'instrumenting costs by revision pairs'
13:46 logmsgbot_: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22444 and 22330 enabling collection on dawiki and ruwiki'
13:22 domas: oh wait, it does work, something else....
13:16 domas: apparently my squid rule for Apple browsers didn't really work, hence the continuing spikes
03:16 domas: blocked MacOSX Atom syndication on the edge, this will save terabytes of diskspace on unsuspecting Safari user computers :-)
02:13 logmsgbot_: tstarling synchronized php-1.5/wmf-config/checkers.php

February 15

23:42 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'uninstrumenting'
23:09 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php
23:09 domas: banned all UA-less requests
22:53 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 'un-disabling UA check on API'
22:14 RoanKattouw: Resynced srv114, was returning empty responses
22:10 logmsgbot_: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)'
22:10 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)'
22:02 logmsgbot_: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:58 logmsgbot_: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r62553'
21:39 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiMain.php 'r62550'
21:30 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'raised wgSearchSuggestCacheExpiry from 1200 to 86400'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62548'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiMain.php 'r62548'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/DefaultSettings.php 'r62548'
20:54 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiBase.php 'r62546'
20:54 logmsgbot_: catrope synchronized php-1.5/api.php 'r62546'
20:53 domas: Roan and me are doing some perf engineering on opensearch & related stuff. \o/
20:40 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'readding stub trace'
20:34 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 'perf comment action'
20:30 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'removing trace hooks'
20:28 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php
20:24 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62543'
19:32 mark: Allowing API requests to be sent to the regular app servers temporarily
19:27 mark: Restarted sq31 and sq31 API backend squids with slightly reduced memory settings
19:18 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php
17:51 Rob: set srv127 to false on lvs3, will re-enabled once in DC to reboot it manually
17:49 logmsgbot_: root synchronized php-1.5/wmf-config/mc.php 'srv127 down taking out of mc rotation'
16:02 RoanKattouw: Resynced srv134 to fix HTTP 500
15:13 RoanKattouw: Resynced srv232, was serving HTTP 500s
02:02 mark: Fixed stale PHP file removal apache cron job in Puppet

February 14

19:25 mark: srv206 spontaneously rebooted
17:12 mark: Added sq32 to the API backend squid group (with an empty cache)
16:38 mark: Confined all API requests to sq31 backend Squid
12:54 RoanKattouw: logmsgbot is broken

February 13

17:44 mark: Made 64 bit java exception for search13 in puppet
17:37 mark: Converted srv214-218 and srv86-87 into API servers
17:02 mark: Throwing all API requests at the new API app server cluster
16:49 mark: Converted srv187-srv189 and srv250-srv253 into API app servers
15:46 mark: Created new Ganglia group "API application servers", deploying with Puppet
15:31 mark: Created API app servers group in Puppet, deploying on srv254-257 to start with
15:24 mark: Created API LVS service on lvs3 and restarted PyBal
15:10 mark: Added DNS entry for api.svc.pmpta.wmnet (LVS ip)
14:18 mark: Enabled more verbose logging on all squids: debug message level 1 for all message categories
14:18 mark: Disabled cache digests on all squids
13:58 mark: Removing sq30 and lower from the Text CARP pool
13:51 mark: Reducing persistent_connection_timeout and request_timeout from 1 min to 20s on all frontend squids, to reduce FD usage
12:50 mark: Restarting knsq* frontend squids with FD limit raised from 32k to 64k
12:23 mark: Grown filesystem /dev/data/ES (/a) on ms3 by 90G, leaving another 98G free
00:17 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Sync CSS style versions'
00:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
00:10 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62403'
00:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62400'
00:01 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62400'

February 12

23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Try deploying r62397 again'
23:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r62397'
23:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62397'
23:22 mark: Deployed sq86
22:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62388'
21:12 mark: Implemented new routing policy for AS43821
21:01 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:59 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62382'
19:15 Rob: sq67 has both disks installed now (had to get one replaced) and has no OS installation yet. holding off on install, not sure if mark wants this up as a squid or for varnish development
18:22 atglenn: resstarting transfer of data (one chunk) from ms8->ms5->ms6, running in screen as root on all three hosts using nc
16:18 Rob: setup project1 and project2 boxes for flaggedrevsdevelopment and project mangement suite testing
15:12 mark: Upgraded our AMS-IX connection from 2x 1G to 10G along the way
15:12 mark: Reloaded br1-knams twice to fix a CAM partitioning problem; not sufficient IP next hops for 8-path routes
11:34 mark: Filtering all prefixes to/from AMS-IX peers
02:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 're-enabling WikiEditor'
02:04 Tim: installed openssh-server on bayes, Erik Z apparently uninstalled it with aptitude
01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'disabled WikiEditor due to complaints about bug 22428'
01:06 atglenn: shoveling over some data from ms8 to ms6 via ms5 :-/
00:48 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Disable live preview entirely, broken by CentralNotice and DismissableSiteNotice'
00:37 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable DismissableSiteNotice for lqt.labs, breaks LiquidThreads live preview'
00:17 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
00:07 Andrew: Running scap
00:02 Andrew: Updating LiquidThreads alpha to trunk state, deploying with scap in a few minutes

February 11

22:06 Fred: implemented disk space check for tridge
21:10 mark: Pooled sq71-85 with full CARP weight (30)
21:06 mark: Pooled sq79-85 frontends in LVS with full load (30)
20:59 mark: Pooled sq79-85 frontends in LVS with load 1
20:51 mark: Pooling sq79-85 in CARP on all frontends, with low CARP weight 10 instead of 30
20:30 mark: Pooling sq79-85 in CARP on frontend sq51 to seed the cache
19:39 atglenn: replication every 15 minutes enabled from ms7 to ms8
19:31 mark: Removed sq16-30 from the text frontend squid pool in LVS - these servers will soon be decommissioned
19:28 mark: Pooled sq71-78 frontend squids in LVS with full load (30)
19:21 mark: Pooled sq71-78 frontend squids in LVS with low load (1) to seed caches
18:30 mark: Pooling sq71-78 backend squids on all frontends, with lower CARP weight (10 instead of 30) to seed the caches
18:04 mark: Deployed new squid config to sq58-66 frontend squid, to seed the caches of backend squids sq71-78
17:27 mark: Deployed new squid config to sq66 frontend squid, to seed the caches of backend squids sq71-78
17:24 mark: Deployed Squid with correct configuration on sq71-78
11:08 RoanKattouw: Load spike on 4CPU Apaches ended, rr.knams back up, downtime seems to be over
11:05 RoanKattouw: Another load spike on the 8 CPU Apaches, approx 11:02-11:-05 UTC
11:00 RoanKattouw: Apache load spike started around 10:42 UTC and coincides with a load dip followed by a monitoring blackout on the Kennisnet Squids and a load spike on the Tampa Squids
10:58 RoanKattouw: 8CPU Apaches stopped going bonkers all of a sudden according to Ganglia, 4CPU ones still have high CPU usage
10:56 RoanKattouw: Site went down, all Apaches have high CPU usage

February 10

22:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix to deploy usability changes sitewide'
22:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62275 to test'
21:28 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264 for real. Running svn up first helps'
21:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264'
21:03 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
21:02 Andrew: Switching strategywiki to opt-out LiquidThreads, rather than opt-in
20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'Deploy r62262'
20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62262'
19:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:21 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62257'
19:05 Rob: storage1 disks replaced, except for the newly dead one, rma placed.
18:39 Rob: shutting down storage1 to replace its bad OS mirror disk
16:52 mark: Fixed MySQL permissions on srv186 as well
16:15 mark: Rigged puppet to deploy a PHP mail.ini file to the apaches, which sets it to call sendmail with -f <> (sending mail with empty envelope sender, like bounces)

February 9

22:06 Andrew: LiquidThreads production software updates completed successfully.
21:58 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Dispatch.php
21:57 mark: Firewalled one ip on singer which was overloading Apache, looked like a Wordpress attack
21:50 mark: Lowered MaxClients setting to 250 on singer
21:47 Andrew: Running scap to deploy LiquidThreads updates
21:45 Andrew: Updating LiquidThreads production deployments to the alpha version in the next few minutes.
21:44 mark: Power cycled singer
21:19 mark: Fixed MySQL system user & permissions on srv151-srv185
21:17 mark: Fixed MySQL instance on srv183
20:47 mark: Fixed MySQL instance on srv156
19:39 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:29 logmsgbot: catrope synchronized php-1.5/skins/common/edit.js 'Deploy r62190. Only for test for now, thank God for style versions'
17:01 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
17:01 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62187'
16:52 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
16:51 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62184'
14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
14:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62181'
02:14 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merge r62158'

February 8

23:25 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for r62041. Varnish needs this or Tampa will serve stale JS off bits'
23:06 mark: Deploying Exim on all application servers as well
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'Deploy r62141'
21:51 mark: Added search3 and search9 to search_pool_1 on lvs3
21:48 mark: Deploying Exim as a simple mail sending-only MTA (with queuing enabled) on squids, image scalers, search servers and core databases
21:38 Andrew: LiquidThreads wikinews rollout complete, though there was some buggage related to replication that had to be resolved.
21:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:27 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:21 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:18 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
21:18 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php
21:13 Andrew: Starting LiquidThreads deployment on enwikinews.
20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Track NTOC and dialogs as well in PrefStats'
13:04 Tim: running fixBug20757.php on enwiki, in a screen on hume
12:51 Tim: locke went into swap half an hour ago and is now dead. Trying reboot.
07:38 Tim: stopped recompressTracked.php for now due to bug 20757
07:04 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'r62120'
06:18 Tim: cleared out db9 root partition (which had 370MB free) by moving 4GB of SQL files from /root and /home/tfinc to /a/backup/junk
06:01 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 're-added ms2 as an ES slave for rc1,cluster22 and the ex-fedora clusters, was depooled for ~6 months'
05:55 Tim: added grant for root@fenari to ms3/2
05:31 Tim: started apache on srv213, srv161 and srv240, was stopped for no apparent reason
04:52 logmsgbot: andrew synchronized php-1.5/includes/ChangesList.php 'Merge r62117'
04:37 Andrew: Deploying LiquidThreads alpha updates with scap
04:23 Andrew: Planning to deploy LiquidThreads alpha (liquidthreads.labs and test) in the next few minutes
01:15 Tim: added mysql and gmetric users on srv155 temporarily, to get those services back up
00:42 Tim: ran apt-get upgrade on srv155. Doing reboot test for new kernel.

February 7

23:36 Andrew: Clarification: Somebody should fix it, not I think it's fixed now.
23:35 Andrew: The issue of centralnotice pulling notices from the wrong place should be fixed now, though
23:34 Andrew: Moved /mnt/upload5/centralnotice/centralnotice.js and /mnt/upload5/centralnotice/wikipedia/centralnotice.js to centralnotice-old.js, should stop fundraising banners from appearing.
23:33 Andrew: Issue with fundraising banners appearing on random pages caused by clients with the URL http://upload.wikimedia.org/centralnotice/wikipedia//centralnotice.js?257z6
22:11 Andrew: Reports that fundraising banners are appearing on random pages for some users.
21:32 domas: we're running on single ES copy.
21:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'disabling broken storage1'
20:05 domas: removing all fixes I've been doing to recover mail. WMF staff will handle this tomorrow, or the problems will be gone by then.
19:00 domas: blackholed few thousand IPs on mchenry via OUTPUT chain (using -j REJECT action)

February 6

19:55 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php
01:09 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/CentralAuth.php 'Merge r61717, because Casey Brown asked me nicely.'
01:08 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/ApiQueryGlobalUserInfo.php 'Merge r61717, because Casey Brown asked me nicely.'
00:58 Andrew: Scapping to deploy r62055 'For usability initiative: merge r62041, r62043'

February 5

06:00 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 'Re-activate LiquidThreads email notifications, DoS has passed'
05:22 Tim: freed up a little space on ms3 by reducing expire_logs_days from 14 to 7, and running flush logs
02:45 apergos: cleared out /tmp on srv169...
00:23 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for jQuery UI stylesheet'
00:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for combined.min.css'
00:03 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'Resync this, doesn't seem to have been picked up right'

February 4

23:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62002'
23:50 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
23:37 RoanKattouw: Running scap to deploy new UsabilityInitiative code
22:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
22:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js 'r61995'
22:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable UsabilityInitiative_alpha on usabilitywiki'
22:20 logmsgbot: tstarling synchronized php-1.5/includes/api/ApiUpload.php 'disabled chunk upload'
22:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
22:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
22:05 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/images/wikiEditor/dialogs/insert-link-external-rtl.png
22:04 RoanKattouw: Deploying r61993 with individual sync-files
21:15 atglenn: turned off hourly snaps on ms7, we'll have daily plus the standard replication
18:49 atglenn: cleared /tmp on srv175 blah blah
18:36 atglenn: cleared out php* files from /tmp on srv163 (need cron job)
10:35 mark: Removed sq11-15 from the upload LVS pool
10:30 mark: Removed sq11-15 from the upload CARP pool
03:28 apergos: added /usr/local/apache/common/php to include path of php on fenari

February 3

23:56 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css
23:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
23:54 RoanKattouw: Deploying r61959 using individual sync-files
22:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Explicitly enable EditWarning by default here. Previously done in extension file'
17:32 mark: Set root password on all servers using Puppet
17:26 Rob: loaded sq68 full with 6 ssd, had to 'borrow' drives from the unracked squids in pmtpa awaiting the rail issue resolution.
17:26 Rob: connected sq67 and sq68 to secondary network ports as requested
16:43 Rob: had to update gmetad and add sq14 and sq17 into puppet as ganglia aggregator hosts. Currently not showing in ganglia, but should when puppet updates their files shortly.
16:37 Rob: updated gmetad_pmtpa in NFS store, as well as on zwinger and spence.
16:20 Rob: !log cleaned sq1-sq10 certificates off sockpuppet
16:10 mark: Rebooting sockpuppet for security upgrades
16:06 Rob: updating DNS to remove lvs3.wikimedia.org, as its technically lvs3.pmtpa.wmnet
16:03 Rob: removed sq1-sq10 in dsh groups, server roles, pybal (already done, just doublechecked), and now starting wipe and removing their network connections.
15:37 Rob: updated video plugin on techblog, had to comment out sumotv support line due to errors.
13:30 mark: Removed sq1-10 from the upload backend Squid pool, preparing for decommissioning
12:27 mark: Removed eiximenis from the text backend pool
12:21 mark: Restarted sq51 with hyperthreading disabled
11:31 Tim-away: exim was flooding its mainlog with errors like "User 'exim' has exceeded the 'max_user_connections' resource (current value: 150)". OTRS is apparently broken as a result. Set the limit to infinity to fix it.
11:10 Tim-away: debugging mysql connection errors from mchenry to db9
11:02 Tim-away: restarting exim4 on mchenry
10:57 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php 'r61922'
10:47 RoanKattouw: Mail delivery reported broken again; last mail delivery to mediawiki-cvs was 3h22m ago
10:14 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version'
10:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html
10:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.combined.min.js
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
10:09 RoanKattouw: Deploying r61919 with individual sync-files
07:45 Tim: killed eximstats on mchenry, was sending the machine into swap
06:02 Andrew: Running scap to deploy LiquidThreads alpha updates
05:09 Andrew: Planning to update LiquidThreads alpha to trunk state in the next few minutes.
03:31 Tim: removing refreshLinks2 jobs from the enwiki job queue with namespace=0, 100k of them put there by a nasty biography template
00:34 domas: restarted varnish on db19

February 2

23:26 mark: PyBal got into a confused state due to a duplicate LVS realserver entry. Set up LVS manually/statically for 30 mins to debug the problem. PyBal is now active again.
23:04 Tim: pmtpa text squids down
22:33 mark: Pooled sq59-68 in LVS (Text)
22:27 apergos: restarted apache on srv170 (corrupted apc cache)
22:25 mark: Increased CARP weight of the new Upload squids sq51-58 from 10 to 30
22:17 mark: Added sq59-66 to the Text pool of Squids, with low CARP weight (10) to seed the caches
21:44 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php
21:42 Andrew: temporarily disabling $wgLqtEnotif to mitigate mail server DoS impact on LiquidThreads post/reply speed.
21:42 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/NewMessagesController.php 'Deploy r61879'
21:42 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'Deploy r61879'
21:04 mark: Removed sq2-10 from the LVS pool
20:46 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version addition for usability alpha'
20:45 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js 'Deploing r61700'
20:38 mark: Added sq52-58 to the backend Squid pool with low CARP weight, to seed the caches
20:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22351: Enable Vector and toolbar on srwikinews'
20:36 mark: Increased Exim load average threshold for delivery on mchenry
19:09 Fred: rebooted singer (went down for unknown- at this time- reasons).
18:25 Fred: OS-installed sq52-ssq58
18:14 mark: OS-installed sq59-sq66
17:23 mark: Thrown sq66 backend squid in the Squid pool, CARP weight 30
06:01 Tim: deploying r61846 via scap
04:25 Andrew: Schema change successfully applied.
04:24 Andrew: Applying schema change thread_signature.sql to mediawikiwiki, strategywiki, testwiki
04:22 Andrew: Applying schema change thread_signature.sql to liquidthreads_labswikimedia
01:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Fix for UsabilityInitiative_alpha config'
01:18 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js 'r61842'
01:13 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Append to $wgStyleVersion when running usability alpha'
01:10 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61841 for real this time'
01:09 logmsgbot: catrope ran sync-common-all
01:07 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61841'
01:05 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61839'
00:51 RoanKattouw: Running scap
00:48 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61837'
00:40 RoanKattouw: Synced extensions/UsabilityInitiative_alpha by hand
00:07 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js
00:07 RoanKattouw: Syncing some CSS and JS files for usability deployment so bits.wikimedia.org will pick them up
00:06 logmsgbot: catrope synchronized php-1.5/skins/common/wikibits.js 'Sync changes to wikibits.js in r61558'

February 1

23:30 RoanKattouw: Testing new usability code on test
23:27 mark: Pooled sq51 frontend squid in the upload LVS cluster, weight 30
23:03 mark: Thrown sq51 backend squid in the Squid pool, CARP weight 30
20:50 Rob: srv217 shutdown until checked
20:50 Rob: srv217 not responsive to network, but LOM works and system is jut not working online, possible bad cable, will check in DC tomorrow
20:43 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Added 36 new Squid servers, removed bart'
20:39 mark: Deployed new wikimedia-base package that no longer manages sysctl.conf post Hardy, letting puppet handle it
20:10 atglenn: restarted webserver on ms7 to read new conf (fix bug 22321)
20:09 mark: Granted cary access to the OTRS db from fenari
19:27 RoanKattouw: srv9 reported down, pinging yields Host Unreachable. srv9 is not in Nagios but seems to be in use
18:51 Fred: swapped memcache node from srv145 to srv196

January 31

21:18 domas: db28 has a failed dimm

January 30

22:29 RoanKattouw: Scheduling more batches for import overnight using at(1), see http://commons.wikimedia.org/wiki/Commons:Batch_uploading/Geograph#Progress
22:13 RoanKattouw: Running image imports overnight on hume, three parallel import scripts started each five hours, see http://commons.wikimedia.org/wiki/Commons:Batch_uploading/Geograph for details
22:12 apergos: restarted morebots, must have died after last night's freenode irc move
04:40 Tim: fixed broken /etc/rc.local on wikitech
04:18 Tim: restarting mysqld on db12
04:00 Tim: analysis: mysqld on db12 hit a bug at 02:45 and froze, most threads in futex. The mysql client failed to set a read timeout, leading to db12 sucking up all available apache threads. Several squids became overloaded, presumably due to the large size of our 503 error messages.
03:21 Tim: restarted all apaches, thereby killing the reads which started before 3:11, possibly as early as 02:45. Site service resumes immediately. ps -lL on db12: http://p.defau.lt/?Jw1EBE0fnV4Rpxe9OP8yAw
03:18 Tim: finally thought to answer the question "are the apaches waiting for something or idle?" Strace confirms that they are waiting for db12 in read().
03:11 Tim: Having trouble believing db12 could be responsible for the CPU spike on the squids. Depooled db12 just in case.
02:56 Andrew phones Tim
02:45 retrolog: db12 ganglia graphs go flat. Apache CPU goes down to near zero. CPU saturates on some of the text squids: sq16, sq17, sq18, sq21, sq22, sq23, sq25, sq26. Frontend squids serve 503s.
02:00 retrolog: Network spike on db12

January 29

23:21 domas: restarted sq37 for being in bad shape.
22:25 tomaszf: starting xml snapshots back up on snapshots3
21:31 atglenn: cleared out cruft from /tmp on srv158 (how about a cron job? :-P )
18:39 logmsgbot: catrope synchronized php-1.5/maintenance/importImages.php 'Deploy r59908'
16:20 Rob: db28 mainboard replaced, updated LOM to work, updated dhcp with new mac address info
02:50 apergos: started copy of media data from ms7 to ms8 (nc running in screen as root on both hosts)
01:51 logmsgbot: andrew synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for new wikitech entry'

January 28

20:15 Rob: racked sq51-sq68, all drac accessible, wired, ready for install
19:05 Rob: updated dns for management ranges on new squid servers
13:56 mark: Added DNS entries for 36 new Squid servers
13:28 mark: Storage1 went down due to I/O failure
13:00 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove aliases made obsolete by r61632'
12:55 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesMl.php 'Deploying r61632'
10:57 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'supressredirect for rollbackers on ruwikisource'
09:30 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Removed redundant huwiki settings'

January 27

18:04 mark: Fixed ExtensionDistributor, which broke due to the image server migration
18:02 logmsgbot: mark synchronized php-1.5/extensions/ExtensionDistributor/svn-invoker.php 's/upload5/upload6/'
16:10 mark: Added a timeout to puppet's apt-get update execution; hopefully this will prevent things from getting stuck for weeks
16:08 mark: Shutdown sq20 once again, please decommission this broken box
16:05 mark: Fixed puppet on db7
16:05 mark: Killed stuck apt-get update processes on all servers
15:58 mark: Made puppet manage NFS mounts /mnt/upload6, /mnt/thumbs and the absence of /mnt/upload5 on all application servers and image scalers
13:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix Category talk alias on mlwiki'
11:57 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Also add a Unicode 5.1 alias for Category talk on mlwiki'
11:52 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22286: Unicode 5.1 namespace alias for Category on mlwiki'
11:46 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryBacklinks.php 'Deploying r61571'
08:12 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js
08:11 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js

January 26

23:44 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enable file uploads.'
23:34 Fred: restarted morebot
23:31 atglenn: turning off web server on ms1 (previously: moved uploads to ms1, mounted ms7 everywhere, updated squid conf)
21:10 atglenn: copying uploaded data since last replication, from ms1 to ms7
21:05 atglenn: turned off file uploads temporarily
21:04 logmsgbot: ariel synchronized wmf-deployment/wmf-config/CommonSettings.php
20:50 Rob: rebooted srv125 per dc task entry by fred
20:35 tomaszf: enabling global file uploads notice
20:29 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'Deploying r61534'
20:29 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'Deploying r61534'
18:38 Rob: shutting down pdf2 to relocate its rack position.
18:37 Rob: !log mobile2 port not moved yet, as its port assignment on wikitech is invalid
18:08 Rob: shutting down mobile2 to relocate its rack location
18:07 Rob: bayes back online
18:00 Rob: bayes management lom had ip issues, resolved, still working on host
17:52 Rob: bayes is not coming back online properly, checking.
17:47 Rob: bayes rack location moved and updated in racktables, powering back up
17:34 Rob: shutting down bayes to move its rack location from b2-pmtpa to a2-pmtpa
14:42 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22274: Set $wgAutoConfirmCount to 10 on itwiktionary'
00:56 Tim: on hume: increased RCT proc count, reduced max lag to 1s, increased lag check frequency

January 25

22:36 Tim: ms3 is reporting "table full" for rc1 enwiki.blobs. Running ALTER TABLE.
19:33 Fred: stopped apache on srv225
19:01 Fred: adjusted apache.conf on srv225 to make it work again...
16:07 Rob: powering srv224 & srv225 back up
16:01 Rob: shutting down srv225 & srv224 to swap their power cables from single to Y split.
15:45 Rob: srv196 memory replaced, back online and in service
15:45 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'pushing out change to two hosts that were down a few minutes ago'
15:40 Rob: shutting down db28 to swap memory around in diagnostics (bad memory in system)
15:25 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'took down srv196 for hardware replacement'
15:20 Rob: shutting down srv196 to replace faulty memory dimm
12:19 Tim: started recompressTracked.sh on hume

January 23

00:16 aZaFred_: implemented ganglia detailled mysql stats for db10.
00:16 aZaFred_: implemented DB Lag detection script for DB10

January 22

23:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.hooks.php 'Deploy r61401'
23:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/CollapsibleTabs/CollapsibleTabs.hooks.php 'Deploy r61403'
21:29 brion: freed some space on srv218 removing old upload temp files from /tmp but / is still pretty tight (445m free)
20:19 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'Deploying r61387'
19:05 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22201: Add WZ: alias on itwiktionary'
15:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable $wgEnotifWatchlist on usabilitywiki'
11:38 mark: Deployed new gmond 3.0.3-2 package that conflicts with the incompatible ganglia 3.1 packages in Ubuntu, to avoid problems in our upcoming upgrade

January 21

14:13 mark: Added srv210 to LVS
14:11 mark: Reenabled srv245 in LVS
13:58 mark: Restarted Apache on srv207, srv210, srv233, srv257, srv245
09:30 mark: Replaced srv121 (decommissioned) by search13 in smokeping
07:59 AaronSchulz: Removed empty ct_tag rows from code_tags
07:47 Tim: cleaned up /tmp on srv180 (was out of disk space)

January 20

20:36 logmsgbot: root synchronized php-1.5/wmf-config/mc.php
16:38 Fred: killed long running queries created by civicrm on db9.
13:56 domas: did lots of changes on wikitech wiki. :)
02:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging on enwiki'
01:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling api logging for enwiki'
01:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling API request logging on dewiki'
00:03 RoanKattouw: Strike my last, was actually *enabling* it on dewiki
00:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on dewiki'
00:02 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on frwiki'

January 19

21:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling API request logging on frwiki'
21:36 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling API request logging on frwiki'
20:34 Fred: bugzilla upgrade completed. It might be up to an hour before attachments can be added due to DNS propagation.
19:41 Fred: Bugzilla as you know it is going down. Be back shortly.
18:21 Rob: updated blog.wikimedia.org wp-stats plugin to 1.6.1
16:03 Rob: ssds installed in db28, during boot it displays a memory error, investigating.
15:20 Rob: db28 offline while disks are swapped to solid state for testing.

January 18

05:02 Tim: removed some old xff logs to free up space for logrotate which I ran with the -f flag
05:00 Tim: fixed logrotate on nfs1, broken due to duplicated entries between /etc/logrotate.d/rsyslog and /etc/logrotate.d/syslog-ng. Left the syslog-ng ones in for now.
04:02 RoanKattouw: NFS /home is full

January 16

14:55 mark: Started backend squid on sq50
14:26 mark: sq19 has bad drive /dev/sdc
14:22 mark: dist-upgrade & reboot on sq19
14:14 mark: Shutdown sq20, bad disk /dev/sda
14:13 mark: Reenabled sq24 frontend in PyBal
14:03 Tim-away: disk space critical on srv167, cleaned up /tmp
06:07 apergos: "recovery", my *ss... cleaned up /tmp on srv181 to get some space back

January 15

23:53 tomaszf: starting test dewiki snapshot on snapshot2
17:17 Fred: modified gmetad config on zwinger and spence to reflect new apache 4cpu aggregator
17:16 Fred: added srv149 as a gmond aggregator in puppet.

January 14

17:28 domas: load-tested and fixed db19 to handle full bits workload (~22k/s), now again serving just tampa part.
01:32 apergos: moved bits to .2 again, all nameservers seem to be up and reflect the change (bits unresponsive again)

January 13

18:30 Andrew: [andrew@zwinger ~]$ sync-file wmf-config/CommonSettings.php 'Disable GIF scaling again, due to issues reported on village pump, bug 22041'
17:05 Rob: updated InitialiseSettings for bug 21174 and 21077
16:24 Rob: updated InitialiseSettings.php Bug 20508 Please enable Extention:NewUserMessage on en.Wikinews
11:50 domas: started overflow-watchdog ( http://p.defau.lt/?4_Y_Lrl9tVKS6fEYwkswKA ) on db19, sent bits load at it again
01:23 atglenn: using .2 for bits.pmtpa, did authdns-update, let's see what happens (bits was failing to respond)

January 12

19:52 rainman-sr: could someone please put search3,9 into search_pool 1 (with search1,4) on lvs3
19:37 Fred: Puppet: set spence as a ganglia aggregator for Misc tree.
19:23 Rob: usability/prototype linode was crashed, had to reboot
17:39 Fred: set wgDefaultSkin back to monobook on wikitech since vector is not operational.
16:08 mark: Arcor clients appear to have problems reaching our sites, traffic to Arcor over AMS-IX has been low since midnight UTC
15:52 Rob: srv222 & srv223 power reestablished, booting.
15:47 Rob: shutting down srv223 & srv222 to change out power cords.
15:46 Rob: db10 moved and back online.
15:36 Rob: db10 moved to sdtpa a2, powering up.
14:50 Rob: taking down db10 to relocate from pmtpa-b1 to sdtpa-a2
14:50 Rob: fixed issues with transcode2 and transcode3, completing base installation.
05:48 logmsgbot: tstarling synchronized php-1.5/includes/HTMLCacheUpdate.php
05:48 logmsgbot: tstarling synchronized php-1.5/includes/BacklinkCache.php
05:47 Tim: deploying r60962
01:17 Tim: on streber: removed a corrupt torrus DB file so it could be rebuilt, torrus should be working now
00:57 Tim: killed frozen torrus cron jobs and ran "torrus compile --tree=Network --force"
00:51 Tim: maybe torrus collector is still broken, trying /etc/init.d/torrus-common force-reload
00:46 Tim: with mpm-prefork managed to debug it fairly easily. Moved away permanently locked DB file render_cache.db, torrus.wikimedia.org is now fixed
00:39 Fred: restarting pdns on ns1
00:38 Tim: switching streber to apache2-mpm-prefork, can't work out why it's not working
00:22 Tim: trying "apache2 -X" on streber
00:00 Tim: restarting apache on streber

January 11

23:38 domas: logging the fact that we had cache layer meltdown at some point in time during the day
22:30 domas: leaving bits.pmtpa on db19's varnish, in case of troubles - uncomment bits.pmtpa .2 record in /etc/powerdns/templates/wikimedia.org and run authdns-update
19:43 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapped memcached from srv125 to srv232'
19:06 Rob: new apaches srv255, srv257 deployed. Updated node groups and synced nagios
19:03 Rob: new apache server srv254 deployed
18:24 atglenn: copy backlog of image data from ms1 to ms7 (running in screen as root on both boxes)
14:43 mark: Rebooting fuchsia, locked up again
14:24 mark: Increased load on knsq16-22 by upping lvs weight from 10 to 15

January 10

23:02 logmsgbot: midom synchronized php-1.5/wmf-config/lucene.php 'rainman asked, rainman guilty, hehehe'
23:01 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php
17:36 rainman-sr: search limit raised to 500 again, interwiki search re-enabled for "other" wikis
16:07 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'take ixia back out'
16:06 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put ixia back'
16:06 rainman-sr: restarting search cluster to deploy search13-19
15:58 domas: all bits serving switched back to text cluster, we have problems with all threads blocking on write(): http://p.defau.lt/?dOBxveiHj_ukjzupEBX3rA
15:19 rainman-sr: configuring search13-19, will leave search20 as spare
14:22 domas: apparently varnish worker threads are blocking on network output, ... :)
12:40 domas: full bits pmtpa load sent to sq1
12:04 domas: sending half of bits load to sq1
12:02 domas: set up separate geo balancing for bits via bits-geo.wikimedia.org
10:45 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'setting extension asset path to bits.wm'
10:42 logmsgbot: midom synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php
10:29 logmsgbot: midom synchronized php-1.5/includes/Setup.php
10:29 logmsgbot: midom synchronized php-1.5/includes/DefaultSettings.php
09:36 domas: moving over static assets to 'bits.wikimedia.org'
09:11 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php
09:09 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php
08:26 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
08:25 river: taking ixia out of rotation to dump commons

January 9

17:31 mark: Upgraded pdns-recursor to 3.1.7.2 on dobson, mchenry, lily
15:32 mark: Temporarily filtering all prefixes from 1299 on br1-knams, due to some balanced link blackholing issue
13:10 logmsgbot: midom ran sync-common-all
12:45 domas: restarted, that is :)
12:44 domas: fixed ns1, was deadlocked

January 8

21:48 Rob: nagios is flapping errors for esams hosts, but they are still up and functional. Perhaps due to new transit setup earlier today.
20:10 Rob: pushing dns update for flaggedrevssandbox project
16:09 Rob: finished mobile2 initial setup, gave hcatlin sudo rights to server
15:46 mark: Brought up transit session to 1257 on br1-knams
14:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:42 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

January 7

20:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22038: Enable Collection extension on skwiki'
20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Fix $wgUploadNavigationUrl on bnwiki'
20:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Enable patrolling on frwikiversity'
19:54 Rob: base install done on mobile2, pdf2, transcode1, transcode2
16:55 Rob: srv254-srv257 racked, drac online, network attached. needs installs
15:46 Rob: racked srv254-srv257. DRAC setup, cables run. Network not plugged in until mark provisions the ports.
02:51 Tim: restarted apache2 on wikitech to fix swapping
02:27 Fred: rebooted pascal as it was once again hung
01:57 mark: Prepended AS 43821 once on incoming prefixes from AS 1299

January 6

22:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable OptIn on trwikimedia'
17:17 Rob: updated dns for tr.wikimedia.org
14:47 Rob: updated techblog software to newest stable revision
14:01 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22027: Enable AbuseFilter and CAPTCHA on trwiki'
13:27 river: restarted slave on db26
12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Set $wgUploadNavigationUrl on bnwiki'
12:37 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'remove debugging code from yesterday'
12:34 RoanKattouw: Resuming GlobalUsage rebuild
08:48 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
08:47 river: removing db26 from s1 to dump

January 5

23:57 RoanKattouw: GlobalUsage refresh script properly handling replag now. Aborted it, will resume in the morning (CET)
23:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/refreshGlobalimagelinks.php 'remove debugging code'
23:54 RoanKattouw: Resuming GlobalUsage refresh script
23:53 logmsgbot: catrope synchronized php-1.5/includes/db/LoadBalancer.php 'Deploying r60705'
23:53 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploying r60705'
22:21 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Debugging'
22:11 RoanKattouw: Aborting GU refresh *again*, somehow it's not detecting slave lag right
22:07 RoanKattouw: Resuming GlobalUsage refresh script
22:04 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploy r60695 (wfWaitForSlaves fix)'
21:44 RoanKattouw: Running refreshGlobalimagelinks.php on all wikis from the smallest up
20:50 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'Deploy r60692 (GlobalUsage fixes)'
20:14 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'added outreach namespace to officewiki'
19:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikiversity'
19:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable GlobalUsage on all public wikis'
17:10 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapping srv110 for spare srv231.'
16:41 Fred: restarted memcached on srv110
16:10 mark: Installed search13-20 as new search servers
15:50 mark: Installed karmic on search19
15:45 Rob: updating dns for mobile2 and pdf2 servers
12:10 mark: dist-upgrade and reboot on sockpuppet
12:01 river: copied old wmvids to ~kate/wmvids
11:37 mark: Installed OS on search18
07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php
07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/api/ApiCodeDiff.php
07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.i18n.php
07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.php
06:48 Tim: restarted apache on srv199, APC cache corruption
02:33 tomaszf: compressing support-requests on locke to free up space
02:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabled $wgCopyUpload and $wgHTTPProxy on commons and test, breaks Lucene Search'

January 4

22:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22002: Add Opinions namespace on bgwikinews'
21:51 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to shut up exceptions thrown by srv218
21:29 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 20312: Re-enable GIF scaling on all wikis'
21:09 RoanKattouw: srv205 is throwing fatal errors because the geoip PHP module is not installed
20:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix typo'
20:45 RoanKattouw: srv218 has a full disk
20:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'plus proxy setting'
20:44 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20512: Enable $wgCopyUploads on test and commons'
20:44 mark: OS-installed Karmic on search13, search14, search16, search17, search20
20:40 RoanKattouw: Restarting refreshGlobalimageusage.php on Commons after it mysteriously disappeared
20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GIF scaling on testwiki'
19:39 RoanKattouw: Running refreshGlobalimagelinks.php on Commons again
19:38 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GlobalUsage on commons again'
19:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Turning on GlobalUsage on testwiki'
19:33 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageQuery.php 'Seems I forgot this file'
19:20 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to kill "MagicWordArray::parseMatch: parameter not found" error
19:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable GlobalUsage again, blank pages on commons'
19:06 RoanKattouw: Running rebuildGlobalimagelinks.php on Commons
19:03 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Re-enabling GlobalUsage'
18:57 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.i18n.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage_body.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/ApiQueryGlobalUsage.php 'r60611'
18:55 RoanKattouw: Deploying r60611 (GlobalUsage update) with individual sync-file's
18:49 RoanKattouw: Truncating globalimagelinks table on commonswiki
18:48 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Temp disable GlobalUsage'
18:46 RoanKattouw: Disabling GlobalUsage temporarily for software update
18:41 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Enabling WikimediaLicenseTexts on Commons'
18:40 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on Commons'
18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaGrammarForms.php 'r60608'
18:23 RoanKattouw: Deploying r60608 (WikimediaMessages update) using sync-file
17:47 Rob: search13-search20 racked and ready for install
17:17 Rob: updating dns for new search servers
14:57 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached box srv84'
14:45 Rob: moved power for asw-b3-pmtpa, srv86 and srv87 may flap.
14:43 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached boxes srv120, srv121'
14:31 Rob: decommissioned srv120, srv121 to make room for new search servers.
14:29 Rob: pulling srv34 from rack to decommission, need the space for new search servers.
14:09 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace memcached on just decommissioned host srv82'
13:57 Rob: srv82, srv83, srv84 decommissioned to make room for new search servers in rack
12:56 Tim: started squid-frontend on knsq8, knsq10, knsq11, knsq13, knsq14, knsq15, all crashed at roughly the same time as knsq9
12:52 Tim: started squid-frontend on knsq9, died at ~17:30 on the 3rd. Syslog shows many crashes, followed by "out of socket memory" a couple of hundred times, then silence

January 3

20:17 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60588, r60590'
17:29 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'added die(); somewher in there'
14:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'ixia is being put on a bench'

January 2

19:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Enable import from Commons on brwikimedia'

December 31

19:45 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Moving tracking back to db9 as it requires insert'
18:38 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Switching to query the slave db10 instead of the master db9'

December 30

23:37 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php
23:35 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
12:51 mark: Set /proc/sys/vm/min_free_kbytes to 65535 on brewster and streber, to see if it helps with the swapper page allocation failure bug
07:57 Tim: fixed private+secure file access, URLs were broken for months at least
07:50 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php
07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'possible resolution for boardwiki problem'
07:45 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'debugging hack'
07:34 Tim: created an account for myself on boardwiki to debug file upload issue

December 29

23:42 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'added tracking for 2009_notice51.'
18:27 tomaszf: dropping retention on storage2 to 4 xml snapshots
13:04 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60468'
12:50 RoanKattouw: Started logmsgbot (running as catrope instead of nobody)
12:32 RoanKattouw: logmsgbot down since fenari reboot, needs to be restarted by a root

December 28

21:02 RoanKattouw: Restarting refreshLinks for enwikibooks on hume
20:49 mark: Created 10G of swap space on fenari
20:07 domas: powercycled fenari, console not responsive
15:00 mark: Installed Karmic on LVS1, setting up a test network for LVS performance testing
14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 17338: Disable $wgRestrictDisplayTitle and enable subpages in the main namespace on rmwiki'
13:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21566: Enable Collection on viwiki'
13:42 RoanKattouw: Running refreshLinks on enwikinews per bug 19404
13:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19404: Set $wgCategoryPrefixedDefaultSortkey=false on enwikinews'
13:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21958: Create Book namespace on enwiki'

December 27

19:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Import sources, subpages for brwikimedia'
11:26 domas: db20 offlined disk array, after reboot booted into netinstall (but saw the array), after reset /SYS, and some operator's no-operation in BIOS and RAID controller setup screens, it booted up properly

December 26

19:59 DaBPunkt: (non-dev-entry) Many apaches in the US died, some db-server reported overusing. Problem fixed itself after some minutes.

December 25

20:24 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php
20:09 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixed frwiktionary config (sysops should be able to remove patroller)'

December 24

22:52 RoanKattouw: Running updateArticleCount.php on arwiki and arwiktionary
20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var AGAIN'
20:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var'
20:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21859: Enable Doublewiki on frwiktionary'
19:03 logmsgbot: catrope ran sync-common-all
18:56 RoanKattouw: Running sync-docroot, sync-apache, sync-common-all, apache-graceful-all
18:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Explicitly add langcodes for recently created wikis'
18:27 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19534: Add Page and Index namespaces on sourceswiki'
17:02 logmsgbot: catrope ran sync-common-all
16:57 RoanKattouw: Syncing updates for wiki creations
16:57 RoanKattouw: Not creating fiwikimedia, already created
16:54 RoanKattouw: Adding fiwikimedia (sans DNS entry) per bug 20502
16:41 RoanKattouw: Adding arbcom_fiwiki (sans DNS entry) per bug 21375
16:32 RoanKattouw: Adding dkwikimedia (sans DNS entry) per bug 21009
16:23 RoanKattouw: Adding brwikimedia (sans DNS entry) per bug 21149
16:14 RoanKattouw: Adding pcdwiki (sans DNS entry) per bug 21634
16:06 RoanKattouw: Made raw, non-highlighted versions of .php files visible through noc.wikimedia.org/conf
15:55 RoanKattouw: Made abusefilter.php visible through noc.wikimedia.org/conf
15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
15:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix name of oldwikisource -> sourceswiki'
15:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable ProofreadPage for oldwikisource, which strangely is not part of the wikisource group'
15:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20402: Enable Collection extension on mswiki'
15:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21389: Set $wgDisabledVariants on Chinese wikis'
14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21527: Enable Special:Cite on itwiktionary'
14:28 RoanKattouw: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21100: Add Tac_gia namespace to $wgNamespacesToBeSearchedDefault on viwikisource'
14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21011: Restrict uploads, change $wgUploadNavigationUrl on enwikibooks'
14:10 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21236: Set $wgEnableNewpagesUserFilter=true; on plwiki'
14:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21370: Create custom namespaces on etwikisource, fix namespace alias on arwikisource'
13:51 RoanKattouw: Running updateArticleCount.php on zhwikisource for bug 20998
13:48 domas: s1 and s2 had rolling restarts to mysql#3193 build
13:48 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Rename autopatroller group to autoreviewer on zhwiki'
13:44 domas: s2 master switched to db15-bin.000001:106
13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
13:32 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'misspelled right name'
13:29 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20721: Add abusefilter-viewprivate right for sysops on enwiki'
13:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20561: Change sitename on tkwiki'
12:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 15237: Localize sitename and Wikipedia namespace on nahwiki'

December 23

16:02 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:35 Rob: db27 back online with new fan controller board.
13:53 Rob: shutting down mysql on db27 for hardware replacement.
12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Actually enable patrolling on frwikibooks and frwiktionary'
08:22 Tim: on isidore, also disabled the CentralNotice job that was running every 20 minutes from /etc/crontab
05:16 Tim: disabled CentralNotice rebuilds for donate.dev.wikimedia.org, was overloading isidore (which is only a single-core pentium)

December 22

23:45 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Added 2009_Notice49 to tracking.'

December 21

19:25 Rob: ms8 still borked.
19:24 Rob: restarted mysql on db27
19:12 Rob: removed the fan boards in db27 and cleaned connections (were filthy with toner or dust or something) and replaced them. system booting back online. (Will watch it for errors for the next week.)
18:36 Rob: shutting down db27 mysql manually for troubleshooting hardware on the system
18:04 Rob: db28 mainboard and fans replaced, booting back online.
17:35 Rob: restarted mysql on srv185, since it appears to be a ext. storage slave.
17:32 Rob: srv185 memory replaced, back online.
17:20 Rob: shutting down srv185 to swap bad dimm1
16:52 Rob: swapped out bad disk in db30, all leds are green now.
16:22 Rob: swapped out bad memory in sq32, booting it back up.
16:13 Rob: shutting down sq32 to swap out bad memory
07:29 logmsgbot: tstarling synchronized php-1.5/extensions/intersection/DynamicPageList.php 'deploying r60254'

December 20

21:29 RoanKattouw: Clean up entries with rev_user_text='conversion script' (capitalize C) on dewiki for bug 21910
21:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'forgot $wgRemoveGroups'
21:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikibooks and frwiktionary'
21:03 RoanKattouw: Running namespaceDupes on simplewiktionary for bug 21906
21:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21906: Add WT: -> Wiktionary: alias on simplewiktionary'
10:25 domas: um, no more stalls?

December 19

20:53 RoanKattouw: Running namespaceDupes on zhwiki
00:41 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding JimmyAppeal[8-9] to tracking'

December 18

22:25 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'revert my last: logo too large'
22:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21213: Logo for acewiki'
21:57 logmsgbot: catrope ran sync-common-all
21:56 RoanKattouw: Closing nlwikinews per bug 20325
21:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix mistake (autopatrol vs. autopatroller)'
20:24 mark: Updated firewall for allowing ssh access from fenari on loudon
13:21 mark: Stopped oprofile on db22
11:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Add autopatroller group, allow sysops to add/remove confirmed on zhwiki'
11:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20910: Set $wgUploadNavigationUrl on nlwiki'
11:18 RoanKattouw: srv205 has full disk
11:09 domas: s1 switched master to db16-bin.000001
11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
00:14 aZa_: restarted srv141. Puppet went crazy cpu hungry.

December 17

23:54 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php
23:53 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
23:15 RoanKattouw: Strike that, frwikinews
23:07 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21436: Activate flood flag on frwikibooks'
20:32 Rob: sq20 reinstalled, squid package installed. Not back in service until I check with Mark about some puppet settings.
20:07 Rob: resynced nagios to remove sq27
20:03 Rob: sq20 disk died, replaced and reinstalling.
20:01 Rob: sq27 decomissioned (stole drive to fix sq20)
19:59 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Allow sysops to add/remove patroller and autopatroller on hewiki'
18:54 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php
17:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21158: Enable $wgBlockAllowsUTEdit on enwikiquote'
16:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21121: Enable flood flag on simplewikibooks'
16:22 RoanKattouw: Running namespaceDupes on iawiktionary
16:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiktionary'
16:17 Rob: pushing update to dns for search13 dns
16:09 RoanKattouw: Running namespaceDupes on iawiki for bug 21241
16:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiki'
16:04 RoanKattouw: Running namespaceDupes on brwiki for bug 21417
16:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21417: Change name of project namespace on brwiki, add old name as alias'
15:52 mark: Killed memcached on browne
15:51 mark: Started ircd on browne
15:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21041: Allow sysops to grant and bcrats to remove transwiki on nowiki'
15:46 Rob: browne back online after move.
15:46 Rob: nagios synced to node group files without will.
15:45 Rob: will decommissioned, pulled from rack
15:40 Rob: shutting down browne to move its rack location, will be back online shortly.
15:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Add patroller and autopatroller groups on hewiki'
15:26 mark: Moved udpmcast from browne to dobson
13:11 mark: Fixed Racktables rack thumb problem by installing php5-gd; it was just serving cached thumbs
11:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21306: Allow sysops to add/remove patroller and autopatrol on hrwiki'
11:00 RoanKattouw: Running namespaceDueps on mlwiki for bug 21277
10:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21277: Add Portal and Portal talk as aliases on mlwiki'
10:53 RoanKattouw: Running namespaceDupes.php on cawikibooks for bug 20980
10:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20980: Add Viquiprojecte namespace on cawikibooks'
10:47 RoanKattouw: Resyncing srv194 and rebuilding caches
10:46 domas: srv194 had 5G-sized oprofile error log, hehe hehehe
10:39 RoanKattouw: srv194 disk still full
10:39 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Allow sysops to add/remove rollbacker on gawiki'

December 16

20:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix mistake'
20:32 RoanKattouw: srv194 disk full
20:32 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Add rollbacker group on gawiki'
19:56 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2/jquery-ui-1.7.2.js 'Deploying r60130'
19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.min.js 'Deploying r60130'
19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.js 'Deploying r60130'
19:53 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed groupOvrrides for fiwiki for arbcom.'
17:44 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'FR labs config - remove excess level'
16:36 hcatlin: new and improved mobile with compressed memcached + new homepags + better utf-8 handling
16:24 hcatlin: taking down mobile1 for a large software update
15:59 mark: Disabled NIS on zwinger
15:54 mark: Disabled NIS on ms1
15:53 mark: Disabled NIS on ms4
15:47 mark: Installed puppet on ms1 and ms7
15:27 Andrew: srv129, srv123, srv120, srv95 seem to be in swapdeath
15:25 logmsgbot: andrew ran sync-common-all
15:24 Andrew: sync-common-all caused memory spike on apaches again, site seems to still be up though
15:17 Andrew: Updating LiquidThreads to trunk state, using sync-common-all
12:58 RoanKattouw: Tim ran rebuildTemplates.php on hume for all languages
12:52 Andrew: Fixed morebots, was choking because the Server Admin Log was archived.
12:50 Andrew: Moved Morebots init script into a line in rc.local. Morebots seems to have been down due to Freenode's DDoS problem, maybe it isn't exiting properly when disconnected from the server
12:46 Tim: ran rebuildTemplates.php on hume for all languages