Server admin log/Archive 20

February 19

18:48 Fred: removed memcached node srv193 and replaced with srv233
16:52 Rob: updated InitiliseSettings with bug 22585
01:31 Fred: puppetized gmond.conf generation using templates

February 18

23:51 RoanKattouw: Fixed search script on yongle, removed UA check exemption for yongle
23:51 RoanKattouw: logmsgbot is broken again
04:07 Tim: re-running storageTypeStats.php on enwiki to identify rows with old_flags='object,utf-8', which need to be handled properly in fixBug20757.php

February 17

17:25 Rob: pushed updates to wordpress installations
17:04 RoanKattouw: Synced wmf-config/checkers.php 'Allow missing UA from yongle'
17:04 RoanKattouw: Restarted logmsgbot
16:47 Fred: test
07:24 Fred: there is a problem with ganglia. It is being worked on.
00:15 RoanKattouw: Resynced srv151, was returning empty responses

February 16

23:31 Fred: upgrading gmond to 3.1.2 everywhere. However, due to the newish module structure, there is a potential that ganglia will hickup while puppet does its job...
22:03 logmsgbot_: mark synchronized php-1.5/wmf-config/checkers.php 'Exceptions'
20:30 Rob: srv127 is online, but not in LVS. Its cert was accepted on sockpuppet, but puppetd --test results in a cert failure on retrieval from sockpuppet.
20:17 Rob: rebooting srv127
20:15 mark: Increased membufs to 40 per COSS dir on the pmtpa upload backend squids
19:51 mark: Increased membufs per COSS dir from 10 to 20 on the new pmtpa squids
18:10 apergos: but documentation can save people precious time when things are on fire
18:02 mark: Documentation is not a substitution for thinking
16:44 mark: Fixed puppet on most servers
16:23 domas: anyone knows why mysqldump on snapshot3 is locking tables? maybe --single-transaction could work better?!!!?
16:22 RoanKattouw: Strike my last, I hear it'll fix itself in a day
16:21 RoanKattouw: Oops, meant to type srv187-189
16:21 RoanKattouw: Ganglia not picking up data for srv1987-189, 193, 214-218, 250-253, 257 even though the boxes are up and gmond is running; been like this for 3 days
15:21 mark: Removing all puppet certs/private keys on all machines
14:55 mark: Puppetmaster is screwed up by a wrong command that deleted all files under /var/lib/puppet
14:25 logmsgbot_: midom synchronized php-1.5/includes/diff/DifferenceInterface.php 'instrumenting costs by revision pairs'
13:46 logmsgbot_: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22444 and 22330 enabling collection on dawiki and ruwiki'
13:22 domas: oh wait, it does work, something else....
13:16 domas: apparently my squid rule for Apple browsers didn't really work, hence the continuing spikes
03:16 domas: blocked MacOSX Atom syndication on the edge, this will save terabytes of diskspace on unsuspecting Safari user computers :-)
02:13 logmsgbot_: tstarling synchronized php-1.5/wmf-config/checkers.php

February 15

23:42 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'uninstrumenting'
23:09 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php
23:09 domas: banned all UA-less requests
22:53 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 'un-disabling UA check on API'
22:14 RoanKattouw: Resynced srv114, was returning empty responses
22:10 logmsgbot_: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)'
22:10 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)'
22:02 logmsgbot_: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:58 logmsgbot_: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r62553'
21:39 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiMain.php 'r62550'
21:30 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'raised wgSearchSuggestCacheExpiry from 1200 to 86400'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62548'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiMain.php 'r62548'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/DefaultSettings.php 'r62548'
20:54 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiBase.php 'r62546'
20:54 logmsgbot_: catrope synchronized php-1.5/api.php 'r62546'
20:53 domas: Roan and me are doing some perf engineering on opensearch & related stuff. \o/
20:40 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'readding stub trace'
20:34 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 'perf comment action'
20:30 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'removing trace hooks'
20:28 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php
20:24 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62543'
19:32 mark: Allowing API requests to be sent to the regular app servers temporarily
19:27 mark: Restarted sq31 and sq31 API backend squids with slightly reduced memory settings
19:18 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php
17:51 Rob: set srv127 to false on lvs3, will re-enabled once in DC to reboot it manually
17:49 logmsgbot_: root synchronized php-1.5/wmf-config/mc.php 'srv127 down taking out of mc rotation'
16:02 RoanKattouw: Resynced srv134 to fix HTTP 500
15:13 RoanKattouw: Resynced srv232, was serving HTTP 500s
02:02 mark: Fixed stale PHP file removal apache cron job in Puppet

February 14

19:25 mark: srv206 spontaneously rebooted
17:12 mark: Added sq32 to the API backend squid group (with an empty cache)
16:38 mark: Confined all API requests to sq31 backend Squid
12:54 RoanKattouw: logmsgbot is broken

February 13

17:44 mark: Made 64 bit java exception for search13 in puppet
17:37 mark: Converted srv214-218 and srv86-87 into API servers
17:02 mark: Throwing all API requests at the new API app server cluster
16:49 mark: Converted srv187-srv189 and srv250-srv253 into API app servers
15:46 mark: Created new Ganglia group "API application servers", deploying with Puppet
15:31 mark: Created API app servers group in Puppet, deploying on srv254-257 to start with
15:24 mark: Created API LVS service on lvs3 and restarted PyBal
15:10 mark: Added DNS entry for api.svc.pmpta.wmnet (LVS ip)
14:18 mark: Enabled more verbose logging on all squids: debug message level 1 for all message categories
14:18 mark: Disabled cache digests on all squids
13:58 mark: Removing sq30 and lower from the Text CARP pool
13:51 mark: Reducing persistent_connection_timeout and request_timeout from 1 min to 20s on all frontend squids, to reduce FD usage
12:50 mark: Restarting knsq* frontend squids with FD limit raised from 32k to 64k
12:23 mark: Grown filesystem /dev/data/ES (/a) on ms3 by 90G, leaving another 98G free
00:17 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Sync CSS style versions'
00:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
00:10 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62403'
00:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62400'
00:01 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62400'

February 12

23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Try deploying r62397 again'
23:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r62397'
23:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62397'
23:22 mark: Deployed sq86
22:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62388'
21:12 mark: Implemented new routing policy for AS43821
21:01 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:59 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62382'
19:15 Rob: sq67 has both disks installed now (had to get one replaced) and has no OS installation yet. holding off on install, not sure if mark wants this up as a squid or for varnish development
18:22 atglenn: resstarting transfer of data (one chunk) from ms8->ms5->ms6, running in screen as root on all three hosts using nc
16:18 Rob: setup project1 and project2 boxes for flaggedrevsdevelopment and project mangement suite testing
15:12 mark: Upgraded our AMS-IX connection from 2x 1G to 10G along the way
15:12 mark: Reloaded br1-knams twice to fix a CAM partitioning problem; not sufficient IP next hops for 8-path routes
11:34 mark: Filtering all prefixes to/from AMS-IX peers
02:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 're-enabling WikiEditor'
02:04 Tim: installed openssh-server on bayes, Erik Z apparently uninstalled it with aptitude
01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'disabled WikiEditor due to complaints about bug 22428'
01:06 atglenn: shoveling over some data from ms8 to ms6 via ms5 :-/
00:48 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Disable live preview entirely, broken by CentralNotice and DismissableSiteNotice'
00:37 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable DismissableSiteNotice for lqt.labs, breaks LiquidThreads live preview'
00:17 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
00:07 Andrew: Running scap
00:02 Andrew: Updating LiquidThreads alpha to trunk state, deploying with scap in a few minutes

February 11

22:06 Fred: implemented disk space check for tridge
21:10 mark: Pooled sq71-85 with full CARP weight (30)
21:06 mark: Pooled sq79-85 frontends in LVS with full load (30)
20:59 mark: Pooled sq79-85 frontends in LVS with load 1
20:51 mark: Pooling sq79-85 in CARP on all frontends, with low CARP weight 10 instead of 30
20:30 mark: Pooling sq79-85 in CARP on frontend sq51 to seed the cache
19:39 atglenn: replication every 15 minutes enabled from ms7 to ms8
19:31 mark: Removed sq16-30 from the text frontend squid pool in LVS - these servers will soon be decommissioned
19:28 mark: Pooled sq71-78 frontend squids in LVS with full load (30)
19:21 mark: Pooled sq71-78 frontend squids in LVS with low load (1) to seed caches
18:30 mark: Pooling sq71-78 backend squids on all frontends, with lower CARP weight (10 instead of 30) to seed the caches
18:04 mark: Deployed new squid config to sq58-66 frontend squid, to seed the caches of backend squids sq71-78
17:27 mark: Deployed new squid config to sq66 frontend squid, to seed the caches of backend squids sq71-78
17:24 mark: Deployed Squid with correct configuration on sq71-78
11:08 RoanKattouw: Load spike on 4CPU Apaches ended, rr.knams back up, downtime seems to be over
11:05 RoanKattouw: Another load spike on the 8 CPU Apaches, approx 11:02-11:-05 UTC
11:00 RoanKattouw: Apache load spike started around 10:42 UTC and coincides with a load dip followed by a monitoring blackout on the Kennisnet Squids and a load spike on the Tampa Squids
10:58 RoanKattouw: 8CPU Apaches stopped going bonkers all of a sudden according to Ganglia, 4CPU ones still have high CPU usage
10:56 RoanKattouw: Site went down, all Apaches have high CPU usage

February 10

22:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix to deploy usability changes sitewide'
22:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62275 to test'
21:28 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264 for real. Running svn up first helps'
21:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264'
21:03 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
21:02 Andrew: Switching strategywiki to opt-out LiquidThreads, rather than opt-in
20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'Deploy r62262'
20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62262'
19:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:21 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62257'
19:05 Rob: storage1 disks replaced, except for the newly dead one, rma placed.
18:39 Rob: shutting down storage1 to replace its bad OS mirror disk
16:52 mark: Fixed MySQL permissions on srv186 as well
16:15 mark: Rigged puppet to deploy a PHP mail.ini file to the apaches, which sets it to call sendmail with -f <> (sending mail with empty envelope sender, like bounces)

February 9

22:06 Andrew: LiquidThreads production software updates completed successfully.
21:58 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Dispatch.php
21:57 mark: Firewalled one ip on singer which was overloading Apache, looked like a Wordpress attack
21:50 mark: Lowered MaxClients setting to 250 on singer
21:47 Andrew: Running scap to deploy LiquidThreads updates
21:45 Andrew: Updating LiquidThreads production deployments to the alpha version in the next few minutes.
21:44 mark: Power cycled singer
21:19 mark: Fixed MySQL system user & permissions on srv151-srv185
21:17 mark: Fixed MySQL instance on srv183
20:47 mark: Fixed MySQL instance on srv156
19:39 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:29 logmsgbot: catrope synchronized php-1.5/skins/common/edit.js 'Deploy r62190. Only for test for now, thank God for style versions'
17:01 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
17:01 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62187'
16:52 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
16:51 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62184'
14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
14:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62181'
02:14 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merge r62158'

February 8

23:25 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for r62041. Varnish needs this or Tampa will serve stale JS off bits'
23:06 mark: Deploying Exim on all application servers as well
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'Deploy r62141'
21:51 mark: Added search3 and search9 to search_pool_1 on lvs3
21:48 mark: Deploying Exim as a simple mail sending-only MTA (with queuing enabled) on squids, image scalers, search servers and core databases
21:38 Andrew: LiquidThreads wikinews rollout complete, though there was some buggage related to replication that had to be resolved.
21:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:27 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:21 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:18 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
21:18 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php
21:13 Andrew: Starting LiquidThreads deployment on enwikinews.
20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Track NTOC and dialogs as well in PrefStats'
13:04 Tim: running fixBug20757.php on enwiki, in a screen on hume
12:51 Tim: locke went into swap half an hour ago and is now dead. Trying reboot.
07:38 Tim: stopped recompressTracked.php for now due to bug 20757
07:04 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'r62120'
06:18 Tim: cleared out db9 root partition (which had 370MB free) by moving 4GB of SQL files from /root and /home/tfinc to /a/backup/junk
06:01 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 're-added ms2 as an ES slave for rc1,cluster22 and the ex-fedora clusters, was depooled for ~6 months'
05:55 Tim: added grant for root@fenari to ms3/2
05:31 Tim: started apache on srv213, srv161 and srv240, was stopped for no apparent reason
04:52 logmsgbot: andrew synchronized php-1.5/includes/ChangesList.php 'Merge r62117'
04:37 Andrew: Deploying LiquidThreads alpha updates with scap
04:23 Andrew: Planning to deploy LiquidThreads alpha (liquidthreads.labs and test) in the next few minutes
01:15 Tim: added mysql and gmetric users on srv155 temporarily, to get those services back up
00:42 Tim: ran apt-get upgrade on srv155. Doing reboot test for new kernel.

February 7

23:36 Andrew: Clarification: Somebody should fix it, not I think it's fixed now.
23:35 Andrew: The issue of centralnotice pulling notices from the wrong place should be fixed now, though
23:34 Andrew: Moved /mnt/upload5/centralnotice/centralnotice.js and /mnt/upload5/centralnotice/wikipedia/centralnotice.js to centralnotice-old.js, should stop fundraising banners from appearing.
23:33 Andrew: Issue with fundraising banners appearing on random pages caused by clients with the URL http://upload.wikimedia.org/centralnotice/wikipedia//centralnotice.js?257z6
22:11 Andrew: Reports that fundraising banners are appearing on random pages for some users.
21:32 domas: we're running on single ES copy.
21:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'disabling broken storage1'
20:05 domas: removing all fixes I've been doing to recover mail. WMF staff will handle this tomorrow, or the problems will be gone by then.
19:00 domas: blackholed few thousand IPs on mchenry via OUTPUT chain (using -j REJECT action)

February 6

19:55 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php
01:09 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/CentralAuth.php 'Merge r61717, because Casey Brown asked me nicely.'
01:08 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/ApiQueryGlobalUserInfo.php 'Merge r61717, because Casey Brown asked me nicely.'
00:58 Andrew: Scapping to deploy r62055 'For usability initiative: merge r62041, r62043'

February 5

06:00 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 'Re-activate LiquidThreads email notifications, DoS has passed'
05:22 Tim: freed up a little space on ms3 by reducing expire_logs_days from 14 to 7, and running flush logs
02:45 apergos: cleared out /tmp on srv169...
00:23 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for jQuery UI stylesheet'
00:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for combined.min.css'
00:03 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'Resync this, doesn't seem to have been picked up right'

February 4

23:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62002'
23:50 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
23:37 RoanKattouw: Running scap to deploy new UsabilityInitiative code
22:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
22:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js 'r61995'
22:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable UsabilityInitiative_alpha on usabilitywiki'
22:20 logmsgbot: tstarling synchronized php-1.5/includes/api/ApiUpload.php 'disabled chunk upload'
22:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
22:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
22:05 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/images/wikiEditor/dialogs/insert-link-external-rtl.png
22:04 RoanKattouw: Deploying r61993 with individual sync-files
21:15 atglenn: turned off hourly snaps on ms7, we'll have daily plus the standard replication
18:49 atglenn: cleared /tmp on srv175 blah blah
18:36 atglenn: cleared out php* files from /tmp on srv163 (need cron job)
10:35 mark: Removed sq11-15 from the upload LVS pool
10:30 mark: Removed sq11-15 from the upload CARP pool
03:28 apergos: added /usr/local/apache/common/php to include path of php on fenari

February 3

23:56 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css
23:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
23:54 RoanKattouw: Deploying r61959 using individual sync-files
22:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Explicitly enable EditWarning by default here. Previously done in extension file'
17:32 mark: Set root password on all servers using Puppet
17:26 Rob: loaded sq68 full with 6 ssd, had to 'borrow' drives from the unracked squids in pmtpa awaiting the rail issue resolution.
17:26 Rob: connected sq67 and sq68 to secondary network ports as requested
16:43 Rob: had to update gmetad and add sq14 and sq17 into puppet as ganglia aggregator hosts. Currently not showing in ganglia, but should when puppet updates their files shortly.
16:37 Rob: updated gmetad_pmtpa in NFS store, as well as on zwinger and spence.
16:20 Rob: !log cleaned sq1-sq10 certificates off sockpuppet
16:10 mark: Rebooting sockpuppet for security upgrades
16:06 Rob: updating DNS to remove lvs3.wikimedia.org, as its technically lvs3.pmtpa.wmnet
16:03 Rob: removed sq1-sq10 in dsh groups, server roles, pybal (already done, just doublechecked), and now starting wipe and removing their network connections.
15:37 Rob: updated video plugin on techblog, had to comment out sumotv support line due to errors.
13:30 mark: Removed sq1-10 from the upload backend Squid pool, preparing for decommissioning
12:27 mark: Removed eiximenis from the text backend pool
12:21 mark: Restarted sq51 with hyperthreading disabled
11:31 Tim-away: exim was flooding its mainlog with errors like "User 'exim' has exceeded the 'max_user_connections' resource (current value: 150)". OTRS is apparently broken as a result. Set the limit to infinity to fix it.
11:10 Tim-away: debugging mysql connection errors from mchenry to db9
11:02 Tim-away: restarting exim4 on mchenry
10:57 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php 'r61922'
10:47 RoanKattouw: Mail delivery reported broken again; last mail delivery to mediawiki-cvs was 3h22m ago
10:14 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version'
10:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html
10:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.combined.min.js
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
10:09 RoanKattouw: Deploying r61919 with individual sync-files
07:45 Tim: killed eximstats on mchenry, was sending the machine into swap
06:02 Andrew: Running scap to deploy LiquidThreads alpha updates
05:09 Andrew: Planning to update LiquidThreads alpha to trunk state in the next few minutes.
03:31 Tim: removing refreshLinks2 jobs from the enwiki job queue with namespace=0, 100k of them put there by a nasty biography template
00:34 domas: restarted varnish on db19

February 2

23:26 mark: PyBal got into a confused state due to a duplicate LVS realserver entry. Set up LVS manually/statically for 30 mins to debug the problem. PyBal is now active again.
23:04 Tim: pmtpa text squids down
22:33 mark: Pooled sq59-68 in LVS (Text)
22:27 apergos: restarted apache on srv170 (corrupted apc cache)
22:25 mark: Increased CARP weight of the new Upload squids sq51-58 from 10 to 30
22:17 mark: Added sq59-66 to the Text pool of Squids, with low CARP weight (10) to seed the caches
21:44 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php
21:42 Andrew: temporarily disabling $wgLqtEnotif to mitigate mail server DoS impact on LiquidThreads post/reply speed.
21:42 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/NewMessagesController.php 'Deploy r61879'
21:42 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'Deploy r61879'
21:04 mark: Removed sq2-10 from the LVS pool
20:46 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version addition for usability alpha'
20:45 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js 'Deploing r61700'
20:38 mark: Added sq52-58 to the backend Squid pool with low CARP weight, to seed the caches
20:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22351: Enable Vector and toolbar on srwikinews'
20:36 mark: Increased Exim load average threshold for delivery on mchenry
19:09 Fred: rebooted singer (went down for unknown- at this time- reasons).
18:25 Fred: OS-installed sq52-ssq58
18:14 mark: OS-installed sq59-sq66
17:23 mark: Thrown sq66 backend squid in the Squid pool, CARP weight 30
06:01 Tim: deploying r61846 via scap
04:25 Andrew: Schema change successfully applied.
04:24 Andrew: Applying schema change thread_signature.sql to mediawikiwiki, strategywiki, testwiki
04:22 Andrew: Applying schema change thread_signature.sql to liquidthreads_labswikimedia
01:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Fix for UsabilityInitiative_alpha config'
01:18 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js 'r61842'
01:13 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Append to $wgStyleVersion when running usability alpha'
01:10 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61841 for real this time'
01:09 logmsgbot: catrope ran sync-common-all
01:07 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61841'
01:05 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61839'
00:51 RoanKattouw: Running scap
00:48 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61837'
00:40 RoanKattouw: Synced extensions/UsabilityInitiative_alpha by hand
00:07 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js
00:07 RoanKattouw: Syncing some CSS and JS files for usability deployment so bits.wikimedia.org will pick them up
00:06 logmsgbot: catrope synchronized php-1.5/skins/common/wikibits.js 'Sync changes to wikibits.js in r61558'

February 1

23:30 RoanKattouw: Testing new usability code on test
23:27 mark: Pooled sq51 frontend squid in the upload LVS cluster, weight 30
23:03 mark: Thrown sq51 backend squid in the Squid pool, CARP weight 30
20:50 Rob: srv217 shutdown until checked
20:50 Rob: srv217 not responsive to network, but LOM works and system is jut not working online, possible bad cable, will check in DC tomorrow
20:43 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Added 36 new Squid servers, removed bart'
20:39 mark: Deployed new wikimedia-base package that no longer manages sysctl.conf post Hardy, letting puppet handle it
20:10 atglenn: restarted webserver on ms7 to read new conf (fix bug 22321)
20:09 mark: Granted cary access to the OTRS db from fenari
19:27 RoanKattouw: srv9 reported down, pinging yields Host Unreachable. srv9 is not in Nagios but seems to be in use
18:51 Fred: swapped memcache node from srv145 to srv196

January 31

21:18 domas: db28 has a failed dimm

January 30

22:29 RoanKattouw: Scheduling more batches for import overnight using at(1), see http://commons.wikimedia.org/wiki/Commons:Batch_uploading/Geograph#Progress
22:13 RoanKattouw: Running image imports overnight on hume, three parallel import scripts started each five hours, see http://commons.wikimedia.org/wiki/Commons:Batch_uploading/Geograph for details
22:12 apergos: restarted morebots, must have died after last night's freenode irc move
04:40 Tim: fixed broken /etc/rc.local on wikitech
04:18 Tim: restarting mysqld on db12
04:00 Tim: analysis: mysqld on db12 hit a bug at 02:45 and froze, most threads in futex. The mysql client failed to set a read timeout, leading to db12 sucking up all available apache threads. Several squids became overloaded, presumably due to the large size of our 503 error messages.
03:21 Tim: restarted all apaches, thereby killing the reads which started before 3:11, possibly as early as 02:45. Site service resumes immediately. ps -lL on db12: http://p.defau.lt/?Jw1EBE0fnV4Rpxe9OP8yAw
03:18 Tim: finally thought to answer the question "are the apaches waiting for something or idle?" Strace confirms that they are waiting for db12 in read().
03:11 Tim: Having trouble believing db12 could be responsible for the CPU spike on the squids. Depooled db12 just in case.
02:56 Andrew phones Tim
02:45 retrolog: db12 ganglia graphs go flat. Apache CPU goes down to near zero. CPU saturates on some of the text squids: sq16, sq17, sq18, sq21, sq22, sq23, sq25, sq26. Frontend squids serve 503s.
02:00 retrolog: Network spike on db12

January 29

23:21 domas: restarted sq37 for being in bad shape.
22:25 tomaszf: starting xml snapshots back up on snapshots3
21:31 atglenn: cleared out cruft from /tmp on srv158 (how about a cron job? :-P )
18:39 logmsgbot: catrope synchronized php-1.5/maintenance/importImages.php 'Deploy r59908'
16:20 Rob: db28 mainboard replaced, updated LOM to work, updated dhcp with new mac address info
02:50 apergos: started copy of media data from ms7 to ms8 (nc running in screen as root on both hosts)
01:51 logmsgbot: andrew synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for new wikitech entry'

January 28

20:15 Rob: racked sq51-sq68, all drac accessible, wired, ready for install
19:05 Rob: updated dns for management ranges on new squid servers
13:56 mark: Added DNS entries for 36 new Squid servers
13:28 mark: Storage1 went down due to I/O failure
13:00 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove aliases made obsolete by r61632'
12:55 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesMl.php 'Deploying r61632'
10:57 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'supressredirect for rollbackers on ruwikisource'
09:30 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Removed redundant huwiki settings'

January 27

18:04 mark: Fixed ExtensionDistributor, which broke due to the image server migration
18:02 logmsgbot: mark synchronized php-1.5/extensions/ExtensionDistributor/svn-invoker.php 's/upload5/upload6/'
16:10 mark: Added a timeout to puppet's apt-get update execution; hopefully this will prevent things from getting stuck for weeks
16:08 mark: Shutdown sq20 once again, please decommission this broken box
16:05 mark: Fixed puppet on db7
16:05 mark: Killed stuck apt-get update processes on all servers
15:58 mark: Made puppet manage NFS mounts /mnt/upload6, /mnt/thumbs and the absence of /mnt/upload5 on all application servers and image scalers
13:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix Category talk alias on mlwiki'
11:57 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Also add a Unicode 5.1 alias for Category talk on mlwiki'
11:52 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22286: Unicode 5.1 namespace alias for Category on mlwiki'
11:46 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryBacklinks.php 'Deploying r61571'
08:12 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js
08:11 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js

January 26

23:44 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enable file uploads.'
23:34 Fred: restarted morebot
23:31 atglenn: turning off web server on ms1 (previously: moved uploads to ms1, mounted ms7 everywhere, updated squid conf)
21:10 atglenn: copying uploaded data since last replication, from ms1 to ms7
21:05 atglenn: turned off file uploads temporarily
21:04 logmsgbot: ariel synchronized wmf-deployment/wmf-config/CommonSettings.php
20:50 Rob: rebooted srv125 per dc task entry by fred
20:35 tomaszf: enabling global file uploads notice
20:29 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'Deploying r61534'
20:29 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'Deploying r61534'
18:38 Rob: shutting down pdf2 to relocate its rack position.
18:37 Rob: !log mobile2 port not moved yet, as its port assignment on wikitech is invalid
18:08 Rob: shutting down mobile2 to relocate its rack location
18:07 Rob: bayes back online
18:00 Rob: bayes management lom had ip issues, resolved, still working on host
17:52 Rob: bayes is not coming back online properly, checking.
17:47 Rob: bayes rack location moved and updated in racktables, powering back up
17:34 Rob: shutting down bayes to move its rack location from b2-pmtpa to a2-pmtpa
14:42 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22274: Set $wgAutoConfirmCount to 10 on itwiktionary'
00:56 Tim: on hume: increased RCT proc count, reduced max lag to 1s, increased lag check frequency

January 25

22:36 Tim: ms3 is reporting "table full" for rc1 enwiki.blobs. Running ALTER TABLE.
19:33 Fred: stopped apache on srv225
19:01 Fred: adjusted apache.conf on srv225 to make it work again...
16:07 Rob: powering srv224 & srv225 back up
16:01 Rob: shutting down srv225 & srv224 to swap their power cables from single to Y split.
15:45 Rob: srv196 memory replaced, back online and in service
15:45 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'pushing out change to two hosts that were down a few minutes ago'
15:40 Rob: shutting down db28 to swap memory around in diagnostics (bad memory in system)
15:25 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'took down srv196 for hardware replacement'
15:20 Rob: shutting down srv196 to replace faulty memory dimm
12:19 Tim: started recompressTracked.sh on hume

January 23

00:16 aZaFred_: implemented ganglia detailled mysql stats for db10.
00:16 aZaFred_: implemented DB Lag detection script for DB10

January 22

23:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.hooks.php 'Deploy r61401'
23:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/CollapsibleTabs/CollapsibleTabs.hooks.php 'Deploy r61403'
21:29 brion: freed some space on srv218 removing old upload temp files from /tmp but / is still pretty tight (445m free)
20:19 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'Deploying r61387'
19:05 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22201: Add WZ: alias on itwiktionary'
15:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable $wgEnotifWatchlist on usabilitywiki'
11:38 mark: Deployed new gmond 3.0.3-2 package that conflicts with the incompatible ganglia 3.1 packages in Ubuntu, to avoid problems in our upcoming upgrade

January 21

14:13 mark: Added srv210 to LVS
14:11 mark: Reenabled srv245 in LVS
13:58 mark: Restarted Apache on srv207, srv210, srv233, srv257, srv245
09:30 mark: Replaced srv121 (decommissioned) by search13 in smokeping
07:59 AaronSchulz: Removed empty ct_tag rows from code_tags
07:47 Tim: cleaned up /tmp on srv180 (was out of disk space)

January 20

20:36 logmsgbot: root synchronized php-1.5/wmf-config/mc.php
16:38 Fred: killed long running queries created by civicrm on db9.
13:56 domas: did lots of changes on wikitech wiki. :)
02:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging on enwiki'
01:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling api logging for enwiki'
01:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling API request logging on dewiki'
00:03 RoanKattouw: Strike my last, was actually *enabling* it on dewiki
00:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on dewiki'
00:02 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on frwiki'

January 19

21:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling API request logging on frwiki'
21:36 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling API request logging on frwiki'
20:34 Fred: bugzilla upgrade completed. It might be up to an hour before attachments can be added due to DNS propagation.
19:41 Fred: Bugzilla as you know it is going down. Be back shortly.
18:21 Rob: updated blog.wikimedia.org wp-stats plugin to 1.6.1
16:03 Rob: ssds installed in db28, during boot it displays a memory error, investigating.
15:20 Rob: db28 offline while disks are swapped to solid state for testing.

January 18

05:02 Tim: removed some old xff logs to free up space for logrotate which I ran with the -f flag
05:00 Tim: fixed logrotate on nfs1, broken due to duplicated entries between /etc/logrotate.d/rsyslog and /etc/logrotate.d/syslog-ng. Left the syslog-ng ones in for now.
04:02 RoanKattouw: NFS /home is full

January 16

14:55 mark: Started backend squid on sq50
14:26 mark: sq19 has bad drive /dev/sdc
14:22 mark: dist-upgrade & reboot on sq19
14:14 mark: Shutdown sq20, bad disk /dev/sda
14:13 mark: Reenabled sq24 frontend in PyBal
14:03 Tim-away: disk space critical on srv167, cleaned up /tmp
06:07 apergos: "recovery", my *ss... cleaned up /tmp on srv181 to get some space back

January 15

23:53 tomaszf: starting test dewiki snapshot on snapshot2
17:17 Fred: modified gmetad config on zwinger and spence to reflect new apache 4cpu aggregator
17:16 Fred: added srv149 as a gmond aggregator in puppet.

January 14

17:28 domas: load-tested and fixed db19 to handle full bits workload (~22k/s), now again serving just tampa part.
01:32 apergos: moved bits to .2 again, all nameservers seem to be up and reflect the change (bits unresponsive again)

January 13

18:30 Andrew: [andrew@zwinger ~]$ sync-file wmf-config/CommonSettings.php 'Disable GIF scaling again, due to issues reported on village pump, bug 22041'
17:05 Rob: updated InitialiseSettings for bug 21174 and 21077
16:24 Rob: updated InitialiseSettings.php Bug 20508 Please enable Extention:NewUserMessage on en.Wikinews
11:50 domas: started overflow-watchdog ( http://p.defau.lt/?4_Y_Lrl9tVKS6fEYwkswKA ) on db19, sent bits load at it again
01:23 atglenn: using .2 for bits.pmtpa, did authdns-update, let's see what happens (bits was failing to respond)

January 12

19:52 rainman-sr: could someone please put search3,9 into search_pool 1 (with search1,4) on lvs3
19:37 Fred: Puppet: set spence as a ganglia aggregator for Misc tree.
19:23 Rob: usability/prototype linode was crashed, had to reboot
17:39 Fred: set wgDefaultSkin back to monobook on wikitech since vector is not operational.
16:08 mark: Arcor clients appear to have problems reaching our sites, traffic to Arcor over AMS-IX has been low since midnight UTC
15:52 Rob: srv222 & srv223 power reestablished, booting.
15:47 Rob: shutting down srv223 & srv222 to change out power cords.
15:46 Rob: db10 moved and back online.
15:36 Rob: db10 moved to sdtpa a2, powering up.
14:50 Rob: taking down db10 to relocate from pmtpa-b1 to sdtpa-a2
14:50 Rob: fixed issues with transcode2 and transcode3, completing base installation.
05:48 logmsgbot: tstarling synchronized php-1.5/includes/HTMLCacheUpdate.php
05:48 logmsgbot: tstarling synchronized php-1.5/includes/BacklinkCache.php
05:47 Tim: deploying r60962
01:17 Tim: on streber: removed a corrupt torrus DB file so it could be rebuilt, torrus should be working now
00:57 Tim: killed frozen torrus cron jobs and ran "torrus compile --tree=Network --force"
00:51 Tim: maybe torrus collector is still broken, trying /etc/init.d/torrus-common force-reload
00:46 Tim: with mpm-prefork managed to debug it fairly easily. Moved away permanently locked DB file render_cache.db, torrus.wikimedia.org is now fixed
00:39 Fred: restarting pdns on ns1
00:38 Tim: switching streber to apache2-mpm-prefork, can't work out why it's not working
00:22 Tim: trying "apache2 -X" on streber
00:00 Tim: restarting apache on streber

January 11

23:38 domas: logging the fact that we had cache layer meltdown at some point in time during the day
22:30 domas: leaving bits.pmtpa on db19's varnish, in case of troubles - uncomment bits.pmtpa .2 record in /etc/powerdns/templates/wikimedia.org and run authdns-update
19:43 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapped memcached from srv125 to srv232'
19:06 Rob: new apaches srv255, srv257 deployed. Updated node groups and synced nagios
19:03 Rob: new apache server srv254 deployed
18:24 atglenn: copy backlog of image data from ms1 to ms7 (running in screen as root on both boxes)
14:43 mark: Rebooting fuchsia, locked up again
14:24 mark: Increased load on knsq16-22 by upping lvs weight from 10 to 15

January 10

23:02 logmsgbot: midom synchronized php-1.5/wmf-config/lucene.php 'rainman asked, rainman guilty, hehehe'
23:01 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php
17:36 rainman-sr: search limit raised to 500 again, interwiki search re-enabled for "other" wikis
16:07 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'take ixia back out'
16:06 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put ixia back'
16:06 rainman-sr: restarting search cluster to deploy search13-19
15:58 domas: all bits serving switched back to text cluster, we have problems with all threads blocking on write(): http://p.defau.lt/?dOBxveiHj_ukjzupEBX3rA
15:19 rainman-sr: configuring search13-19, will leave search20 as spare
14:22 domas: apparently varnish worker threads are blocking on network output, ... :)
12:40 domas: full bits pmtpa load sent to sq1
12:04 domas: sending half of bits load to sq1
12:02 domas: set up separate geo balancing for bits via bits-geo.wikimedia.org
10:45 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'setting extension asset path to bits.wm'
10:42 logmsgbot: midom synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php
10:29 logmsgbot: midom synchronized php-1.5/includes/Setup.php
10:29 logmsgbot: midom synchronized php-1.5/includes/DefaultSettings.php
09:36 domas: moving over static assets to 'bits.wikimedia.org'
09:11 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php
09:09 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php
08:26 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
08:25 river: taking ixia out of rotation to dump commons

January 9

17:31 mark: Upgraded pdns-recursor to 3.1.7.2 on dobson, mchenry, lily
15:32 mark: Temporarily filtering all prefixes from 1299 on br1-knams, due to some balanced link blackholing issue
13:10 logmsgbot: midom ran sync-common-all
12:45 domas: restarted, that is :)
12:44 domas: fixed ns1, was deadlocked

January 8

21:48 Rob: nagios is flapping errors for esams hosts, but they are still up and functional. Perhaps due to new transit setup earlier today.
20:10 Rob: pushing dns update for flaggedrevssandbox project
16:09 Rob: finished mobile2 initial setup, gave hcatlin sudo rights to server
15:46 mark: Brought up transit session to 1257 on br1-knams
14:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:42 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

January 7

20:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22038: Enable Collection extension on skwiki'
20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Fix $wgUploadNavigationUrl on bnwiki'
20:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Enable patrolling on frwikiversity'
19:54 Rob: base install done on mobile2, pdf2, transcode1, transcode2
16:55 Rob: srv254-srv257 racked, drac online, network attached. needs installs
15:46 Rob: racked srv254-srv257. DRAC setup, cables run. Network not plugged in until mark provisions the ports.
02:51 Tim: restarted apache2 on wikitech to fix swapping
02:27 Fred: rebooted pascal as it was once again hung
01:57 mark: Prepended AS 43821 once on incoming prefixes from AS 1299

January 6

22:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable OptIn on trwikimedia'
17:17 Rob: updated dns for tr.wikimedia.org
14:47 Rob: updated techblog software to newest stable revision
14:01 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22027: Enable AbuseFilter and CAPTCHA on trwiki'
13:27 river: restarted slave on db26
12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Set $wgUploadNavigationUrl on bnwiki'
12:37 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'remove debugging code from yesterday'
12:34 RoanKattouw: Resuming GlobalUsage rebuild
08:48 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
08:47 river: removing db26 from s1 to dump

January 5

23:57 RoanKattouw: GlobalUsage refresh script properly handling replag now. Aborted it, will resume in the morning (CET)
23:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/refreshGlobalimagelinks.php 'remove debugging code'
23:54 RoanKattouw: Resuming GlobalUsage refresh script
23:53 logmsgbot: catrope synchronized php-1.5/includes/db/LoadBalancer.php 'Deploying r60705'
23:53 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploying r60705'
22:21 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Debugging'
22:11 RoanKattouw: Aborting GU refresh *again*, somehow it's not detecting slave lag right
22:07 RoanKattouw: Resuming GlobalUsage refresh script
22:04 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploy r60695 (wfWaitForSlaves fix)'
21:44 RoanKattouw: Running refreshGlobalimagelinks.php on all wikis from the smallest up
20:50 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'Deploy r60692 (GlobalUsage fixes)'
20:14 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'added outreach namespace to officewiki'
19:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikiversity'
19:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable GlobalUsage on all public wikis'
17:10 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapping srv110 for spare srv231.'
16:41 Fred: restarted memcached on srv110
16:10 mark: Installed search13-20 as new search servers
15:50 mark: Installed karmic on search19
15:45 Rob: updating dns for mobile2 and pdf2 servers
12:10 mark: dist-upgrade and reboot on sockpuppet
12:01 river: copied old wmvids to ~kate/wmvids
11:37 mark: Installed OS on search18
07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php
07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/api/ApiCodeDiff.php
07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.i18n.php
07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.php
06:48 Tim: restarted apache on srv199, APC cache corruption
02:33 tomaszf: compressing support-requests on locke to free up space
02:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabled $wgCopyUpload and $wgHTTPProxy on commons and test, breaks Lucene Search'

January 4

22:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22002: Add Opinions namespace on bgwikinews'
21:51 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to shut up exceptions thrown by srv218
21:29 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 20312: Re-enable GIF scaling on all wikis'
21:09 RoanKattouw: srv205 is throwing fatal errors because the geoip PHP module is not installed
20:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix typo'
20:45 RoanKattouw: srv218 has a full disk
20:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'plus proxy setting'
20:44 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20512: Enable $wgCopyUploads on test and commons'
20:44 mark: OS-installed Karmic on search13, search14, search16, search17, search20
20:40 RoanKattouw: Restarting refreshGlobalimageusage.php on Commons after it mysteriously disappeared
20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GIF scaling on testwiki'
19:39 RoanKattouw: Running refreshGlobalimagelinks.php on Commons again
19:38 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GlobalUsage on commons again'
19:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Turning on GlobalUsage on testwiki'
19:33 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageQuery.php 'Seems I forgot this file'
19:20 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to kill "MagicWordArray::parseMatch: parameter not found" error
19:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable GlobalUsage again, blank pages on commons'
19:06 RoanKattouw: Running rebuildGlobalimagelinks.php on Commons
19:03 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Re-enabling GlobalUsage'
18:57 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.i18n.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage_body.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/ApiQueryGlobalUsage.php 'r60611'
18:55 RoanKattouw: Deploying r60611 (GlobalUsage update) with individual sync-file's
18:49 RoanKattouw: Truncating globalimagelinks table on commonswiki
18:48 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Temp disable GlobalUsage'
18:46 RoanKattouw: Disabling GlobalUsage temporarily for software update
18:41 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Enabling WikimediaLicenseTexts on Commons'
18:40 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on Commons'
18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaGrammarForms.php 'r60608'
18:23 RoanKattouw: Deploying r60608 (WikimediaMessages update) using sync-file
17:47 Rob: search13-search20 racked and ready for install
17:17 Rob: updating dns for new search servers
14:57 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached box srv84'
14:45 Rob: moved power for asw-b3-pmtpa, srv86 and srv87 may flap.
14:43 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached boxes srv120, srv121'
14:31 Rob: decommissioned srv120, srv121 to make room for new search servers.
14:29 Rob: pulling srv34 from rack to decommission, need the space for new search servers.
14:09 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace memcached on just decommissioned host srv82'
13:57 Rob: srv82, srv83, srv84 decommissioned to make room for new search servers in rack
12:56 Tim: started squid-frontend on knsq8, knsq10, knsq11, knsq13, knsq14, knsq15, all crashed at roughly the same time as knsq9
12:52 Tim: started squid-frontend on knsq9, died at ~17:30 on the 3rd. Syslog shows many crashes, followed by "out of socket memory" a couple of hundred times, then silence

January 3

20:17 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60588, r60590'
17:29 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'added die(); somewher in there'
14:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'ixia is being put on a bench'

January 2

19:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Enable import from Commons on brwikimedia'

December 31

19:45 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Moving tracking back to db9 as it requires insert'
18:38 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Switching to query the slave db10 instead of the master db9'

December 30

23:37 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php
23:35 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
12:51 mark: Set /proc/sys/vm/min_free_kbytes to 65535 on brewster and streber, to see if it helps with the swapper page allocation failure bug
07:57 Tim: fixed private+secure file access, URLs were broken for months at least
07:50 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php
07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'possible resolution for boardwiki problem'
07:45 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'debugging hack'
07:34 Tim: created an account for myself on boardwiki to debug file upload issue

December 29

23:42 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'added tracking for 2009_notice51.'
18:27 tomaszf: dropping retention on storage2 to 4 xml snapshots
13:04 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60468'
12:50 RoanKattouw: Started logmsgbot (running as catrope instead of nobody)
12:32 RoanKattouw: logmsgbot down since fenari reboot, needs to be restarted by a root

December 28

21:02 RoanKattouw: Restarting refreshLinks for enwikibooks on hume
20:49 mark: Created 10G of swap space on fenari
20:07 domas: powercycled fenari, console not responsive
15:00 mark: Installed Karmic on LVS1, setting up a test network for LVS performance testing
14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 17338: Disable $wgRestrictDisplayTitle and enable subpages in the main namespace on rmwiki'
13:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21566: Enable Collection on viwiki'
13:42 RoanKattouw: Running refreshLinks on enwikinews per bug 19404
13:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19404: Set $wgCategoryPrefixedDefaultSortkey=false on enwikinews'
13:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21958: Create Book namespace on enwiki'

December 27

19:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Import sources, subpages for brwikimedia'
11:26 domas: db20 offlined disk array, after reboot booted into netinstall (but saw the array), after reset /SYS, and some operator's no-operation in BIOS and RAID controller setup screens, it booted up properly

December 26

19:59 DaBPunkt: (non-dev-entry) Many apaches in the US died, some db-server reported overusing. Problem fixed itself after some minutes.

December 25

20:24 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php
20:09 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixed frwiktionary config (sysops should be able to remove patroller)'

December 24

22:52 RoanKattouw: Running updateArticleCount.php on arwiki and arwiktionary
20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var AGAIN'
20:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var'
20:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21859: Enable Doublewiki on frwiktionary'
19:03 logmsgbot: catrope ran sync-common-all
18:56 RoanKattouw: Running sync-docroot, sync-apache, sync-common-all, apache-graceful-all
18:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Explicitly add langcodes for recently created wikis'
18:27 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19534: Add Page and Index namespaces on sourceswiki'
17:02 logmsgbot: catrope ran sync-common-all
16:57 RoanKattouw: Syncing updates for wiki creations
16:57 RoanKattouw: Not creating fiwikimedia, already created
16:54 RoanKattouw: Adding fiwikimedia (sans DNS entry) per bug 20502
16:41 RoanKattouw: Adding arbcom_fiwiki (sans DNS entry) per bug 21375
16:32 RoanKattouw: Adding dkwikimedia (sans DNS entry) per bug 21009
16:23 RoanKattouw: Adding brwikimedia (sans DNS entry) per bug 21149
16:14 RoanKattouw: Adding pcdwiki (sans DNS entry) per bug 21634
16:06 RoanKattouw: Made raw, non-highlighted versions of .php files visible through noc.wikimedia.org/conf
15:55 RoanKattouw: Made abusefilter.php visible through noc.wikimedia.org/conf
15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
15:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix name of oldwikisource -> sourceswiki'
15:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable ProofreadPage for oldwikisource, which strangely is not part of the wikisource group'
15:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20402: Enable Collection extension on mswiki'
15:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21389: Set $wgDisabledVariants on Chinese wikis'
14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21527: Enable Special:Cite on itwiktionary'
14:28 RoanKattouw: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21100: Add Tac_gia namespace to $wgNamespacesToBeSearchedDefault on viwikisource'
14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21011: Restrict uploads, change $wgUploadNavigationUrl on enwikibooks'
14:10 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21236: Set $wgEnableNewpagesUserFilter=true; on plwiki'
14:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21370: Create custom namespaces on etwikisource, fix namespace alias on arwikisource'
13:51 RoanKattouw: Running updateArticleCount.php on zhwikisource for bug 20998
13:48 domas: s1 and s2 had rolling restarts to mysql#3193 build
13:48 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Rename autopatroller group to autoreviewer on zhwiki'
13:44 domas: s2 master switched to db15-bin.000001:106
13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
13:32 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'misspelled right name'
13:29 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20721: Add abusefilter-viewprivate right for sysops on enwiki'
13:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20561: Change sitename on tkwiki'
12:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 15237: Localize sitename and Wikipedia namespace on nahwiki'

December 23

16:02 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:35 Rob: db27 back online with new fan controller board.
13:53 Rob: shutting down mysql on db27 for hardware replacement.
12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Actually enable patrolling on frwikibooks and frwiktionary'
08:22 Tim: on isidore, also disabled the CentralNotice job that was running every 20 minutes from /etc/crontab
05:16 Tim: disabled CentralNotice rebuilds for donate.dev.wikimedia.org, was overloading isidore (which is only a single-core pentium)

December 22

23:45 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Added 2009_Notice49 to tracking.'

December 21

19:25 Rob: ms8 still borked.
19:24 Rob: restarted mysql on db27
19:12 Rob: removed the fan boards in db27 and cleaned connections (were filthy with toner or dust or something) and replaced them. system booting back online. (Will watch it for errors for the next week.)
18:36 Rob: shutting down db27 mysql manually for troubleshooting hardware on the system
18:04 Rob: db28 mainboard and fans replaced, booting back online.
17:35 Rob: restarted mysql on srv185, since it appears to be a ext. storage slave.
17:32 Rob: srv185 memory replaced, back online.
17:20 Rob: shutting down srv185 to swap bad dimm1
16:52 Rob: swapped out bad disk in db30, all leds are green now.
16:22 Rob: swapped out bad memory in sq32, booting it back up.
16:13 Rob: shutting down sq32 to swap out bad memory
07:29 logmsgbot: tstarling synchronized php-1.5/extensions/intersection/DynamicPageList.php 'deploying r60254'

December 20

21:29 RoanKattouw: Clean up entries with rev_user_text='conversion script' (capitalize C) on dewiki for bug 21910
21:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'forgot $wgRemoveGroups'
21:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikibooks and frwiktionary'
21:03 RoanKattouw: Running namespaceDupes on simplewiktionary for bug 21906
21:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21906: Add WT: -> Wiktionary: alias on simplewiktionary'
10:25 domas: um, no more stalls?

December 19

20:53 RoanKattouw: Running namespaceDupes on zhwiki
00:41 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding JimmyAppeal[8-9] to tracking'

December 18

22:25 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'revert my last: logo too large'
22:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21213: Logo for acewiki'
21:57 logmsgbot: catrope ran sync-common-all
21:56 RoanKattouw: Closing nlwikinews per bug 20325
21:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix mistake (autopatrol vs. autopatroller)'
20:24 mark: Updated firewall for allowing ssh access from fenari on loudon
13:21 mark: Stopped oprofile on db22
11:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Add autopatroller group, allow sysops to add/remove confirmed on zhwiki'
11:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20910: Set $wgUploadNavigationUrl on nlwiki'
11:18 RoanKattouw: srv205 has full disk
11:09 domas: s1 switched master to db16-bin.000001
11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
00:14 aZa_: restarted srv141. Puppet went crazy cpu hungry.

December 17

23:54 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php
23:53 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
23:15 RoanKattouw: Strike that, frwikinews
23:07 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21436: Activate flood flag on frwikibooks'
20:32 Rob: sq20 reinstalled, squid package installed. Not back in service until I check with Mark about some puppet settings.
20:07 Rob: resynced nagios to remove sq27
20:03 Rob: sq20 disk died, replaced and reinstalling.
20:01 Rob: sq27 decomissioned (stole drive to fix sq20)
19:59 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Allow sysops to add/remove patroller and autopatroller on hewiki'
18:54 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php
17:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21158: Enable $wgBlockAllowsUTEdit on enwikiquote'
16:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21121: Enable flood flag on simplewikibooks'
16:22 RoanKattouw: Running namespaceDupes on iawiktionary
16:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiktionary'
16:17 Rob: pushing update to dns for search13 dns
16:09 RoanKattouw: Running namespaceDupes on iawiki for bug 21241
16:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiki'
16:04 RoanKattouw: Running namespaceDupes on brwiki for bug 21417
16:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21417: Change name of project namespace on brwiki, add old name as alias'
15:52 mark: Killed memcached on browne
15:51 mark: Started ircd on browne
15:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21041: Allow sysops to grant and bcrats to remove transwiki on nowiki'
15:46 Rob: browne back online after move.
15:46 Rob: nagios synced to node group files without will.
15:45 Rob: will decommissioned, pulled from rack
15:40 Rob: shutting down browne to move its rack location, will be back online shortly.
15:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Add patroller and autopatroller groups on hewiki'
15:26 mark: Moved udpmcast from browne to dobson
13:11 mark: Fixed Racktables rack thumb problem by installing php5-gd; it was just serving cached thumbs
11:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21306: Allow sysops to add/remove patroller and autopatrol on hrwiki'
11:00 RoanKattouw: Running namespaceDueps on mlwiki for bug 21277
10:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21277: Add Portal and Portal talk as aliases on mlwiki'
10:53 RoanKattouw: Running namespaceDupes.php on cawikibooks for bug 20980
10:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20980: Add Viquiprojecte namespace on cawikibooks'
10:47 RoanKattouw: Resyncing srv194 and rebuilding caches
10:46 domas: srv194 had 5G-sized oprofile error log, hehe hehehe
10:39 RoanKattouw: srv194 disk still full
10:39 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Allow sysops to add/remove rollbacker on gawiki'

December 16

20:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix mistake'
20:32 RoanKattouw: srv194 disk full
20:32 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Add rollbacker group on gawiki'
19:56 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2/jquery-ui-1.7.2.js 'Deploying r60130'
19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.min.js 'Deploying r60130'
19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.js 'Deploying r60130'
19:53 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed groupOvrrides for fiwiki for arbcom.'
17:44 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'FR labs config - remove excess level'
16:36 hcatlin: new and improved mobile with compressed memcached + new homepags + better utf-8 handling
16:24 hcatlin: taking down mobile1 for a large software update
15:59 mark: Disabled NIS on zwinger
15:54 mark: Disabled NIS on ms1
15:53 mark: Disabled NIS on ms4
15:47 mark: Installed puppet on ms1 and ms7
15:27 Andrew: srv129, srv123, srv120, srv95 seem to be in swapdeath
15:25 logmsgbot: andrew ran sync-common-all
15:24 Andrew: sync-common-all caused memory spike on apaches again, site seems to still be up though
15:17 Andrew: Updating LiquidThreads to trunk state, using sync-common-all
12:58 RoanKattouw: Tim ran rebuildTemplates.php on hume for all languages
12:52 Andrew: Fixed morebots, was choking because the Server Admin Log was archived.
12:50 Andrew: Moved Morebots init script into a line in rc.local. Morebots seems to have been down due to Freenode's DDoS problem, maybe it isn't exiting properly when disconnected from the server
12:46 Tim: ran rebuildTemplates.php on hume for all languages

Server admin log/Archive 20

February 19

February 18

February 17

February 16

February 15

February 14

February 13

February 12

February 11

February 10

February 9

February 8

February 7

February 6

February 5

February 4

February 3

February 2

February 1

January 31

January 30

January 29

January 28

January 27

January 26

January 25

January 23

January 22

January 21

January 20

January 19

January 18

January 16

January 15

January 14

January 13

January 12

January 11

January 10

January 9

January 8

January 7

January 6

January 5

January 4

January 3

January 2

December 31

December 30

December 29

December 28

December 27

December 26

December 25

December 24

December 23

December 22

December 21

December 20

December 19

December 18

December 17

December 16

Archives

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Ops documentation

Wiki

Toolbox