Server admin log/Archive 20
From Wikitech
< Server admin log(Difference between revisions)
(test! (domas)) |
|||
| Line 1: | Line 1: | ||
== January 20 == | == January 20 == | ||
| − | |||
* 02:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging on enwiki' | * 02:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging on enwiki' | ||
* 01:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling api logging for enwiki' | * 01:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling api logging for enwiki' | ||
Revision as of 13:55, 20 January 2010
January 20
- 02:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging on enwiki'
- 01:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling api logging for enwiki'
- 01:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling API request logging on dewiki'
- 00:03 RoanKattouw: Strike my last, was actually *enabling* it on dewiki
- 00:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on dewiki'
- 00:02 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on frwiki'
January 19
- 21:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling API request logging on frwiki'
- 21:36 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling API request logging on frwiki'
- 20:34 Fred: bugzilla upgrade completed. It might be up to an hour before attachments can be added due to DNS propagation.
- 19:41 Fred: Bugzilla as you know it is going down. Be back shortly.
- 18:21 Rob: updated blog.wikimedia.org wp-stats plugin to 1.6.1
- 16:03 Rob: ssds installed in db28, during boot it displays a memory error, investigating.
- 15:20 Rob: db28 offline while disks are swapped to solid state for testing.
January 18
- 05:02 Tim: removed some old xff logs to free up space for logrotate which I ran with the -f flag
- 05:00 Tim: fixed logrotate on nfs1, broken due to duplicated entries between /etc/logrotate.d/rsyslog and /etc/logrotate.d/syslog-ng. Left the syslog-ng ones in for now.
- 04:02 RoanKattouw: NFS /home is full
January 16
- 14:55 mark: Started backend squid on sq50
- 14:26 mark: sq19 has bad drive /dev/sdc
- 14:22 mark: dist-upgrade & reboot on sq19
- 14:14 mark: Shutdown sq20, bad disk /dev/sda
- 14:13 mark: Reenabled sq24 frontend in PyBal
- 14:03 Tim-away: disk space critical on srv167, cleaned up /tmp
- 06:07 apergos: "recovery", my *ss... cleaned up /tmp on srv181 to get some space back
January 15
- 23:53 tomaszf: starting test dewiki snapshot on snapshot2
- 17:17 Fred: modified gmetad config on zwinger and spence to reflect new apache 4cpu aggregator
- 17:16 Fred: added srv149 as a gmond aggregator in puppet.
January 14
- 17:28 domas: load-tested and fixed db19 to handle full bits workload (~22k/s), now again serving just tampa part.
- 01:32 apergos: moved bits to .2 again, all nameservers seem to be up and reflect the change (bits unresponsive again)
January 13
- 18:30 Andrew: [andrew@zwinger ~]$ sync-file wmf-config/CommonSettings.php 'Disable GIF scaling again, due to issues reported on village pump, bug 22041'
- 17:05 Rob: updated InitialiseSettings for bug 21174 and 21077
- 16:24 Rob: updated InitialiseSettings.php Bug 20508 Please enable Extention:NewUserMessage on en.Wikinews
- 11:50 domas: started overflow-watchdog ( http://p.defau.lt/?4_Y_Lrl9tVKS6fEYwkswKA ) on db19, sent bits load at it again
- 01:23 atglenn: using .2 for bits.pmtpa, did authdns-update, let's see what happens (bits was failing to respond)
January 12
- 19:52 rainman-sr: could someone please put search3,9 into search_pool 1 (with search1,4) on lvs3
- 19:37 Fred: Puppet: set spence as a ganglia aggregator for Misc tree.
- 19:23 Rob: usability/prototype linode was crashed, had to reboot
- 17:39 Fred: set wgDefaultSkin back to monobook on wikitech since vector is not operational.
- 16:08 mark: Arcor clients appear to have problems reaching our sites, traffic to Arcor over AMS-IX has been low since midnight UTC
- 15:52 Rob: srv222 & srv223 power reestablished, booting.
- 15:47 Rob: shutting down srv223 & srv222 to change out power cords.
- 15:46 Rob: db10 moved and back online.
- 15:36 Rob: db10 moved to sdtpa a2, powering up.
- 14:50 Rob: taking down db10 to relocate from pmtpa-b1 to sdtpa-a2
- 14:50 Rob: fixed issues with transcode2 and transcode3, completing base installation.
- 05:48 logmsgbot: tstarling synchronized php-1.5/includes/HTMLCacheUpdate.php
- 05:48 logmsgbot: tstarling synchronized php-1.5/includes/BacklinkCache.php
- 05:47 Tim: deploying r60962
- 01:17 Tim: on streber: removed a corrupt torrus DB file so it could be rebuilt, torrus should be working now
- 00:57 Tim: killed frozen torrus cron jobs and ran "torrus compile --tree=Network --force"
- 00:51 Tim: maybe torrus collector is still broken, trying /etc/init.d/torrus-common force-reload
- 00:46 Tim: with mpm-prefork managed to debug it fairly easily. Moved away permanently locked DB file render_cache.db, torrus.wikimedia.org is now fixed
- 00:39 Fred: restarting pdns on ns1
- 00:38 Tim: switching streber to apache2-mpm-prefork, can't work out why it's not working
- 00:22 Tim: trying "apache2 -X" on streber
- 00:00 Tim: restarting apache on streber
January 11
- 23:38 domas: logging the fact that we had cache layer meltdown at some point in time during the day
- 22:30 domas: leaving bits.pmtpa on db19's varnish, in case of troubles - uncomment bits.pmtpa .2 record in /etc/powerdns/templates/wikimedia.org and run authdns-update
- 19:43 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapped memcached from srv125 to srv232'
- 19:06 Rob: new apaches srv255, srv257 deployed. Updated node groups and synced nagios
- 19:03 Rob: new apache server srv254 deployed
- 18:24 atglenn: copy backlog of image data from ms1 to ms7 (running in screen as root on both boxes)
- 14:43 mark: Rebooting fuchsia, locked up again
- 14:24 mark: Increased load on knsq16-22 by upping lvs weight from 10 to 15
January 10
- 23:02 logmsgbot: midom synchronized php-1.5/wmf-config/lucene.php 'rainman asked, rainman guilty, hehehe'
- 23:01 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php
- 17:36 rainman-sr: search limit raised to 500 again, interwiki search re-enabled for "other" wikis
- 16:07 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'take ixia back out'
- 16:06 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put ixia back'
- 16:06 rainman-sr: restarting search cluster to deploy search13-19
- 15:58 domas: all bits serving switched back to text cluster, we have problems with all threads blocking on write(): http://p.defau.lt/?dOBxveiHj_ukjzupEBX3rA
- 15:19 rainman-sr: configuring search13-19, will leave search20 as spare
- 14:22 domas: apparently varnish worker threads are blocking on network output, ... :)
- 12:40 domas: full bits pmtpa load sent to sq1
- 12:04 domas: sending half of bits load to sq1
- 12:02 domas: set up separate geo balancing for bits via bits-geo.wikimedia.org
- 10:45 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'setting extension asset path to bits.wm'
- 10:42 logmsgbot: midom synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php
- 10:29 logmsgbot: midom synchronized php-1.5/includes/Setup.php
- 10:29 logmsgbot: midom synchronized php-1.5/includes/DefaultSettings.php
- 09:36 domas: moving over static assets to 'bits.wikimedia.org'
- 09:11 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php
- 09:09 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php
- 08:26 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
- 08:25 river: taking ixia out of rotation to dump commons
January 9
- 17:31 mark: Upgraded pdns-recursor to 3.1.7.2 on dobson, mchenry, lily
- 15:32 mark: Temporarily filtering all prefixes from 1299 on br1-knams, due to some balanced link blackholing issue
- 13:10 logmsgbot: midom ran sync-common-all
- 12:45 domas: restarted, that is :)
- 12:44 domas: fixed ns1, was deadlocked
January 8
- 21:48 Rob: nagios is flapping errors for esams hosts, but they are still up and functional. Perhaps due to new transit setup earlier today.
- 20:10 Rob: pushing dns update for flaggedrevssandbox project
- 16:09 Rob: finished mobile2 initial setup, gave hcatlin sudo rights to server
- 15:46 mark: Brought up transit session to 1257 on br1-knams
- 14:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 11:42 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
January 7
- 20:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22038: Enable Collection extension on skwiki'
- 20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Fix $wgUploadNavigationUrl on bnwiki'
- 20:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Enable patrolling on frwikiversity'
- 19:54 Rob: base install done on mobile2, pdf2, transcode1, transcode2
- 16:55 Rob: srv254-srv257 racked, drac online, network attached. needs installs
- 15:46 Rob: racked srv254-srv257. DRAC setup, cables run. Network not plugged in until mark provisions the ports.
- 02:51 Tim: restarted apache2 on wikitech to fix swapping
- 02:27 Fred: rebooted pascal as it was once again hung
- 01:57 mark: Prepended AS 43821 once on incoming prefixes from AS 1299
January 6
- 22:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable OptIn on trwikimedia'
- 17:17 Rob: updated dns for tr.wikimedia.org
- 14:47 Rob: updated techblog software to newest stable revision
- 14:01 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22027: Enable AbuseFilter and CAPTCHA on trwiki'
- 13:27 river: restarted slave on db26
- 12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Set $wgUploadNavigationUrl on bnwiki'
- 12:37 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'remove debugging code from yesterday'
- 12:34 RoanKattouw: Resuming GlobalUsage rebuild
- 08:48 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
- 08:47 river: removing db26 from s1 to dump
January 5
- 23:57 RoanKattouw: GlobalUsage refresh script properly handling replag now. Aborted it, will resume in the morning (CET)
- 23:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/refreshGlobalimagelinks.php 'remove debugging code'
- 23:54 RoanKattouw: Resuming GlobalUsage refresh script
- 23:53 logmsgbot: catrope synchronized php-1.5/includes/db/LoadBalancer.php 'Deploying r60705'
- 23:53 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploying r60705'
- 22:21 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Debugging'
- 22:11 RoanKattouw: Aborting GU refresh *again*, somehow it's not detecting slave lag right
- 22:07 RoanKattouw: Resuming GlobalUsage refresh script
- 22:04 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploy r60695 (wfWaitForSlaves fix)'
- 21:44 RoanKattouw: Running refreshGlobalimagelinks.php on all wikis from the smallest up
- 20:50 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'Deploy r60692 (GlobalUsage fixes)'
- 20:14 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'added outreach namespace to officewiki'
- 19:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikiversity'
- 19:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable GlobalUsage on all public wikis'
- 17:10 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapping srv110 for spare srv231.'
- 16:41 Fred: restarted memcached on srv110
- 16:10 mark: Installed search13-20 as new search servers
- 15:50 mark: Installed karmic on search19
- 15:45 Rob: updating dns for mobile2 and pdf2 servers
- 12:10 mark: dist-upgrade and reboot on sockpuppet
- 12:01 river: copied old wmvids to ~kate/wmvids
- 11:37 mark: Installed OS on search18
- 07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php
- 07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/api/ApiCodeDiff.php
- 07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.i18n.php
- 07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.php
- 06:48 Tim: restarted apache on srv199, APC cache corruption
- 02:33 tomaszf: compressing support-requests on locke to free up space
- 02:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabled $wgCopyUpload and $wgHTTPProxy on commons and test, breaks Lucene Search'
January 4
- 22:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22002: Add Opinions namespace on bgwikinews'
- 21:51 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to shut up exceptions thrown by srv218
- 21:29 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 20312: Re-enable GIF scaling on all wikis'
- 21:09 RoanKattouw: srv205 is throwing fatal errors because the geoip PHP module is not installed
- 20:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix typo'
- 20:45 RoanKattouw: srv218 has a full disk
- 20:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'plus proxy setting'
- 20:44 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20512: Enable $wgCopyUploads on test and commons'
- 20:44 mark: OS-installed Karmic on search13, search14, search16, search17, search20
- 20:40 RoanKattouw: Restarting refreshGlobalimageusage.php on Commons after it mysteriously disappeared
- 20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GIF scaling on testwiki'
- 19:39 RoanKattouw: Running refreshGlobalimagelinks.php on Commons again
- 19:38 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GlobalUsage on commons again'
- 19:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Turning on GlobalUsage on testwiki'
- 19:33 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageQuery.php 'Seems I forgot this file'
- 19:20 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to kill "MagicWordArray::parseMatch: parameter not found" error
- 19:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable GlobalUsage again, blank pages on commons'
- 19:06 RoanKattouw: Running rebuildGlobalimagelinks.php on Commons
- 19:03 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Re-enabling GlobalUsage'
- 18:57 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.i18n.php 'r60611'
- 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'r60611'
- 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'r60611'
- 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'r60611'
- 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage_body.php 'r60611'
- 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/ApiQueryGlobalUsage.php 'r60611'
- 18:55 RoanKattouw: Deploying r60611 (GlobalUsage update) with individual sync-file's
- 18:49 RoanKattouw: Truncating globalimagelinks table on commonswiki
- 18:48 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Temp disable GlobalUsage'
- 18:46 RoanKattouw: Disabling GlobalUsage temporarily for software update
- 18:41 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Enabling WikimediaLicenseTexts on Commons'
- 18:40 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on Commons'
- 18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
- 18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
- 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r60608'
- 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r60608'
- 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.php 'r60608'
- 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.php 'r60608'
- 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaGrammarForms.php 'r60608'
- 18:23 RoanKattouw: Deploying r60608 (WikimediaMessages update) using sync-file
- 17:47 Rob: search13-search20 racked and ready for install
- 17:17 Rob: updating dns for new search servers
- 14:57 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached box srv84'
- 14:45 Rob: moved power for asw-b3-pmtpa, srv86 and srv87 may flap.
- 14:43 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached boxes srv120, srv121'
- 14:31 Rob: decommissioned srv120, srv121 to make room for new search servers.
- 14:29 Rob: pulling srv34 from rack to decommission, need the space for new search servers.
- 14:09 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace memcached on just decommissioned host srv82'
- 13:57 Rob: srv82, srv83, srv84 decommissioned to make room for new search servers in rack
- 12:56 Tim: started squid-frontend on knsq8, knsq10, knsq11, knsq13, knsq14, knsq15, all crashed at roughly the same time as knsq9
- 12:52 Tim: started squid-frontend on knsq9, died at ~17:30 on the 3rd. Syslog shows many crashes, followed by "out of socket memory" a couple of hundred times, then silence
January 3
- 20:17 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60588, r60590'
- 17:29 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'added die(); somewher in there'
- 14:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 14:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'ixia is being put on a bench'
January 2
- 19:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Enable import from Commons on brwikimedia'
December 31
- 19:45 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Moving tracking back to db9 as it requires insert'
- 18:38 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Switching to query the slave db10 instead of the master db9'
December 30
- 23:37 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php
- 23:35 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
- 12:51 mark: Set /proc/sys/vm/min_free_kbytes to 65535 on brewster and streber, to see if it helps with the swapper page allocation failure bug
- 07:57 Tim: fixed private+secure file access, URLs were broken for months at least
- 07:50 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php
- 07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'possible resolution for boardwiki problem'
- 07:45 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'debugging hack'
- 07:34 Tim: created an account for myself on boardwiki to debug file upload issue
December 29
- 23:42 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'added tracking for 2009_notice51.'
- 18:27 tomaszf: dropping retention on storage2 to 4 xml snapshots
- 13:04 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60468'
- 12:50 RoanKattouw: Started logmsgbot (running as catrope instead of nobody)
- 12:32 RoanKattouw: logmsgbot down since fenari reboot, needs to be restarted by a root
December 28
- 21:02 RoanKattouw: Restarting refreshLinks for enwikibooks on hume
- 20:49 mark: Created 10G of swap space on fenari
- 20:07 domas: powercycled fenari, console not responsive
- 15:00 mark: Installed Karmic on LVS1, setting up a test network for LVS performance testing
- 14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 17338: Disable $wgRestrictDisplayTitle and enable subpages in the main namespace on rmwiki'
- 13:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21566: Enable Collection on viwiki'
- 13:42 RoanKattouw: Running refreshLinks on enwikinews per bug 19404
- 13:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19404: Set $wgCategoryPrefixedDefaultSortkey=false on enwikinews'
- 13:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21958: Create Book namespace on enwiki'
December 27
- 19:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Import sources, subpages for brwikimedia'
- 11:26 domas: db20 offlined disk array, after reboot booted into netinstall (but saw the array), after reset /SYS, and some operator's no-operation in BIOS and RAID controller setup screens, it booted up properly
December 26
- 19:59 DaBPunkt: (non-dev-entry) Many apaches in the US died, some db-server reported overusing. Problem fixed itself after some minutes.
December 25
- 20:24 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php
- 20:09 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixed frwiktionary config (sysops should be able to remove patroller)'
December 24
- 22:52 RoanKattouw: Running updateArticleCount.php on arwiki and arwiktionary
- 20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var AGAIN'
- 20:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var'
- 20:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21859: Enable Doublewiki on frwiktionary'
- 19:03 logmsgbot: catrope ran sync-common-all
- 18:56 RoanKattouw: Running sync-docroot, sync-apache, sync-common-all, apache-graceful-all
- 18:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Explicitly add langcodes for recently created wikis'
- 18:27 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19534: Add Page and Index namespaces on sourceswiki'
- 17:02 logmsgbot: catrope ran sync-common-all
- 16:57 RoanKattouw: Syncing updates for wiki creations
- 16:57 RoanKattouw: Not creating fiwikimedia, already created
- 16:54 RoanKattouw: Adding fiwikimedia (sans DNS entry) per bug 20502
- 16:41 RoanKattouw: Adding arbcom_fiwiki (sans DNS entry) per bug 21375
- 16:32 RoanKattouw: Adding dkwikimedia (sans DNS entry) per bug 21009
- 16:23 RoanKattouw: Adding brwikimedia (sans DNS entry) per bug 21149
- 16:14 RoanKattouw: Adding pcdwiki (sans DNS entry) per bug 21634
- 16:06 RoanKattouw: Made raw, non-highlighted versions of .php files visible through noc.wikimedia.org/conf
- 15:55 RoanKattouw: Made abusefilter.php visible through noc.wikimedia.org/conf
- 15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
- 15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
- 15:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix name of oldwikisource -> sourceswiki'
- 15:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable ProofreadPage for oldwikisource, which strangely is not part of the wikisource group'
- 15:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20402: Enable Collection extension on mswiki'
- 15:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21389: Set $wgDisabledVariants on Chinese wikis'
- 14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21527: Enable Special:Cite on itwiktionary'
- 14:28 RoanKattouw: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21100: Add Tac_gia namespace to $wgNamespacesToBeSearchedDefault on viwikisource'
- 14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21011: Restrict uploads, change $wgUploadNavigationUrl on enwikibooks'
- 14:10 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21236: Set $wgEnableNewpagesUserFilter=true; on plwiki'
- 14:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21370: Create custom namespaces on etwikisource, fix namespace alias on arwikisource'
- 13:51 RoanKattouw: Running updateArticleCount.php on zhwikisource for bug 20998
- 13:48 domas: s1 and s2 had rolling restarts to mysql#3193 build
- 13:48 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Rename autopatroller group to autoreviewer on zhwiki'
- 13:44 domas: s2 master switched to db15-bin.000001:106
- 13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 13:32 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'misspelled right name'
- 13:29 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20721: Add abusefilter-viewprivate right for sysops on enwiki'
- 13:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20561: Change sitename on tkwiki'
- 12:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 15237: Localize sitename and Wikipedia namespace on nahwiki'
December 23
- 16:02 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 14:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 14:35 Rob: db27 back online with new fan controller board.
- 13:53 Rob: shutting down mysql on db27 for hardware replacement.
- 12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Actually enable patrolling on frwikibooks and frwiktionary'
- 08:22 Tim: on isidore, also disabled the CentralNotice job that was running every 20 minutes from /etc/crontab
- 05:16 Tim: disabled CentralNotice rebuilds for donate.dev.wikimedia.org, was overloading isidore (which is only a single-core pentium)
December 22
- 23:45 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Added 2009_Notice49 to tracking.'
December 21
- 19:25 Rob: ms8 still borked.
- 19:24 Rob: restarted mysql on db27
- 19:12 Rob: removed the fan boards in db27 and cleaned connections (were filthy with toner or dust or something) and replaced them. system booting back online. (Will watch it for errors for the next week.)
- 18:36 Rob: shutting down db27 mysql manually for troubleshooting hardware on the system
- 18:04 Rob: db28 mainboard and fans replaced, booting back online.
- 17:35 Rob: restarted mysql on srv185, since it appears to be a ext. storage slave.
- 17:32 Rob: srv185 memory replaced, back online.
- 17:20 Rob: shutting down srv185 to swap bad dimm1
- 16:52 Rob: swapped out bad disk in db30, all leds are green now.
- 16:22 Rob: swapped out bad memory in sq32, booting it back up.
- 16:13 Rob: shutting down sq32 to swap out bad memory
- 07:29 logmsgbot: tstarling synchronized php-1.5/extensions/intersection/DynamicPageList.php 'deploying r60254'
December 20
- 21:29 RoanKattouw: Clean up entries with rev_user_text='conversion script' (capitalize C) on dewiki for bug 21910
- 21:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'forgot $wgRemoveGroups'
- 21:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikibooks and frwiktionary'
- 21:03 RoanKattouw: Running namespaceDupes on simplewiktionary for bug 21906
- 21:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21906: Add WT: -> Wiktionary: alias on simplewiktionary'
- 10:25 domas: um, no more stalls?
December 19
- 20:53 RoanKattouw: Running namespaceDupes on zhwiki
- 00:41 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding JimmyAppeal[8-9] to tracking'
December 18
- 22:25 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'revert my last: logo too large'
- 22:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21213: Logo for acewiki'
- 21:57 logmsgbot: catrope ran sync-common-all
- 21:56 RoanKattouw: Closing nlwikinews per bug 20325
- 21:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix mistake (autopatrol vs. autopatroller)'
- 20:24 mark: Updated firewall for allowing ssh access from fenari on loudon
- 13:21 mark: Stopped oprofile on db22
- 11:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Add autopatroller group, allow sysops to add/remove confirmed on zhwiki'
- 11:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20910: Set $wgUploadNavigationUrl on nlwiki'
- 11:18 RoanKattouw: srv205 has full disk
- 11:09 domas: s1 switched master to db16-bin.000001
- 11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 11:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
- 00:14 aZa_: restarted srv141. Puppet went crazy cpu hungry.
December 17
- 23:54 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php
- 23:53 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
- 23:15 RoanKattouw: Strike that, frwikinews
- 23:07 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21436: Activate flood flag on frwikibooks'
- 20:32 Rob: sq20 reinstalled, squid package installed. Not back in service until I check with Mark about some puppet settings.
- 20:07 Rob: resynced nagios to remove sq27
- 20:03 Rob: sq20 disk died, replaced and reinstalling.
- 20:01 Rob: sq27 decomissioned (stole drive to fix sq20)
- 19:59 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Allow sysops to add/remove patroller and autopatroller on hewiki'
- 18:54 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php
- 17:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21158: Enable $wgBlockAllowsUTEdit on enwikiquote'
- 16:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21121: Enable flood flag on simplewikibooks'
- 16:22 RoanKattouw: Running namespaceDupes on iawiktionary
- 16:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiktionary'
- 16:17 Rob: pushing update to dns for search13 dns
- 16:09 RoanKattouw: Running namespaceDupes on iawiki for bug 21241
- 16:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiki'
- 16:04 RoanKattouw: Running namespaceDupes on brwiki for bug 21417
- 16:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21417: Change name of project namespace on brwiki, add old name as alias'
- 15:52 mark: Killed memcached on browne
- 15:51 mark: Started ircd on browne
- 15:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21041: Allow sysops to grant and bcrats to remove transwiki on nowiki'
- 15:46 Rob: browne back online after move.
- 15:46 Rob: nagios synced to node group files without will.
- 15:45 Rob: will decommissioned, pulled from rack
- 15:40 Rob: shutting down browne to move its rack location, will be back online shortly.
- 15:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Add patroller and autopatroller groups on hewiki'
- 15:26 mark: Moved udpmcast from browne to dobson
- 13:11 mark: Fixed Racktables rack thumb problem by installing php5-gd; it was just serving cached thumbs
- 11:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21306: Allow sysops to add/remove patroller and autopatrol on hrwiki'
- 11:00 RoanKattouw: Running namespaceDueps on mlwiki for bug 21277
- 10:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21277: Add Portal and Portal talk as aliases on mlwiki'
- 10:53 RoanKattouw: Running namespaceDupes.php on cawikibooks for bug 20980
- 10:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20980: Add Viquiprojecte namespace on cawikibooks'
- 10:47 RoanKattouw: Resyncing srv194 and rebuilding caches
- 10:46 domas: srv194 had 5G-sized oprofile error log, hehe hehehe
- 10:39 RoanKattouw: srv194 disk still full
- 10:39 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Allow sysops to add/remove rollbacker on gawiki'
December 16
- 20:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix mistake'
- 20:32 RoanKattouw: srv194 disk full
- 20:32 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Add rollbacker group on gawiki'
- 19:56 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2/jquery-ui-1.7.2.js 'Deploying r60130'
- 19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.min.js 'Deploying r60130'
- 19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.js 'Deploying r60130'
- 19:53 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed groupOvrrides for fiwiki for arbcom.'
- 17:44 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'FR labs config - remove excess level'
- 16:36 hcatlin: new and improved mobile with compressed memcached + new homepags + better utf-8 handling
- 16:24 hcatlin: taking down mobile1 for a large software update
- 15:59 mark: Disabled NIS on zwinger
- 15:54 mark: Disabled NIS on ms1
- 15:53 mark: Disabled NIS on ms4
- 15:47 mark: Installed puppet on ms1 and ms7
- 15:27 Andrew: srv129, srv123, srv120, srv95 seem to be in swapdeath
- 15:25 logmsgbot: andrew ran sync-common-all
- 15:24 Andrew: sync-common-all caused memory spike on apaches again, site seems to still be up though
- 15:17 Andrew: Updating LiquidThreads to trunk state, using sync-common-all
- 12:58 RoanKattouw: Tim ran rebuildTemplates.php on hume for all languages
- 12:52 Andrew: Fixed morebots, was choking because the Server Admin Log was archived.
- 12:50 Andrew: Moved Morebots init script into a line in rc.local. Morebots seems to have been down due to Freenode's DDoS problem, maybe it isn't exiting properly when disconnected from the server
- 12:46 Tim: ran rebuildTemplates.php on hume for all languages
Archives
- Server admin log/Archive 1 (2004 Jun - 2004 Sep)
- Server admin log/Archive 2 (2004 Oct - 2004 Nov)
- Server admin log/Archive 3 (2004 Dec - 2005 Mar)
- Server admin log/Archive 4 (2005 Apr - 2005 Jul)
- Server admin log/Archive 5 (2005 Aug - 2005 Oct)
- Server admin log/Archive 6 (2005 Nov - 2006 Feb)
- Server admin log/Archive 7 (2006 Mar - 2006 Jun)
- Server admin log/Archive 8 (2006 Jul - 2006 Sep)
- Server admin log/Archive 9 (2006 Oct - 2007 Jan)
- Server admin log/Archive 10 (2007 Feb - 2007 Jun)
- Server admin log/Archive 11 (2007 Jul - 2007 Dec)
- Server admin log/Archive 12 (2008 Jan - 2008 Jul)
- Server admin log/2008-08
- Server admin log/2008-09
- Server admin log/Archive 13 (2008 Oct - 2009 Jun)
- Server admin log/Archive 14 (2009 Jun - 2009 Dec)
