User contributions
(Latest | Earliest) View (newer 500 | older 500) (20 | 50 | 100 | 250 | 500)
- 01:44, 30 January 2013 (diff | hist) . . (-87) . . Search (→Index Sharding) (top)
- 01:41, 30 January 2013 (diff | hist) . . (-405,444) . . SearchShards (removed outdated data) (top)
- 22:46, 29 January 2013 (diff | hist) . . (+18) . . Server admin log (date header)
- 03:11, 21 January 2013 (diff | hist) . . (+4) . . Puppet (--parseonly has been removed. Please use 'puppet parser validate <manifest>')
- 04:00, 3 January 2013 (diff | hist) . . (-115) . . Eqiad Migration Planning (→Actually Failing Over: clarify DNS and Swift situation)
- 03:51, 3 January 2013 (diff | hist) . . (-52) . . Eqiad Migration Planning (→Actually Failing Over: no swift replication to reverse)
- 00:22, 31 December 2012 (diff | hist) . . (+28) . . N Reset password (Redirected page to Password reset) (top)
- 00:22, 31 December 2012 (diff | hist) . . (+1,689) . . N Password reset (how to reset a password in MW)
- 21:57, 27 December 2012 (diff | hist) . . (+868) . . PoolCounter (how to test) (top)
- 22:43, 19 December 2012 (diff | hist) . . (+60) . . Lists.wikimedia.org (→Alter arbcom-l archive access list) (top)
- 22:32, 19 December 2012 (diff | hist) . . (+1,221) . . Lists.wikimedia.org (arbcom-l access controls)
- 10:14, 4 December 2012 (diff | hist) . . (0) . . Server history (updated links due to mailman breakage) (top)
- 10:12, 28 November 2012 (diff | hist) . . (+135) . . Dell/PowerEdge R410 & R610 (→External Links: useful doc) (top)
- 23:39, 15 October 2012 (diff | hist) . . (-9) . . Schema changes (→To be scheduled: job queue improvements merged)
- 03:52, 26 September 2012 (diff | hist) . . (+133) . . How to deploy code (→Case 1c: extension update: alternative)
- 05:56, 25 September 2012 (diff | hist) . . (+561) . . Morebots (updated documentation, apparently nobody actually bothers to read the code)
- 04:34, 7 September 2012 (diff | hist) . . (+14) . . Sun Fire X4240 (updated link) (top)
- 04:32, 7 September 2012 (diff | hist) . . (+9) . . Sun Fire X4500 and X4540 (updated link) (top)
- 13:21, 17 August 2012 (diff | hist) . . (+806) . . N User:Tim/Switching log server (Created page with "To change the MW log server: * In CommonSettings.php, edit $wmfUdp2logDest. Sync that out. * Touch InitialiseSettings.php and sync that out since $wmfUdp2logDest is implemented ...") (top)
- 01:08, 10 August 2012 (diff | hist) . . (0) . . m Server admin log/Archive 21 (→August 10)
- 12:52, 31 January 2012 (diff | hist) . . (+361) . . Gerrit (doc link and some random factoids)
- 06:21, 31 January 2012 (diff | hist) . . (+20) . . N Gerrit.wikimedia.org (Redirected page to Gerrit) (top)
- 06:21, 31 January 2012 (diff | hist) . . (+745) . . Gerrit (some docs)
- 01:13, 27 October 2011 (diff | hist) . . (+64) . . Deployments (→Week of October 31st: VipsScaler)
- 22:04, 16 October 2011 (diff | hist) . . (+1,301) . . Media server/FileBackend (etherpad paste) (top)
- 00:53, 27 September 2011 (diff | hist) . . (+1,611) . . N MessageCache (some docs about MessageCache) (top)
- 00:37, 27 September 2011 (diff | hist) . . (+26) . . Nm Message cache (Redirected page to MessageCache) (top)
- 04:17, 20 September 2011 (diff | hist) . . (+61) . . Server admin log/Archive 20 (→September 20: retrolog for LDAP change)
- 14:07, 31 August 2011 (diff | hist) . . (+109) . . External storage/Update 2011-08 (→Background and context: text storage row types)
- 23:55, 5 July 2011 (diff | hist) . . (+51) . . Deployments (→Week of July 4: dboc)
- 23:16, 24 June 2011 (diff | hist) . . (-31) . . Incident response (not in order)
- 23:00, 24 June 2011 (diff | hist) . . (-5) . . m Incident response
- 22:59, 24 June 2011 (diff | hist) . . (+2,770) . . Incident response (more about apache CPU usage)
- 07:45, 24 June 2011 (diff | hist) . . (+3) . . m Incident response (→MySQL overload)
- 03:07, 24 June 2011 (diff | hist) . . (+2,976) . . Incident response (fixing, communication)
- 02:35, 24 June 2011 (diff | hist) . . (+2,384) . . Incident response (more about apache, some stuff about mysql)
- 02:09, 24 June 2011 (diff | hist) . . (+6,862) . . N Incident response (some initial text on incident response, work so far)
- 06:38, 23 June 2011 (diff | hist) . . (+15) . . Deployments (scheduled subpagelist3)
- 05:21, 23 June 2011 (diff | hist) . . (+41) . . Deployments (→Week of June 27: MySQL parser cache)
- 01:51, 15 June 2011 (diff | hist) . . (+1,749) . . SecurePoll (duplicate removal)
- 01:30, 15 June 2011 (diff | hist) . . (+3,327) . . SecurePoll (encryption and tallying)
- 06:54, 25 May 2011 (diff | hist) . . (+791) . . Server admin log/Archive 20 (→May 24: some retrologging for the downtime around 1300)
- 04:18, 29 April 2011 (diff | hist) . . (+1,113) . . N Delete a wiki (how to delete a wiki)
- 03:50, 29 April 2011 (diff | hist) . . (+29) . . Add a wiki (see also)
- 23:40, 26 April 2011 (diff | hist) . . (-1) . . Add a server (update)
- 23:49, 18 April 2011 (diff | hist) . . (-101) . . Close a wiki (quicker way to close a wiki, no need to run sync-common-all)
- 00:59, 31 March 2011 (diff | hist) . . (+42) . . Ehcache (→Configuration: object size/count)
- 04:44, 30 March 2011 (diff | hist) . . (+922) . . Ehcache (configuration and monitoring)
- 04:28, 30 March 2011 (diff | hist) . . (+845) . . Ehcache (tomcat installation)
- 06:07, 18 March 2011 (diff | hist) . . (+30) . . PoolCounter (→$wgPoolCounterConf: worker count expense)
- 06:01, 18 March 2011 (diff | hist) . . (-1) . . m PoolCounter (→$wgPoolCounterConf)
- 05:40, 18 March 2011 (diff | hist) . . (+25) . . N Poolcounter (Redirected page to PoolCounter) (top)
- 05:39, 18 March 2011 (diff | hist) . . (+25) . . N Pool counter (Redirected page to PoolCounter) (top)
- 05:39, 18 March 2011 (diff | hist) . . (+3,259) . . N PoolCounter (PoolCounter docs)
- 05:48, 20 January 2011 (diff | hist) . . (+273) . . 1.17 deployment plan (→Deployment Steps/Sequence: collation) (top)
- 05:36, 20 January 2011 (diff | hist) . . (+287) . . 1.17 deployment plan (→Risks & Mitigations: collation update)
- 03:32, 22 November 2010 (diff | hist) . . (+292) . . LDAP (→Change directory manager password: another method)
- 04:10, 17 November 2010 (diff | hist) . . (-36) . . Mobile.wikipedia.org (update)
- 17:47, 24 October 2010 (diff | hist) . . (+442) . . N Heterogeneous deployment/old (notes from etherpad)
- 17:45, 24 October 2010 (diff | hist) . . (+831) . . N Wmsync (wmsync notes)
- 15:51, 24 October 2010 (diff | hist) . . (+3,455) . . SecurePoll (how to make an arbcom election)
- 14:31, 21 September 2010 (diff | hist) . . (+1) . . m Kernel upgrades 2010-09
- 05:10, 21 September 2010 (diff | hist) . . (+159) . . Kernel upgrades 2010-09 (updated)
- 02:56, 21 September 2010 (diff | hist) . . (+6) . . m Server admin log/Archive 20 (→September 21)
- 14:35, 20 September 2010 (diff | hist) . . (+122) . . Kernel upgrades 2010-09
- 14:32, 20 September 2010 (diff | hist) . . (+378) . . N Kernel upgrades 2010-09 (kernel upgrade status)
- 15:09, 18 August 2010 (diff | hist) . . (0) . . m Server admin log/Archive 20
- 06:24, 6 August 2010 (diff | hist) . . (-2) . . ExtensionDistributor (s/zwinger/fenari)
- 07:25, 12 July 2010 (diff | hist) . . (+39) . . m Nagios (→On each client)
- 07:19, 12 July 2010 (diff | hist) . . (+264) . . Nagios (→On each client: more steps)
- 05:24, 12 July 2010 (diff | hist) . . (+156) . . m Nagios (→On each client)
- 05:12, 12 July 2010 (diff | hist) . . (+135) . . Nagios (→On each client: documented solaris NRPE installation)
- 03:39, 12 July 2010 (diff | hist) . . (+147) . . LVS (→LVS director list: +m)
- 01:21, 6 July 2010 (diff | hist) . . (0) . . m Codereview-proxy.wikimedia.org
- 00:14, 6 July 2010 (diff | hist) . . (-30) . . Codereview-proxy.wikimedia.org (updated)
- 04:45, 5 July 2010 (diff | hist) . . (+86) . . Nagios (updated bot info)
- 01:08, 5 July 2010 (diff | hist) . . (+5) . . UDP based profiling (update)
- 02:35, 8 June 2010 (diff | hist) . . (+20) . . Nm Wikimedia-mobile (Redirected page to Mobile) (top)
- 03:30, 7 June 2010 (diff | hist) . . (+40) . . N IDRAC6 (Redirected page to Dell PowerEdge R410 & R610)
- 03:30, 7 June 2010 (diff | hist) . . (+40) . . N DRAC6 (Redirected page to Dell PowerEdge R410 & R610)
- 01:54, 7 June 2010 (diff | hist) . . (+19) . . m Wikibugs (→How to set it up)
- 01:53, 7 June 2010 (diff | hist) . . (+1) . . m Wikibugs
- 01:53, 7 June 2010 (diff | hist) . . (+516) . . Wikibugs (updated)
- 01:27, 7 June 2010 (diff | hist) . . (+8) . . Wikibugs
- 01:01, 6 June 2010 (diff | hist) . . (+1) . . m Squid log format
- 08:26, 3 June 2010 (diff | hist) . . (-1) . . m Server admin log/Archive 20 (→June 3)
- 07:13, 3 June 2010 (diff | hist) . . (+152) . . LVS (→LVS director list: updated)
- 03:08, 25 May 2010 (diff | hist) . . (-57) . . Mobile (fixed circular reference)
- 03:08, 25 May 2010 (diff | hist) . . (+46) . . Mobile.wikipedia.org (fixed circular reference)
- 01:13, 25 May 2010 (diff | hist) . . (+8) . . Server admin log/Archive 20
- 04:45, 24 February 2010 (diff | hist) . . (+65) . . m Text storage data (→Bugs)
- 06:04, 23 February 2010 (diff | hist) . . (+634) . . Text storage data (bugs)
- 06:27, 22 February 2010 (diff | hist) . . (+3) . . Puppet (updated directory name)
- 05:42, 22 February 2010 (diff | hist) . . (+110) . . MySQL (openbsd vs traditional)
- 22:18, 18 February 2010 (diff | hist) . . (-7) . . Text storage data (breakdown of object,utf-8)
- 07:34, 18 February 2010 (diff | hist) . . (+72) . . Text storage data (utf-8,gzip,external/simple -- non-recompressed)
- 06:09, 18 February 2010 (diff | hist) . . (-3) . . m Text storage data
- 06:09, 18 February 2010 (diff | hist) . . (+151) . . Text storage data (→Explanation: more on object,utf-8)
- 06:03, 18 February 2010 (diff | hist) . . (-84) . . Text storage data (utf8 seems to be from RCT rather than CO)
- 05:56, 18 February 2010 (diff | hist) . . (-13) . . Text storage data (CO wouldn't create uncompressed ES objects)
- 05:54, 18 February 2010 (diff | hist) . . (+1,792) . . Text storage data (analysis)
- 05:19, 18 February 2010 (diff | hist) . . (+866) . . N Text storage data (Created page with 'Text row types as of 2010-02-18 <pre> Count Type ------------------------------------------------ 9 0,external/simple pointer 435 0/[none] 1482941 [n...')
- 11:40, 11 February 2010 (diff | hist) . . (+41) . . External storage
- 04:18, 30 January 2010 (diff | hist) . . (+39) . . Server admin log/Archive 20 (restarting mysqld on db12)
- 04:01, 30 January 2010 (diff | hist) . . (-196) . . m Server admin log/Archive 20
- 04:00, 30 January 2010 (diff | hist) . . (+289) . . Server admin log/Archive 20 (analysis)
- 04:00, 30 January 2010 (diff | hist) . . (+975) . . Server admin log/Archive 20 (retrolog)
- 06:46, 5 January 2010 (diff | hist) . . (+253) . . N Syslog (documented) (top)
- 23:00, 4 January 2010 (diff | hist) . . (-10) . . Squid logging (s/henbane/locke)
- 00:33, 30 December 2009 (diff | hist) . . (+95) . . Civicrm.wikimedia.org (source)
- 02:45, 10 December 2009 (diff | hist) . . (+161) . . Server admin log/Archive 20 (correction)
- 01:51, 23 November 2009 (diff | hist) . . (+156) . . Logmsgbot (update)
- 01:45, 23 November 2009 (diff | hist) . . (+23) . . Dsh (fenari update) (top)
- 01:34, 17 November 2009 (diff | hist) . . (+370) . . Robot policy (user agent) (top)
- 01:27, 17 November 2009 (diff | hist) . . (+3,915) . . N Robot policy (robot policy)
- 22:27, 5 November 2009 (diff | hist) . . (-1) . . APT repository (s/khaldun/brewster)
- 18:22, 26 October 2009 (diff | hist) . . (+501) . . MySQL (→High write rate: more sources of write-driven lag)
- 21:43, 20 October 2009 (diff | hist) . . (-7) . . m MySQL
- 21:40, 20 October 2009 (diff | hist) . . (+3,247) . . MySQL (about replication lag)
- 19:59, 20 October 2009 (diff | hist) . . (+1,210) . . MySQL (another section)
- 17:32, 20 October 2009 (diff | hist) . . (+28) . . N Apt.wikimedia.org (APT) (top)
- 17:31, 20 October 2009 (diff | hist) . . (+284) . . MySQL (data dir copy)
- 16:15, 20 October 2009 (diff | hist) . . (+1,782) . . MySQL (some more docs)
- 21:09, 19 October 2009 (diff | hist) . . (+112) . . MySQL (more fixme)
- 06:26, 24 September 2009 (diff | hist) . . (+17) . . m Server admin log/Archive 20
- 07:25, 11 August 2009 (diff | hist) . . (+58) . . Server admin log/Archive 20 (→August 11: log for domas)
- 06:23, 11 August 2009 (diff | hist) . . (+82) . . Server admin log/Archive 20 (→August 11)
- 04:26, 10 August 2009 (diff | hist) . . (+85) . . Server history (another post)
- 02:36, 7 August 2009 (diff | hist) . . (+213) . . Server history (more entries)
- 15:34, 28 July 2009 (diff | hist) . . (+914) . . N SecurePoll (updating translations)
- 15:20, 22 July 2009 (diff | hist) . . (+3) . . m IRCD (→Firewall)
- 15:16, 22 July 2009 (diff | hist) . . (+443) . . IRCD (firewall configuration)
- 16:10, 15 July 2009 (diff | hist) . . (0) . . m Uncommitted files in live working copy
- 15:52, 15 July 2009 (diff | hist) . . (+1,408) . . Uncommitted files in live working copy (more comments, sorted maintenance dir)
- 15:08, 15 July 2009 (diff | hist) . . (+291) . . Uncommitted files in live working copy (→Main dir: some comments)
- 05:37, 14 July 2009 (diff | hist) . . (-11) . . m Linode (fixed wikitext)
- 15:23, 13 July 2009 (diff | hist) . . (+60) . . m Monitoring procedure
- 15:20, 13 July 2009 (diff | hist) . . (+962) . . N Monitoring procedure (proposed monitoring procedure)
- 13:27, 17 June 2009 (diff | hist) . . (-249) . . Schema changes (removed servers section)
- 14:22, 10 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers)
- 15:50, 9 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers)
- 09:19, 9 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers)
- 08:58, 9 June 2009 (diff | hist) . . (+6) . . Schema changes (→Servers: marked masters)
- 08:36, 9 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers)
- 06:52, 9 June 2009 (diff | hist) . . (+25) . . Schema changes (fpc_level was already done)
- 06:09, 9 June 2009 (diff | hist) . . (0) . . Schema changes
- 06:03, 9 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers)
- 06:03, 9 June 2009 (diff | hist) . . (+118) . . Schema changes
- 05:03, 9 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers: another 6)
- 16:21, 8 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers)
- 10:40, 8 June 2009 (diff | hist) . . (0) . . Schema changes
- 10:09, 8 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers)
- 08:49, 8 June 2009 (diff | hist) . . (0) . . Schema changes (→Servers: db15 done)
- 07:31, 8 June 2009 (diff | hist) . . (0) . . Schema changes
- 07:21, 8 June 2009 (diff | hist) . . (+243) . . Schema changes (server scoresheet)
- 07:10, 8 June 2009 (diff | hist) . . (+31) . . Schema changes (patch-categorylinksindex.sql)
- 04:36, 8 June 2009 (diff | hist) . . (-81) . . Schema changes (CA patch reverted)
- 02:42, 7 June 2009 (diff | hist) . . (+58) . . Schema changes (hide_patrol)
- 16:13, 6 June 2009 (diff | hist) . . (-53) . . Schema changes (reformatted)
- 16:05, 6 June 2009 (diff | hist) . . (+342) . . Schema changes (more schema changes)
- 07:25, 26 May 2009 (diff | hist) . . (-357) . . Server admin log/Archive 20 (fixed revision list)
- 14:08, 21 May 2009 (diff | hist) . . (-226) . . Switch master (updated documentation for new switch method)
- 02:44, 20 May 2009 (diff | hist) . . (+3) . . m Server admin log/Archive 20 (→May 20: typo)
- 07:30, 19 May 2009 (diff | hist) . . (+23) . . N Batch jobs (Batch jobs moved to Cron jobs) (top)
- 07:30, 19 May 2009 (diff | hist) . . (0) . . m Cron jobs (Batch jobs moved to Cron jobs)
- 01:46, 29 April 2009 (diff | hist) . . (+281) . . Add a server (--no-remove)
- 01:35, 29 April 2009 (diff | hist) . . (+7) . . m Add a server (→List of things that will break if you try to install MediaWiki without following this procedure)
- 01:34, 29 April 2009 (diff | hist) . . (+978) . . Add a server (List of things that will break if you try to install MediaWiki without following this procedure)
- 01:22, 29 April 2009 (diff | hist) . . (+1,185) . . N Add a server (how to add a server (ubuntu version))
- 01:20, 28 April 2009 (diff | hist) . . (+178) . . Service IPs (new external storage VIPs) (top)
- 05:48, 20 April 2009 (diff | hist) . . (+141) . . IP addresses (→Internal global assignments: taking a /24 for ES)
- 00:58, 25 March 2009 (diff | hist) . . (-5) . . Sun Fire X4240 (updated doc link)
- 02:43, 20 March 2009 (diff | hist) . . (+50) . . Locke (renamed to locke)
- 02:42, 20 March 2009 (diff | hist) . . (0) . . m Locke (Db6 moved to Locke: server renamed)
- 02:42, 20 March 2009 (diff | hist) . . (+19) . . N Db6 (Db6 moved to Locke: server renamed) (top)
- 00:40, 20 March 2009 (diff | hist) . . (+42) . . Server admin log/Archive 20 (fixed completely wrong log entry from platonides)
- 23:34, 19 March 2009 (diff | hist) . . (+1) . . m Automated installation (→System Install)
- 22:00, 19 March 2009 (diff | hist) . . (+29) . . m Server admin log/Archive 20
- 16:34, 11 March 2009 (diff | hist) . . (+10) . . m Schema changes (→rc_user_text Index)
- 00:42, 11 March 2009 (diff | hist) . . (-302) . . Schema changes (removed some boilerplate junk and a patch I just rejected)
- 23:45, 10 March 2009 (diff | hist) . . (0) . . m Schema changes (→rc_interwiki & rc_fragment)
- 09:55, 1 March 2009 (diff | hist) . . (+29) . . Server admin log/Archive 20 (→March 1: edit)
- 08:11, 26 February 2009 (diff | hist) . . (+9) . . Server admin log/Archive 20 (→February 26: server name)
- 10:09, 23 February 2009 (diff | hist) . . (-338) . . Job queue (new package)
- 04:36, 23 February 2009 (diff | hist) . . (+9) . . Cron jobs (→CentralNotice (donation banner) template update: update)
- 08:16, 19 February 2009 (diff | hist) . . (+300) . . Server admin log/Archive 20 (retrolog)
- 07:56, 19 February 2009 (diff | hist) . . (-1) . . m Job queue
- 07:54, 19 February 2009 (diff | hist) . . (+166) . . Job queue (how to stop)
- 05:56, 18 February 2009 (diff | hist) . . (+169) . . Bugzilla.wikimedia.org (attachment domain)
- 03:22, 17 February 2009 (diff | hist) . . (+159) . . Server admin log/Archive 20 (→February 17: MW_DIFF_VERSION)
- 03:01, 17 February 2009 (diff | hist) . . (+72) . . Server admin log/Archive 20 (→February 17: removed live hacks from extension/Cite, updated to r47350.)
- 05:05, 16 February 2009 (diff | hist) . . (+214) . . Job queue (script contents)
- 04:39, 11 February 2009 (diff | hist) . . (-42) . . Bugzilla.wikimedia.org (updates)
- 01:32, 5 February 2009 (diff | hist) . . (+4) . . Cron jobs (updates)
- 00:27, 5 February 2009 (diff | hist) . . (+24) . . N Cron (Redirecting to Batch jobs)
- 06:51, 2 February 2009 (diff | hist) . . (+231) . . OTRS (mod_perl notes)
- 06:48, 2 February 2009 (diff | hist) . . (+72) . . OTRS (more updates)
- 00:58, 2 February 2009 (diff | hist) . . (+28) . . N ILOM (Redirecting to Sun Fire X4240)
- 02:42, 1 February 2009 (diff | hist) . . (+603) . . Server admin log/Archive 20 (work so far)
- 01:01, 1 February 2009 (diff | hist) . . (+199) . . Server admin log/Archive 20 (revoked write access from the otrs mysql user, to prevent any further damage. Making a copy of the binlogs. The plan is to do forensics first and then recovery second.)
- 14:22, 30 January 2009 (diff | hist) . . (+167) . . Server admin log/Archive 20 (→January 30: done for now)
- 06:58, 30 January 2009 (diff | hist) . . (+143) . . Server admin log/Archive 20 (RCT should be finished now, no more connections are expected on cluster13 or 14. Current connection counts: 123943575, 295618929.)
- 06:48, 30 January 2009 (diff | hist) . . (+39) . . Server admin log/Archive 20 (20:14 recompressTracked.php finished)
- 04:23, 30 January 2009 (diff | hist) . . (+1,185) . . OTRS (configuration rant)
- 03:11, 29 January 2009 (diff | hist) . . (0) . . m OTRS (→To upgrade)
- 03:08, 29 January 2009 (diff | hist) . . (-24) . . OTRS (→To upgrade)
- 06:36, 28 January 2009 (diff | hist) . . (-60) . . OTRS (→To upgrade)
- 06:30, 28 January 2009 (diff | hist) . . (-131) . . OTRS (some updates)
- 14:30, 27 January 2009 (diff | hist) . . (+88) . . Server admin log/Archive 20 (→January 27: retrolog for domas)
- 08:21, 27 January 2009 (diff | hist) . . (+70) . . Server admin log/Archive 20 (→January 27: depooled db3 and db4)
- 07:11, 27 January 2009 (diff | hist) . . (+175) . . Server admin log/Archive 20 (→January 27: There was a bug in recompressTracked.php which caused the last batch of orphans for any given wiki to be skipped. Re-running recompressTracked.php to repair it.)
- 06:03, 27 January 2009 (diff | hist) . . (+208) . . Server admin log/Archive 20 (retrolog (bot broken))
- 05:56, 27 January 2009 (diff | hist) . . (-270) . . Job queue (ubuntu job runners)
- 05:30, 27 January 2009 (diff | hist) . . (+41) . . Server admin log/Archive 20 (→January 27)
- 04:29, 16 January 2009 (diff | hist) . . (+35) . . N CiviCRM (Redirecting to Civicrm.wikimedia.org) (top)
- 04:29, 16 January 2009 (diff | hist) . . (+35) . . N Civicrm (Redirecting to Civicrm.wikimedia.org) (top)
- 13:18, 7 January 2009 (diff | hist) . . (+129) . . Server admin log/Archive 20 (→January 7: recompressTracked.php complete)
- 11:16, 6 January 2009 (diff | hist) . . (+42) . . Server admin log/Archive 20 (→January 6: cleaned up binlogs on srv105)
- 22:07, 2 January 2009 (diff | hist) . . (+100) . . Server admin log/Archive 20 (→January 2: killed recompressTracked for now, not waiting for slaves properly. db3 and db4 lagged.)
- 08:50, 30 December 2008 (diff | hist) . . (-8) . . m Server admin log/Archive 20
- 08:52, 29 December 2008 (diff | hist) . . (+45) . . Server admin log/Archive 20 (→December 29: deleted some binlogs on srv108.)
- 05:32, 29 December 2008 (diff | hist) . . (+1) . . Server admin log/Archive 20 (→December 29: done)
- 04:53, 29 December 2008 (diff | hist) . . (+155) . . Server admin log/Archive 20 (→December 29: external storage)
- 03:42, 29 December 2008 (diff | hist) . . (+50) . . Server admin log/Archive 20 (→December 29: running orphanStats.php on all wikis)
- 03:28, 29 December 2008 (diff | hist) . . (+63) . . Server admin log/Archive 20 (→December 29: restarted apache on srv33, srv146, srv169, srv172)
- 03:17, 29 December 2008 (diff | hist) . . (+61) . . Server admin log/Archive 20 (cleaned up binlogs on srv105)
- 00:53, 26 December 2008 (diff | hist) . . (+46) . . Server admin log/Archive 20 (→December 26: started mysql on db19)
- 00:51, 26 December 2008 (diff | hist) . . (+201) . . Server admin log/Archive 20 (retrolog)
- 02:42, 23 December 2008 (diff | hist) . . (+103) . . Server admin log/Archive 20 (→December 23: disk space stuff)
- 03:29, 21 December 2008 (diff | hist) . . (+101) . . Server admin log/Archive 20 (depooled db4 temporarily to speed up a long running trackBlobs query)
- 08:55, 19 December 2008 (diff | hist) . . (+157) . . Server admin log/Archive 20 (→December 19: reduced run rate for rebuildTemplates.php)
- 06:48, 9 December 2008 (diff | hist) . . (+63) . . Server admin log/Archive 20 (→December 9: srv146 did not have /mnt/upload5 mounted. Fixed.)
- 13:20, 8 December 2008 (diff | hist) . . (+120) . . Server admin log/Archive 20 (→December 8: clarify)
- 13:19, 8 December 2008 (diff | hist) . . (+117) . . Server admin log/Archive 20 (IWF proxies)
- 23:59, 2 December 2008 (diff | hist) . . (+73) . . Server admin log/Archive 20 (→December 2: added CNAME poke.wikimedia.org for SMS notification project)
- 11:53, 2 December 2008 (diff | hist) . . (+43) . . Server admin log/Archive 20 (→December 2)
- 11:41, 2 December 2008 (diff | hist) . . (+172) . . Server admin log/Archive 20 (→December 2: backend spike)
- 05:51, 13 November 2008 (diff | hist) . . (0) . . m Server admin log/Archive 20 (→November 13)
- 05:47, 13 November 2008 (diff | hist) . . (+80) . . Server admin log/Archive 20 (more info)
- 01:47, 13 November 2008 (diff | hist) . . (+141) . . Server admin log/Archive 20 (→November 13)
- 03:27, 12 November 2008 (diff | hist) . . (0) . . Server admin log/Archive 20
- 03:27, 12 November 2008 (diff | hist) . . (0) . . m Server admin log/Archive 20 (fixed dates)
- 01:21, 12 November 2008 (diff | hist) . . (+153) . . Server admin log/Archive 20 (→November 13: site notice error)
- 01:16, 12 November 2008 (diff | hist) . . (-6) . . Cron jobs (/usr/bin)
- 01:10, 12 November 2008 (diff | hist) . . (+149) . . Server admin log/Archive 20 (→November 13: fixed syslogs)
- 01:00, 12 November 2008 (diff | hist) . . (+262) . . Server admin log/Archive 20 (srv125 errors)
- 00:29, 12 November 2008 (diff | hist) . . (+12) . . Server admin log/Archive 20 (→November 13: bug)
- 00:28, 12 November 2008 (diff | hist) . . (+101) . . Server admin log/Archive 20 (→November 13: updated ExtensionDistributor configuration to account for amane -> ms1 storage move.)
- 00:27, 12 November 2008 (diff | hist) . . (-2) . . ExtensionDistributor (updated)
- 00:15, 12 November 2008 (diff | hist) . . (+127) . . Server admin log/Archive 20 (→November 12: fixed lang obj cache)
- 23:50, 11 November 2008 (diff | hist) . . (+95) . . Server admin log/Archive 20 (→November 12: restored FlaggedRevs stats job as per Batch jobs, removal was not documented.)
- 23:49, 11 November 2008 (diff | hist) . . (+1) . . Cron jobs (→FlaggedRevs stats update: fix minute part)
- 23:46, 11 November 2008 (diff | hist) . . (+361) . . Cron jobs (CentralNotice)
- 23:39, 11 November 2008 (diff | hist) . . (+178) . . Server admin log/Archive 20 (→November 12: rebuildTemplates.php happ)
- 23:30, 11 November 2008 (diff | hist) . . (+104) . . Server admin log/Archive 20 (→November 11: updated CentralNotice templates on hume (which has enough memory to do it, unlike zwinger))
- 22:13, 11 November 2008 (diff | hist) . . (+118) . . Server admin log/Archive 20 (→November 11: db1)
- 12:07, 11 November 2008 (diff | hist) . . (+228) . . Server admin log/Archive 20 (→November 11: theory)
- 01:08, 7 November 2008 (diff | hist) . . (+324) . . Server admin log/Archive 20 (→November 7: db10 stuff)
- 21:55, 6 November 2008 (diff | hist) . . (+76) . . Squid log format (time format change)
- 02:36, 6 November 2008 (diff | hist) . . (+94) . . Server admin log/Archive 20 (→November 6: two entries for mark)
- 02:18, 6 November 2008 (diff | hist) . . (+250) . . Server admin log/Archive 20 (→November 6: retrolog)
- 16:39, 4 November 2008 (diff | hist) . . (+39) . . Server admin log/Archive 20 (→November 4: max_rows)
- 15:42, 4 November 2008 (diff | hist) . . (+62) . . Server admin log/Archive 20 (→November 4: master switch)
- 01:35, 4 November 2008 (diff | hist) . . (+11) . . Server admin log/Archive 20 (→November 4: and srv106)
- 01:34, 4 November 2008 (diff | hist) . . (+39) . . Server admin log/Archive 20 (→November 4: fixing max_rows on srv105)
- 01:29, 4 November 2008 (diff | hist) . . (+79) . . Server admin log/Archive 20 (→November 3: cluster17 full)
- 23:30, 3 November 2008 (diff | hist) . . (+231) . . Server admin log/Archive 20 (→November 3: installed xdiff and gmp on hume)
- 21:59, 3 November 2008 (diff | hist) . . (0) . . m Server admin log/Archive 20 (→November 3)
- 21:43, 3 November 2008 (diff | hist) . . (+55) . . Server admin log/Archive 20 (→November 3: initialising srv159/171/182 as cluster20.)
- 21:24, 3 November 2008 (diff | hist) . . (+137) . . Server admin log/Archive 20 (→November 3: srv159 needs to be an ext store, and so will be moved from the disk-intensive image scaler role back to an ordinary apache.)
- 17:14, 2 November 2008 (diff | hist) . . (+84) . . Server admin log/Archive 20 (→November 2: cleaning up /tmp)
- 16:59, 2 November 2008 (diff | hist) . . (+105) . . Server admin log/Archive 20 (set ImageMagick temporary directory)
- 11:26, 1 November 2008 (diff | hist) . . (+201) . . Server admin log/Archive 20 (→November 1: enabled subpages in the main namespace by default for all Wikisource wikis)
- 07:58, 1 November 2008 (diff | hist) . . (+65) . . Server admin log/Archive 20 (disabled ParserDiffTest)
- 23:13, 30 October 2008 (diff | hist) . . (+53) . . Server admin log/Archive 20 (→October 30: fixed disk space on srv159, db1, srv103)
- 04:32, 26 October 2008 (diff | hist) . . (+57) . . Server admin log/Archive 20 (→October 26: fixed the "service_ips" hostgroup in nagios)
- 03:04, 26 October 2008 (diff | hist) . . (+37) . . Server admin log/Archive 20 (→October 26: hardware reboot of db18)
- 02:55, 26 October 2008 (diff | hist) . . (+326) . . Server admin log/Archive 20 (→October 25: mysqld on db18 apparently hit a kernel bug)
- 08:18, 25 October 2008 (diff | hist) . . (+87) . . Server admin log/Archive 20 (→October 25: svn up to 42536 for API overload fix. Re-enabling disabled query modules.)
- 05:56, 25 October 2008 (diff | hist) . . (+75) . . Server admin log/Archive 20 (→October 25: svn up/scap to 42531 (for properly tested Interwiki.php fix).)
- 05:22, 25 October 2008 (diff | hist) . . (+371) . . Server admin log/Archive 20 (→October 25: retrolog)
- 04:34, 25 October 2008 (diff | hist) . . (+63) . . Server admin log/Archive 20 (→October 25)
- 04:22, 25 October 2008 (diff | hist) . . (+105) . . Server admin log/Archive 20 (→October 25: disabled drafts)
- 04:13, 25 October 2008 (diff | hist) . . (+121) . . Server admin log/Archive 20 (→October 25: synced php-1.5 to srv35 and ran "make -B" in the serialized directory. Seems to have fixed test. Will scap.)
- 09:58, 24 October 2008 (diff | hist) . . (+103) . . Server admin log/Archive 20 (→October 24: removed decommissioned servers from monitoring)
- 07:51, 24 October 2008 (diff | hist) . . (+129) . . Server admin log/Archive 20 (→October 24: two entries)
- 05:39, 24 October 2008 (diff | hist) . . (+75) . . Server admin log/Archive 20 (→October 24)
- 15:39, 23 October 2008 (diff | hist) . . (+153) . . Server admin log/Archive 20 (→October 23: /a/tmp)
- 15:25, 23 October 2008 (diff | hist) . . (+60) . . Server admin log/Archive 20 (→October 23: removed temporary files on image scalers again)
- 11:04, 23 October 2008 (diff | hist) . . (+113) . . Server admin log/Archive 20 (→October 23: Added disk space monitoring for image scalers. Also added apache monitoring which was also missing.)
- 10:54, 23 October 2008 (diff | hist) . . (+111) . . Server admin log/Archive 20 (→October 23: freed up disk space on image scalers, magick-* temporary files were filling their root partitions)
- 10:51, 23 October 2008 (diff | hist) . . (+92) . . Server admin log/Archive 20 (→October 23: re-added cluster19 to the default write list. Not sure who took it out or why.)
- 10:33, 23 October 2008 (diff | hist) . . (+63) . . Server admin log/Archive 20 (→October 23: freed up some space on srv103 (was down to 500MB))
- 10:29, 23 October 2008 (diff | hist) . . (+47) . . Server admin log/Archive 20 (→October 23: fixed monitoring for MegaRAID SAS)
- 07:16, 23 October 2008 (diff | hist) . . (+216) . . Server admin log/Archive 20 (→October 23: Set up monitoring of RAID status for all Ubuntu DB servers using the wikimedia-raid-utils package)
- 05:09, 23 October 2008 (diff | hist) . . (+64) . . Server admin log/Archive 20 (→October 22: running CodeReview svnImport.php)
- 13:50, 21 October 2008 (diff | hist) . . (+96) . . Server admin log/Archive 20 (→October 21: Did a demonstration hack of nagios from CSRF to arbitrary shell. Disabled cmd.cgi.)
- 04:15, 21 October 2008 (diff | hist) . . (+117) . . Server admin log/Archive 20 (→October 20: Brought srv43-47 up as image scalers with mem limit 6 x 200MB = 1200MB (2GB physical))
- 05:11, 20 October 2008 (diff | hist) . . (+86) . . Server admin log/Archive 20 (→October 20: increased concurrency on srv159 to 15, for mem limit 15 x 200MB = 3000MB)
- 02:38, 20 October 2008 (diff | hist) . . (+48) . . Server admin log/Archive 20 (→October 20: installed NRPE on khaldun and db20)
- 02:22, 20 October 2008 (diff | hist) . . (+127) . . Server admin log/Archive 20 (→October 20: moved disk space checks on the ext stores from the "apaches" service group to the relevant ext store service group)
- 02:21, 20 October 2008 (diff | hist) . . (+231) . . Nagios (NRPE installation)
- 01:54, 20 October 2008 (diff | hist) . . (+64) . . Server admin log/Archive 20 (NRPE)
- 01:46, 20 October 2008 (diff | hist) . . (+63) . . m Server admin log/Archive 20 (→October 20)
- 01:32, 20 October 2008 (diff | hist) . . (+199) . . Server admin log/Archive 20 (→October 18: fixed various things)
- 13:49, 18 October 2008 (diff | hist) . . (+62) . . Server admin log/Archive 20 (→October 18: down again, single process allocating all memory)
- 07:41, 18 October 2008 (diff | hist) . . (+77) . . Server admin log/Archive 20 (→October 18: taken down)
- 06:32, 18 October 2008 (diff | hist) . . (-13) . . Server admin log/Archive 20 (removed broken log entry thing, edit token is saved into the parser cache and so is wrong if someone else tries to edit)
- 06:28, 18 October 2008 (diff | hist) . . (+30) . . Server admin log/Archive 20 (→October 18: restarted srv160)
- 05:46, 18 October 2008 (diff | hist) . . (+109) . . Server admin log/Archive 20 (→October 18: took srv160 into the purple for a much more convincing overload, and different oprofile results)
- 04:21, 18 October 2008 (diff | hist) . . (+140) . . Server admin log/Archive 20 (→October 18: used oprofile to determine what part of the kernel is responsible for the system CPU spike. Looks like a spinlock in dnotify.)
- 03:13, 18 October 2008 (diff | hist) . . (+180) . . Server admin log/Archive 20 (simulated a memory-intensive request rate spike to srv160. Large system CPU response spike, but it didn't go down completely. Will try a bigger one.)
- 09:06, 17 October 2008 (diff | hist) . . (+86) . . Server admin log/Archive 20 (→October 17: copying srv129 and srv139 ES data directories to storage2:/export/backup)
- 02:49, 17 October 2008 (diff | hist) . . (+114) . . Server admin log/Archive 20 (→October 17: excessive lag on db16, killed long-running queries and temporarily depooled. CUPS odyssey continues.)
- 02:00, 17 October 2008 (diff | hist) . . (+62) . . Server admin log/Archive 20 (→October 17: removing cups on all servers where it is running)
- 01:05, 17 October 2008 (diff | hist) . . (+51) . . Server admin log/Archive 20 (→October 16: srv43-47 log entry based on uptime)
- 14:46, 16 October 2008 (diff | hist) . . (+95) . . Server admin log/Archive 20 (→October 16: added a more comprehensive redirection list to squid.conf.php for storage1 images)
- 14:06, 16 October 2008 (diff | hist) . . (+147) . . Server admin log/Archive 20 (→October 16: redirected images for /wikipedia/en/ to storage1, apparently they were moved a while ago. Refactored the relevant squid.conf section.)
- 13:38, 16 October 2008 (diff | hist) . . (+157) . . Server admin log/Archive 20 (→October 16: disabled directory index on amane. Was generating massive amounts of NFS traffic by generating a directory index for some timeline directories.)
- 12:52, 16 October 2008 (diff | hist) . . (+85) . . Server admin log/Archive 20 (→October 16: increased memory limit)
- 11:39, 16 October 2008 (diff | hist) . . (+71) . . Server admin log/Archive 20 (→October 16: cleaned up temporary files on srv159, had filled its disk)
- 11:38, 16 October 2008 (diff | hist) . . (+54) . . Server admin log/Archive 20 (→October 16)
- 10:07, 16 October 2008 (diff | hist) . . (+80) . . Server admin log/Archive 20 (→October 16: removed sq50 from the squid node lists and uninstalled squid on it)
- 10:02, 16 October 2008 (diff | hist) . . (+437) . . Server admin log/Archive 20 (→October 16: downtime report)
- 09:14, 16 October 2008 (diff | hist) . . (+231) . . Dsh (command lines)
- 08:54, 16 October 2008 (diff | hist) . . (+36) . . Dsh (single quotes)
- 08:52, 16 October 2008 (diff | hist) . . (+17) . . N DSH (Redirecting to Dsh) (top)
- 08:52, 16 October 2008 (diff | hist) . . (+735) . . Dsh (two of them)
- 08:27, 16 October 2008 (diff | hist) . . (+51) . . Server admin log/Archive 20 (→October 16: Restarted ntpd on search7, was broken)
- 07:04, 16 October 2008 (diff | hist) . . (+276) . . Server admin log/Archive 20 (→October 15: NTP issues)
- 12:01, 15 October 2008 (diff | hist) . . (+114) . . Server admin log/Archive 20 (→October 15: trying to bring srv159 up as an image scaler)
- 11:42, 15 October 2008 (diff | hist) . . (+75) . . Server admin log/Archive 20 (→October 15: srv127 died just the same. Mark suggests using one with DRAC next.)
- 10:32, 15 October 2008 (diff | hist) . . (+126) . . Server admin log/Archive 20 (→October 15: all image scalers (srv43 and srv100) swapped to death again. Preparing srv127 as an image scaler with swap off.)
- 08:44, 15 October 2008 (diff | hist) . . (+181) . . Server admin log/Archive 20 (→October 15: reduced depool-threshold for the scalers to 0.1 since srv100 is quite capable of handling the load by itself while we're waiting for the other servers to come back up.)
- 07:45, 15 October 2008 (diff | hist) . . (+130) . . Server admin log/Archive 20 (→October 15: half the scaling cluster went down again, ganglia shows high system CPU. Installing wikimedia-task-scaler on srv100.)
- 02:31, 15 October 2008 (diff | hist) . . (+64) . . Server admin log/Archive 20 (→October 15: moved image scalers into their own ganglia cluster)
- 02:22, 15 October 2008 (diff | hist) . . (+295) . . Server admin log/Archive 20 (→October 15: image scalers)
- 02:54, 14 October 2008 (diff | hist) . . (+67) . . Server admin log/Archive 20 (→October 14: fixed)
- 00:26, 14 October 2008 (diff | hist) . . (+224) . . Service IPs (more IPs)
- 00:22, 14 October 2008 (diff | hist) . . (+128) . . IP addresses (→Internal global assignments: more service IPs)
- 00:11, 14 October 2008 (diff | hist) . . (+54) . . Server admin log/Archive 20 (→October 14)
- 00:06, 14 October 2008 (diff | hist) . . (+70) . . Server admin log/Archive 20 (→October 13: switched image scaling LVS to srv43-47)
- 23:45, 13 October 2008 (diff | hist) . . (+56) . . Server admin log/Archive 20 (→October 13: prepping srv43-47 as image scaling servers)
- 13:28, 13 October 2008 (diff | hist) . . (+64) . . Server admin log/Archive 20 (→October 13: another domas log entry)
- 13:25, 13 October 2008 (diff | hist) . . (+123) . . Server admin log/Archive 20 (→October 13: more logging for domas)
- 13:12, 13 October 2008 (diff | hist) . . (+44) . . Server admin log/Archive 20 (→October 13: log for domas)
- 09:11, 13 October 2008 (diff | hist) . . (+33) . . Server admin log/Archive 20 (→October 13: restarted logmsgbot)
- 09:08, 13 October 2008 (diff | hist) . . (+73) . . m Logmsgbot
- 09:07, 13 October 2008 (diff | hist) . . (+2) . . m Logmsgbot
- 09:06, 13 October 2008 (diff | hist) . . (+1,269) . . N Udprec (New page: Simple UDP receiver, outputs on stdout. Probably written by River. <source lang="c"> #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> #incl...) (top)
- 09:04, 13 October 2008 (diff | hist) . . (+154) . . N Logmsgbot (New page: logmsgbot is udprec piped into ircecho. <tt> cd /home/kate ./udprec 53412 127.0.0.1 | ircecho '#wikimedia-tech' logmsgbot irc.freenode.net </tt>)
- 08:28, 13 October 2008 (diff | hist) . . (+37) . . Server admin log/Archive 20 (→October 13: folded s2b back into s2)
- 08:06, 13 October 2008 (diff | hist) . . (+70) . . Server admin log/Archive 20 (→October 13: db13 in rotation. Domas is copying from db15 to lomaria.)
- 07:39, 13 October 2008 (diff | hist) . . (+41) . . Server admin log/Archive 20 (→October 13)
- 04:51, 13 October 2008 (diff | hist) . . (+21) . . Server admin log/Archive 20 (→October 13: copying)
- 03:28, 13 October 2008 (diff | hist) . . (+50) . . Server admin log/Archive 20 (→October 13: Preparing for copy from db15 to db13)
- 19:53, 12 October 2008 (diff | hist) . . (+146) . . Server admin log/Archive 20 (→October 12)
- 19:21, 12 October 2008 (diff | hist) . . (+28) . . Server admin log/Archive 20 (→October 12: depooled db15.)
- 19:10, 12 October 2008 (diff | hist) . . (+104) . . Server admin log/Archive 20 (→October 12: split off some wikis into s2b and put db8 on it. To reduce I/O and hopefully stop the lag.)
- 18:52, 12 October 2008 (diff | hist) . . (+97) . . Server admin log/Archive 20 (→October 12: db15 still chronically lagged. Offloading all s2 RCL and category queries to db13.)
- 18:39, 12 October 2008 (diff | hist) . . (+52) . . Server admin log/Archive 20 (→October 12: offloading commons RCL queries to db13)
- 18:37, 12 October 2008 (diff | hist) . . (+48) . . Server admin log/Archive 20 (→October 12: dewiki r/w with ixia (master) only)
- 18:34, 12 October 2008 (diff | hist) . . (+57) . . Server admin log/Archive 20 (→October 12)
- 18:26, 12 October 2008 (diff | hist) . . (+69) . . Server admin log/Archive 20 (→October 12: balancing load. Fixed ganglia on various mysql servers.)
- 18:07, 12 October 2008 (diff | hist) . . (+85) . . Server admin log/Archive 20 (→October 12: going to r/w on s2. Not s2a yet because db15/db8 can't handle the load.)
- 17:46, 12 October 2008 (diff | hist) . . (+48) . . Server admin log/Archive 20 (→October 12: db8->db15 copy finished, deploying)
- 17:34, 12 October 2008 (diff | hist) . . (+40) . . Server admin log/Archive 20 (→October 12: installed NRPE on thistle.)
- 17:16, 12 October 2008 (diff | hist) . . (+7) . . Server admin log/Archive 20 (→October 12)
- 17:14, 12 October 2008 (diff | hist) . . (+99) . . Server admin log/Archive 20 (→October 12: copied binaries from db11 to db15 and thistle)
- 16:44, 12 October 2008 (diff | hist) . . (+333) . . Server admin log/Archive 20 (→October 12)
- 16:08, 12 October 2008 (diff | hist) . . (+35) . . Server admin log/Archive 20 (→October 12: repooled ixia/db8 r/o)
- 15:54, 12 October 2008 (diff | hist) . . (+93) . . Server admin log/Archive 20 (removed ixia binlogs 290-349. 270-289 were deleted during the initial response.)
- 14:38, 12 October 2008 (diff | hist) . . (+73) . . Server admin log/Archive 20 (→October 12: deployed r41995 as a live patch to replace buggy temp hack.)
- 14:16, 12 October 2008 (diff | hist) . . (+100) . . Server admin log/Archive 20 (cleaned up binlogs on db2. Yes the horse has bolted, but we may as well shut the gate.)
- 14:12, 12 October 2008 (diff | hist) . . (+46) . . Server admin log/Archive 20 (→October 12: copy now in progress as planned.)
- 13:51, 12 October 2008 (diff | hist) . . (+185) . . Server admin log/Archive 20 (→October 12: the plan)
- 13:41, 12 October 2008 (diff | hist) . . (+143) . . Server admin log/Archive 20 (ixia (s2 master) disk full. Classic scenario, binlogs stopped first, writing continued for 10 minutes before replag was reported.)
- 02:56, 12 October 2008 (diff | hist) . . (+101) . . Server admin log/Archive 20 (→October 11: attempted to disable directory listing on ms1. Gave up after a while.)
- 13:06, 10 October 2008 (diff | hist) . . (+80) . . Server admin log/Archive 20 (→October 10: reverted live patch and merged properly tested fix r41928 instead.)
- 12:37, 10 October 2008 (diff | hist) . . (+219) . . Server admin log/Archive 20 (s2 issues)
- 09:32, 2 October 2008 (diff | hist) . . (+88) . . Server admin log/Archive 20 (→October 1: srv104 (cluster18) hit max_rows, finally. Removed it from the write list.)
- 08:56, 2 October 2008 (diff | hist) . . (+59) . . Server admin log/Archive 20 (→October 1: fixed ipb_allow_usertalk default on all wikis)
- 15:46, 30 September 2008 (diff | hist) . . (+75) . . Server admin log/Archive 20 (doing some manual purges of URLs requested on #wikimedia-tech)
- 13:12, 30 September 2008 (diff | hist) . . (+165) . . Server admin log/Archive 20 (→September 30: cluster13 and cluster14 both have only one server left in rotation.)
- 10:17, 30 September 2008 (diff | hist) . . (+68) . . Server admin log/Archive 20 (→September 30: Switched ES cluster 3-10 to use Ubuntu servers (again))
- 10:03, 30 September 2008 (diff | hist) . . (+51) . . Server admin log/Archive 20 (→September 30: srv127)
- 10:01, 30 September 2008 (diff | hist) . . (+39) . . Server admin log/Archive 20 (→September 30: thistle down)
- 09:41, 30 September 2008 (diff | hist) . . (+47) . . N UDP logging (two things) (top)
- 09:39, 30 September 2008 (diff | hist) . . (+889) . . N MediaWiki UDP logging (docs for UDP logging) (top)
- 09:26, 30 September 2008 (diff | hist) . . (+46) . . Server admin log/Archive 20 (→September 30: Set up MediaWiki UDP logging)
- 08:06, 30 September 2008 (diff | hist) . . (+147) . . Server admin log/Archive 20 (→September 30: removed the ORDER BY clauses from the ApiQueryCategoryMembers queries, to work around MySQL bug, probably involving truncated indexes)
- 07:10, 30 September 2008 (diff | hist) . . (+183) . . Server admin log/Archive 20 (→September 29: ixia issues)
- 08:48, 29 September 2008 (diff | hist) . . (+57) . . Server admin log/Archive 20 (→September 29: put db7 back into watchlist rotation (99%))
- 12:07, 28 September 2008 (diff | hist) . . (+82) . . Server admin log/Archive 20 (→September 28: fixed db7)
- 11:54, 28 September 2008 (diff | hist) . . (+236) . . Server admin log/Archive 20 (→September 28: db7 issues)
- 05:39, 28 September 2008 (diff | hist) . . (+36) . . Server admin log/Archive 20 (→September 28: svn up/scap to r41337.)
- 04:45, 28 September 2008 (diff | hist) . . (+103) . . Server admin log/Archive 20 (→September 27: applying patch-ipb_allow_usertalk.sql on all DBs)
- 03:33, 25 September 2008 (diff | hist) . . (+205) . . Server admin log/Archive 20 (→September 25: stopped apache on srv167, was segfaulting again.)
- 03:04, 25 September 2008 (diff | hist) . . (+77) . . Server admin log/Archive 20 (→September 25: restarted segfaulting apaches srv111,srv168,srv154,srv167,srv46)
- 02:30, 25 September 2008 (diff | hist) . . (+309) . . Server admin log/Archive 20 (fixed srv35)
- 07:18, 24 September 2008 (diff | hist) . . (+72) . . Server admin log/Archive 20 (→September 23: installed wikimedia-nis-client on db20)
- 14:54, 23 September 2008 (diff | hist) . . (+349) . . Server admin log/Archive 20 (→September 23: two more)
- 13:56, 23 September 2008 (diff | hist) . . (+42) . . Server admin log/Archive 20 (→September 23: started rsync daemon on db20)
- 13:55, 23 September 2008 (diff | hist) . . (+93) . . Server admin log/Archive 20 (→September 23: two events)
- 01:21, 23 September 2008 (diff | hist) . . (+202) . . Server admin log/Archive 20 (→September 22: copying everything from the recovered suda image except /home/kate/xx, /home/from-zwinger and /home/wikipedia/logs)
- 10:11, 21 September 2008 (diff | hist) . . (+135) . . Server admin log/Archive 20 (→September 20: disabled srv106's switch port. Was running the job queue with old configuration, inaccessible by ssh.)
- 15:03, 20 September 2008 (diff | hist) . . (+163) . . Server admin log/Archive 20 (→September 20: re-enabled Special:Export with $wgExportAllowHistory=false. Please find some way of doing transwiki requests which doesn't involve crashing the site)
- 14:32, 20 September 2008 (diff | hist) . . (+342) . . Server admin log/Archive 20 (→September 20: ES overload)
- 10:12, 20 September 2008 (diff | hist) . . (+9) . . Memcached (old)
- 10:10, 20 September 2008 (diff | hist) . . (+68) . . Server admin log/Archive 20 (→September 20: removed srv69 and srv118 from the memcached list, down)
- 09:03, 20 September 2008 (diff | hist) . . (+112) . . Server admin log/Archive 20 (→September 20: ES on srv84 had new passwords, was not accepting connections from 3.23 clients on srv32-34. Fixed.)
- 08:52, 20 September 2008 (diff | hist) . . (+72) . . Server admin log/Archive 20 (→September 20: srv137 has a corrupt read-only file system on /usr/local/mysql/data2)
- 08:49, 20 September 2008 (diff | hist) . . (+182) . . Server admin log/Archive 20 (→September 20: srv110, srv137)
- 05:35, 20 September 2008 (diff | hist) . . (+132) . . Server admin log/Archive 20 (→September 20: svn.wikimedia.org not reachable from zwinger via IPv6, causing very slow operation due to timeouts. Hacked /etc/hosts.)
- 05:16, 20 September 2008 (diff | hist) . . (+35) . . Server admin log/Archive 20 (→September 20: svn up/scap to r41053)
- 01:07, 20 September 2008 (diff | hist) . . (+206) . . Server admin log/Archive 20 (→September 20: ES migration failed on all clusters except cluster3 (the cluster I used to test the script), due to MySQL 4.0-4.1 version differences. Restarting with mysqldump --default-character-)
- 00:15, 20 September 2008 (diff | hist) . . (+148) . . Server admin log/Archive 20 (restarted segfaulting apaches)
- 07:37, 19 September 2008 (diff | hist) . . (+252) . . Server admin log/Archive 20 (→September 19: srv38 issues)
- 02:40, 19 September 2008 (diff | hist) . . (+29) . . N Wmerrors (Redirecting to PHP error pages) (top)
- 02:39, 19 September 2008 (diff | hist) . . (+854) . . N PHP error pages (docs for wmerrors)
- 01:57, 19 September 2008 (diff | hist) . . (+110) . . Server admin log/Archive 20 (→September 19: the issue with ES was the lack of a master pos wait between transfer and slave shutdown. Fixing.)
- 01:02, 19 September 2008 (diff | hist) . . (+237) . . Server admin log/Archive 20 (→September 18: restarting apaches)
- 11:45, 18 September 2008 (diff | hist) . . (+55) . . Server admin log/Archive 20 (→September 18: reverted db.php change, still has issues)
- 11:18, 18 September 2008 (diff | hist) . . (+89) . . m Server admin log/Archive 20 (→September 18)
- 11:10, 18 September 2008 (diff | hist) . . (+96) . . Server admin log/Archive 20 (in db.php, switched ES clusters 3-10 to use the ubuntu servers)
- 22:46, 15 September 2008 (diff | hist) . . (+81) . . Server admin log/Archive 20 (rebooting srv151)
- 15:59, 12 September 2008 (diff | hist) . . (+135) . . Switch master (two more steps following observed pitfall)
- 15:31, 12 September 2008 (diff | hist) . . (+57) . . Server admin log/Archive 20 (→September 12: set read_only=0 on srv108 (Rob rebooted it))
- 14:58, 12 September 2008 (diff | hist) . . (+86) . . Server admin log/Archive 20 (→September 12: migrate-all)
- 14:31, 12 September 2008 (diff | hist) . . (+66) . . Server admin log/Archive 20 (→September 12: Removed the corrupted ES installations on srv151-176)
- 09:02, 12 September 2008 (diff | hist) . . (+74) . . Server admin log/Archive 20 (→September 12: reverted, blob corruption due to charset conversion observ)
- 07:59, 12 September 2008 (diff | hist) . . (+85) . . Server admin log/Archive 20 (→September 12: Experimentally switched db.php to use the ubuntu servers for cluster3/4)
- 07:52, 12 September 2008 (diff | hist) . . (+302) . . Server admin log/Archive 20 (→September 11: lock cluster3 and 4)
- 05:31, 11 September 2008 (diff | hist) . . (+90) . . Server admin log/Archive 20 (→September 10: Migrating cluster4. Testing new binlog deletion feature.)
- 10:59, 10 September 2008 (diff | hist) . . (+134) . . Server admin log/Archive 20 (copied cluster3 to srv151, srv163 and srv175, second attempt, seems to have worked this time)
- 10:26, 9 September 2008 (diff | hist) . . (+69) . . Server admin log/Archive 20 (→September 9: copying ES data from srv32 to srv151, srv163 and srv175)
- 10:16, 9 September 2008 (diff | hist) . . (+101) . . Server admin log/Archive 20 (testing and debugging the ubuntu ES migration script)
- 00:49, 9 September 2008 (diff | hist) . . (+103) . . Server admin log/Archive 20 (granted root access to zwinger on all ES servers)
- 09:59, 8 September 2008 (diff | hist) . . (+60) . . IPMI (old)
- 06:17, 8 September 2008 (diff | hist) . . (+46) . . Server admin log/Archive 20 (→September 8)
- 05:34, 8 September 2008 (diff | hist) . . (+51) . . Server admin log/Archive 20 (→September 8: srv146 down, removed from ES rotation)
- 05:08, 8 September 2008 (diff | hist) . . (+75) . . Server admin log/Archive 20 (accidentally crashed srv37. Needs restart.)
- 17:04, 5 September 2008 (diff | hist) . . (+58) . . Server admin log/Archive 20 (→September 5: Updated trusted-xff.cdb. Fixes AOL problems.)
- 04:52, 5 September 2008 (diff | hist) . . (+114) . . Server admin log/Archive 20 (sync-file screwup caused thumbnails to be created in the source image directory- →September 5)
- 03:18, 5 September 2008 (diff | hist) . . (+224) . . Server admin log/Archive 20 (→September 4: srv151)
- 16:48, 4 September 2008 (diff | hist) . . (+133) . . Server admin log/Archive 20 (→September 4: re-enabled thumb.php after some of the culprits came to talk to me on #wikimedia-tech and promised to reform their ways)
- 11:09, 4 September 2008 (diff | hist) . . (+55) . . Server admin log/Archive 20 (→September 4: fixed APC on srv38 and srv39, was broken.)
- 10:39, 4 September 2008 (diff | hist) . . (+104) . . Server admin log/Archive 20 (→September 4: srv38 and srv39 have been overloaded since 05:50. Blocked thumb.php for external clients.)
- 10:14, 4 September 2008 (diff | hist) . . (+179) . . Ubuntu todo (mysql) (top)
- 10:12, 4 September 2008 (diff | hist) . . (+122) . . Ubuntu todo (more strikethrough)
- 05:35, 4 September 2008 (diff | hist) . . (+90) . . Server admin log/Archive 20 (→September 4)
- 04:11, 4 September 2008 (diff | hist) . . (+90) . . Server admin log/Archive 20 (upgrading to wmerrors-1.0.2 on all mediawiki-installation)
- 04:28, 3 September 2008 (diff | hist) . . (+35) . . Server admin log/Archive 20 (→September 3: svn up/scap to r40356)
- 04:17, 3 September 2008 (diff | hist) . . (+40) . . Server admin log/Archive 20 (→September 3: populating ss_active_users)
- 03:25, 3 September 2008 (diff | hist) . . (+67) . . Server admin log/Archive 20 (applying patch-ss_active_users.sql)
- 09:43, 1 September 2008 (diff | hist) . . (+63) . . Server admin log/Archive 20 (→September 1: srv102 done, re-added cluster17 to the write list)
- 04:13, 1 September 2008 (diff | hist) . . (+55) . . Server admin log/Archive 20 (→September 1: repooled ES on srv107, schema change done)
- 03:57, 1 September 2008 (diff | hist) . . (+73) . . Server admin log/Archive 20 (→September 1: srv105)
- 03:50, 1 September 2008 (diff | hist) . . (+103) . . Server admin log/Archive 20 (→September 1: starting max_rows change on srv102. srv107 is actually stopped due to disk full, fixing.)
- 03:41, 1 September 2008 (diff | hist) . . (+55) . . Server admin log/Archive 20 (→September 1: switching masters on cluster17 to srv103.)
- 02:18, 1 September 2008 (diff | hist) . . (+68) . . Server admin log/Archive 20 (→September 1: Killed job runner on srv107 to speed up schema change.)
- 02:16, 1 September 2008 (diff | hist) . . (+89) . . Server admin log/Archive 20 (→August 31: Brought srv142 and srv145 into ES rotation in cluster16.)
- 18:42, 31 August 2008 (diff | hist) . . (+477) . . Server admin log/Archive 20 (status)
- 18:33, 31 August 2008 (diff | hist) . . (+91) . . Server admin log/Archive 20 (→August 31: re-adding cluster19 to the write list, without srv107 which is still altering)
- 16:23, 31 August 2008 (diff | hist) . . (+100) . . Server admin log/Archive 20 (→August 31: srv141 didn't work out, out of disk space, trying copy to srv142 instead)
- 14:49, 31 August 2008 (diff | hist) . . (+48) . . Server admin log/Archive 20 (→August 31: srv103 and srv110 done, repooling.)
- 14:15, 31 August 2008 (diff | hist) . . (+9) . . m Server admin log/Archive 20 (→August 31)
- 14:09, 31 August 2008 (diff | hist) . . (+86) . . Server admin log/Archive 20 (→August 31: srv108 done, changed master to srv108, started max_rows change on srv107)
- 13:55, 31 August 2008 (diff | hist) . . (+93) . . Server admin log/Archive 20 (→August 31: started max_rows change on srv110. Not patient enough to do them one at a time.)
- 13:46, 31 August 2008 (diff | hist) . . (+182) . . Server admin log/Archive 20 (→August 31: retrolog)
- 13:31, 31 August 2008 (diff | hist) . . (+100) . . Server admin log/Archive 20 (→August 31: taking srv145 out of rotation for copy to new ext store srv141)
- 12:49, 31 August 2008 (diff | hist) . . (+49) . . Server admin log/Archive 20 (→August 31: srv109 finished, starting on srv108)
- 11:49, 31 August 2008 (diff | hist) . . (+76) . . Server admin log/Archive 20 (→August 31: taking srv103 out of rotation for copy to new ext store srv110)
- 11:41, 31 August 2008 (diff | hist) . . (+61) . . Server admin log/Archive 20 (→August 31: alter table blobs max_rows=10000000; on srv109.)
- 11:35, 31 August 2008 (diff | hist) . . (+192) . . Server admin log/Archive 20 (saving existing)
- 10:36, 31 August 2008 (diff | hist) . . (+187) . . Server admin log/Archive 20 (→August 31: reports that cluster19 has gone the same way)
- 10:12, 31 August 2008 (diff | hist) . . (+176) . . Server admin log/Archive 20 (→August 31: cluster17 master srv102 has stopped being writable for enwiki due to exhausted MyISAM index table size (max_rows=1000000))
- 08:29, 31 August 2008 (diff | hist) . . (+117) . . Server admin log/Archive 20 (On srv189: added ddebs.ubuntu.com to sources.list. Installed debug symbols for apache.)
- 17:18, 30 August 2008 (diff | hist) . . (+82) . . Server admin log/Archive 20 (convert)
- 13:14, 29 August 2008 (diff | hist) . . (+335) . . Server admin log/Archive 20 (fixed kuwiki)
- 14:15, 28 August 2008 (diff | hist) . . (+57) . . Server admin log/Archive 20 (→August 28: They became visible due to Rob's changes to langlist.)
- 13:36, 28 August 2008 (diff | hist) . . (+214) . . Server admin log/Archive 20 (→August 28: removed dkwiktionary and dkwikibooks from all.dblist)
- 06:05, 28 August 2008 (diff | hist) . . (+200) . . Server admin log/Archive 20 (→August 27: chmod 777)
- 08:46, 27 August 2008 (diff | hist) . . (+476) . . Cron jobs (updated)
- 08:43, 27 August 2008 (diff | hist) . . (+62) . . Server admin log/Archive 20 (→August 27: added FlaggedRevs stats update to crontab on hume)
- 08:09, 27 August 2008 (diff | hist) . . (+89) . . Server admin log/Archive 20 (running FlaggedRevs/maintenance/updateLinks.php on dewiki)
- 13:33, 26 August 2008 (diff | hist) . . (+54) . . Server admin log/Archive 20 (→August 26: disabled some user account on otrs-wiki)
- 07:31, 24 August 2008 (diff | hist) . . (+66) . . Server admin log/Archive 20 (→August 24: deployed the TrustedXFF extension that I just wrote.)
- 03:01, 24 August 2008 (diff | hist) . . (+310) . . Server admin log/Archive 20 (offloaded db9)
- 11:54, 21 August 2008 (diff | hist) . . (+39) . . Server admin log/Archive 20 (→August 21)
- 02:46, 21 August 2008 (diff | hist) . . (+65) . . Server admin log/Archive 20 (→August 21: returned db16 to general load, a less critical role)
- 02:42, 21 August 2008 (diff | hist) . . (+110) . . Server admin log/Archive 20 (recovery spike)
- 02:39, 21 August 2008 (diff | hist) . . (+135) . . Server admin log/Archive 20 (→August 21: installed stuff)
- 02:31, 21 August 2008 (diff | hist) . . (+86) . . Server admin log/Archive 20 (db16 offloaded)
- 02:18, 21 August 2008 (diff | hist) . . (+53) . . m Server admin log/Archive 20
- 02:16, 21 August 2008 (diff | hist) . . (+111) . . Server admin log/Archive 20 (db16 is chronically lagged, probably overloaded with inflexible query group load)
- 04:15, 20 August 2008 (diff | hist) . . (+65) . . Server admin log/Archive 20 (upgrading hume)
- 08:42, 19 August 2008 (diff | hist) . . (+31) . . Server admin log/Archive 20 (→August 19: done with lomaria)
- 07:48, 19 August 2008 (diff | hist) . . (+110) . . Server admin log/Archive 20 (taking lomaria out of rotation)
- 07:34, 19 August 2008 (diff | hist) . . (+9) . . Spread (old) (top)
(Latest | Earliest) View (newer 500 | older 500) (20 | 50 | 100 | 250 | 500)
