Server admin log/Archive 5

28 March 05:09 (UTC, but cached)

10 April

~23:00: kate: upgraded squid to STABLE9+patches (see squid builds) + restarted all squids.
mark: All squids are running with too few FDs (1024), and if noone replaces all daemons by the new one Kate just built, we may have a problem tomorrow during peak hours...
19:15 midom: srv7 is now in squid service
19:07 brion: MWDaemon's memory usage got high enough it started swapping. Hung connections ate up apaches and hung the site until it was restarted.
5:30 brion: lucene search server active for en.wikipedia.org, running on vincent.

15:45 midom: dropped thttpd (as it was using 32bit mmaps) on dumps in favor of lighttpd. It has superb performance, serves 3500hits/s under ab and served 70MB/s from benet in small reqs... Extreme recommendations for using lighttpd for image uploads.
10:15 brion: running lucene search indexer on vincent (pulling enwiki from benet).
05:25 brion: added additional is rcbots to #is.wikipedia for tionary/books/quote

16:00 midom: redirected http://download.wikimedia.org/ to benet, misses tomeraider and uploads...
13:00 Tim: switched to Mark's squid binary on the French squids

Mark, Tim: implemented Multicast HTCP purging on all FL apaches/squids. French Squids still need a binary replacement.

21:44 mark: Put port gi0/26 on csw1-pmtpa into trunking mode: vlans 1-2 only, with vlan 2 being the native vlan, no LACP negotiation
11:30 midom: benet put into dump operation
10:55 brion: reinstalled PHP on zwinger and apaches, compiled with memory limit and mbstring options enabled. This was left out when upgrading to 4.3.11.
2:40 brion: added NetCabo proxies to trusted proxy list (inconveniently shared by Jorge and a Nazi vandal on pt.wikipedia.org)

15:30 jeluf: disbaled logging of upload.wikimedia.org
15:15 midom: yet another image server overload. rotated 30G upload.wikimedia logfile, could be fragmentation overhead.
12:00 midom: moved log_bin.0[0123]? (40G worth of binlogs) from ariel to khaldun/avicenna backup/arielbinlog, reclaimed some master disk space.
- Do we need those binlogs for anything?
07:48 Tim: Started memcached on browne, it was in the list but not running. Fixed startup scripts. Noticed that browne can't contact albert on 10/8, modified yum.conf accordingly.

22:30 midom: Enabled recentchanges-based watchlist hack. Servers go faaaast.
23:15 brion: set default block expiry to 1h on dewiki by request of various admins