Network design
From Wikitech
The purpose of this page is to give an overview of the current design of the network of the Wikimedia servers, and to provide a place to develop a new and improved network scheme.
Contents |
Automatically generated information
Generated 2005-02-24
Default gateway
[root@zwinger node_groups]# dsh -N working "route | grep default" executing 'route | grep default' albert: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 ariel: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0 avicenna: default izwinger 0.0.0.0 UG 0 0 0 eth1 bacon: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 bart: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 bayle: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 browne: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 dalembert: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 diderot: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0 friedrich: default izwinger 0.0.0.0 UG 0 0 0 eth1 goeje: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 harris: default izwinger 0.0.0.0 UG 0 0 0 eth1 suda: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0 tingxi: default izwinger 0.0.0.0 UG 0 0 0 eth0 will: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0 zwinger: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0 hypatia: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 humboldt: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 kluge: default izwinger 0.0.0.0 UG 0 0 0 eth1 khaldun: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0 larousse: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0 webster: default izwinger 0.0.0.0 UG 0 0 0 eth1 holbach: default izwinger 0.0.0.0 UG 0 0 0 eth1 benet: default izwinger 0.0.0.0 UG 0 0 0 eth0 ibiruni: default 10.255.255.254 0.0.0.0 UG 0 0 0 eth0 irose: default izwinger 0.0.0.0 UG 0 0 0 eth0 ismellie: default izwinger 0.0.0.0 UG 0 0 0 eth0 ianthony: default izwinger 0.0.0.0 UG 0 0 0 eth0 ennael: default router-wikipedi 0.0.0.0 UG 0 0 0 eth0 chloe: default router-wikipedi 0.0.0.0 UG 0 0 0 eth0 bleuenn: default router-wikipedi 0.0.0.0 UG 0 0 0 eth0
Cables connected
[root@zwinger node_groups]# dsh -N working mii-tool executing 'mii-tool' albert: SIOCGMIIPHY on 'eth0' failed: Operation not supported albert: SIOCGMIIPHY on 'eth1' failed: Operation not supported albert: no MII interfaces found alrazi: eth0: no link alrazi: eth1: negotiated 100baseTx-FD, link ok ariel: eth0: negotiated 100baseTx-FD, link ok ariel: eth1: negotiated 100baseTx-FD flow-control, link ok avicenna: eth0: no link avicenna: eth1: negotiated 100baseTx-FD flow-control, link ok bacon: eth0: negotiated 100baseTx-FD, link ok bacon: eth1: negotiated 100baseTx-FD flow-control, link ok bart: eth0: negotiated 100baseTx-FD, link ok bart: eth1: no link bayle: eth0: negotiated 100baseTx-FD, link ok bayle: eth1: no link browne: eth0: negotiated 100baseTx-FD, link ok browne: eth1: no link dalembert: eth0: no link dalembert: eth1: negotiated 100baseTx-FD, link ok diderot: eth0: no link diderot: eth1: negotiated 100baseTx-FD flow-control, link ok friedrich: eth0: no link friedrich: eth1: negotiated 100baseTx-FD, link ok goeje: eth0: no link goeje: eth1: negotiated 100baseTx-FD flow-control, link ok harris: eth0: no link harris: eth1: negotiated 100baseTx-FD flow-control, link ok suda: eth0: negotiated 100baseTx-FD, link ok suda: eth1: negotiated 100baseTx-FD flow-control, link ok tingxi: eth0: negotiated 100baseTx-FD flow-control, link ok tingxi: eth1: no link will: eth0: negotiated 100baseTx-FD, link ok will: eth1: negotiated 100baseTx-FD flow-control, link ok zwinger: eth0: negotiated 100baseTx-FD, link ok zwinger: eth1: negotiated 100baseTx-FD, link ok hypatia: eth0: no link hypatia: eth1: negotiated 100baseTx-FD flow-control, link ok humboldt: eth0: no link humboldt: eth1: negotiated 100baseTx-FD flow-control, link ok kluge: eth0: no link kluge: eth1: negotiated 100baseTx-FD, link ok khaldun: eth0: no link khaldun: eth1: negotiated 100baseTx-FD flow-control, link ok larousse: eth0: negotiated 100baseTx-FD, link ok larousse: eth1: negotiated 100baseTx-FD flow-control, link ok webster: eth0: negotiated 100baseTx-FD, link ok webster: eth1: negotiated 100baseTx-FD flow-control, link ok holbach: eth0: negotiated 100baseTx-FD, link ok holbach: eth1: negotiated 100baseTx-FD flow-control, link ok benet: eth0: negotiated 100baseTx-FD, link ok ibiruni: eth0: negotiated 100baseTx-FD flow-control, link ok ibiruni: eth1: 10 Mbit, half duplex, no link irose: eth0: negotiated 100baseTx-FD, link ok irose: eth1: 10 Mbit, half duplex, no link ismellie: eth0: negotiated 100baseTx-FD, link ok ismellie: eth1: 10 Mbit, half duplex, no link ianthony: eth0: negotiated 100baseTx-FD, link ok ianthony: eth1: 10 Mbit, half duplex, no link ennael: eth0: negotiated 100baseTx-FD flow-control, link ok ennael: eth1: no link chloe: eth0: negotiated 100baseTx-FD flow-control, link ok chloe: eth1: no link bleuenn: eth0: negotiated 100baseTx-FD flow-control, link ok bleuenn: eth1: no link
IP addresses
[root@zwinger node_groups]# dsh -N working 'ifconfig | grep "inet addr" | grep -v 127.0.0.1' executing 'ifconfig | grep "inet addr" | grep -v 127.0.0.1' albert: inet addr:207.142.131.201 Bcast:207.142.131.255 Mask:255.255.255.192 albert: inet addr:10.0.0.5 Bcast:10.0.255.255 Mask:255.255.0.0 alrazi: inet addr:10.0.0.10 Bcast:10.0.255.255 Mask:255.255.0.0 ariel: inet addr:207.142.131.244 Bcast:207.142.131.255 Mask:255.255.255.192 ariel: inet addr:10.0.0.2 Bcast:10.0.255.255 Mask:255.255.0.0 avicenna: inet addr:207.142.131.249 Bcast:207.142.131.255 Mask:255.255.255.192 avicenna: inet addr:10.0.0.14 Bcast:10.0.255.255 Mask:255.255.0.0 bacon: inet addr:207.142.131.200 Bcast:207.142.131.255 Mask:255.255.255.192 bacon: inet addr:10.0.0.3 Bcast:10.0.255.255 Mask:255.255.0.0 bart: inet addr:207.142.131.227 Bcast:207.142.131.255 Mask:255.255.255.192 bayle: inet addr:207.142.131.228 Bcast:207.142.131.255 Mask:255.255.255.192 browne: inet addr:207.142.131.229 Bcast:207.142.131.255 Mask:255.255.255.192 dalembert: inet addr:207.142.131.194 Bcast:207.142.131.255 Mask:255.255.255.192 dalembert: inet addr:10.0.0.13 Bcast:10.0.255.255 Mask:255.255.0.0 diderot: inet addr:207.142.131.250 Bcast:207.142.131.255 Mask:255.255.255.192 diderot: inet addr:10.0.0.7 Bcast:10.0.255.255 Mask:255.255.0.0 friedrich: inet addr:10.0.0.11 Bcast:10.0.255.255 Mask:255.255.0.0 goeje: inet addr:207.142.131.198 Bcast:207.142.131.255 Mask:255.255.255.192 goeje: inet addr:10.0.0.9 Bcast:10.0.255.255 Mask:255.255.0.0 harris: inet addr:207.142.131.199 Bcast:207.142.131.255 Mask:255.255.255.192 harris: inet addr:10.0.0.8 Bcast:10.0.255.255 Mask:255.255.0.0 suda: inet addr:207.142.131.226 Bcast:207.142.131.255 Mask:255.255.255.192 suda: inet addr:10.0.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 tingxi: inet addr:10.0.0.12 Bcast:10.0.255.255 Mask:255.255.0.0 will: inet addr:207.142.131.243 Bcast:207.142.131.255 Mask:255.255.255.192 will: inet addr:207.142.131.235 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.245 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.248 Bcast:207.142.131.255 Mask:255.255.255.255 will: inet addr:207.142.131.205 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.247 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.246 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.202 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.203 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.204 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.206 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:207.142.131.236 Bcast:207.142.131.255 Mask:255.255.255.0 will: inet addr:10.0.0.21 Bcast:10.0.255.255 Mask:255.255.0.0 zwinger: inet addr:207.142.131.234 Bcast:207.142.131.255 Mask:255.255.255.192 zwinger: inet addr:207.142.131.207 Bcast:207.142.131.255 Mask:255.255.255.255 zwinger: inet addr:207.142.131.216 Bcast:207.142.131.255 Mask:255.255.255.255 zwinger: inet addr:10.0.0.4 Bcast:10.0.255.255 Mask:255.255.0.0 hypatia: inet addr:207.142.131.212 Bcast:207.142.131.255 Mask:255.255.255.192 hypatia: inet addr:10.0.0.212 Bcast:10.0.255.255 Mask:255.255.0.0 humboldt: inet addr:207.142.131.213 Bcast:207.142.131.255 Mask:255.255.255.192 humboldt: inet addr:10.0.0.213 Bcast:10.0.255.255 Mask:255.255.0.0 kluge: inet addr:10.0.0.214 Bcast:10.0.255.255 Mask:255.255.0.0 khaldun: inet addr:207.142.131.218 Bcast:207.142.131.255 Mask:255.255.255.192 khaldun: inet addr:10.0.0.218 Bcast:10.0.255.255 Mask:255.255.0.0 larousse: inet addr:207.142.131.208 Bcast:207.142.131.255 Mask:255.255.255.192 larousse: inet addr:10.0.0.22 Bcast:10.0.255.255 Mask:255.255.0.0 larousse: inet addr:10.10.0.1 P-t-P:10.10.0.2 Mask:255.255.255.255 webster: inet addr:207.142.131.221 Bcast:207.142.131.255 Mask:255.255.255.192 webster: inet addr:10.0.0.23 Bcast:10.0.0.255 Mask:255.255.255.0 holbach: inet addr:207.142.131.224 Bcast:207.142.131.255 Mask:255.255.255.192 holbach: inet addr:10.0.0.24 Bcast:10.0.0.255 Mask:255.255.255.0 benet: inet addr:10.0.0.29 Bcast:10.0.255.255 Mask:255.255.0.0 ibiruni: inet addr:10.0.0.25 Bcast:10.255.255.255 Mask:255.0.0.0 irose: inet addr:10.0.0.26 Bcast:10.0.255.255 Mask:255.255.0.0 ismellie: inet addr:10.0.0.27 Bcast:10.0.255.255 Mask:255.255.0.0 ianthony: inet addr:10.0.0.28 Bcast:10.0.255.255 Mask:255.255.0.0 ennael: inet addr:212.85.150.131 Bcast:212.85.150.135 Mask:255.255.255.248 ennael: inet addr:192.168.0.30 Bcast:192.168.0.255 Mask:255.255.255.0 chloe: inet addr:212.85.150.132 Bcast:212.85.150.135 Mask:255.255.255.248 chloe: inet addr:192.168.0.10 Bcast:192.168.0.255 Mask:255.255.255.0 chloe: inet addr:10.10.0.2 P-t-P:10.10.0.1 Mask:255.255.255.255 bleuenn: inet addr:212.85.150.133 Bcast:212.85.150.135 Mask:255.255.255.248 bleuenn: inet addr:192.168.0.20 Bcast:192.168.0.255 Mask:255.255.255.0
Overall system design
The folowing is the general system design plan which the network layer must efficiently accommodate.
- Databases in a central pool with each serving a subset of the wikis, so each has high cache efficiency and the total number needed to handle any query load is minimised. Database servers cost US$5,000-$8,000 each, depending on exact equipment.
- A central pair of old text database servers (part of the long term storage growth plan for the databases, to move this high volume and seldom accessed data off costly and comparatively small disk systems).
- Memcached caching spread on apaches across the whole cluster, producing one very large cache pool, accessible from any apache and stored on half or more of the apaches. Segmenting the pool would decrease the overall hit rate, increasing the number of apaches and database servers required for any given system load level.
- Load balancing of squids and apaches, currently expected to use two or three systems between the internet and the squids and the same set between the squids and the apaches.
A key network systems design requirement is efficient access from any apache to any apache running memcached (expected to be more than half of all apaches) and efficient access from any apache to any database server. Losing this capability would dramatically increase overall system cost.
Current situation
Wikimedia servers reside in two racks along with Bomis servers, hosted at Candidhosting. Wikimedia/Bomis have a dedicated IP range, 207.142.131.192/26. There are two gateways: 207.142.131.193 and 207.142.131.225. Total burstable bandwidth is 1000 Mbit/s, delivered through one optic fiber 1000base-SX link.
Wikimedia owns a Cisco 3560G-48TX switch, and three Netgear Gigabit switches.
