Network design

From Wikitech
(Difference between revisions)
Jump to: navigation, search
m (→‎Current situation)
(removed out-of-date information)
Line 1: Line 1:
 
The purpose of this page is to give an overview of the current '''design of the network''' of the Wikimedia servers, and to provide a place to develop a new and improved network scheme.
 
The purpose of this page is to give an overview of the current '''design of the network''' of the Wikimedia servers, and to provide a place to develop a new and improved network scheme.
−
 
−
==Automatically generated information==
 
−
 
−
Generated 2005-02-24
 
−
 
−
===Default gateway===
 
−
 
−
<pre>
 
−
[root@zwinger node_groups]# dsh -N working "route | grep default"
 
−
executing 'route | grep default'
 
−
albert:        default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
ariel:          default        207.142.131.225 0.0.0.0        UG    0      0        0 eth0
 
−
avicenna:      default        izwinger        0.0.0.0        UG    0      0        0 eth1
 
−
bacon:          default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
bart:          default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
bayle:          default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
browne:        default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
dalembert:      default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
diderot:        default        207.142.131.225 0.0.0.0        UG    0      0        0 eth0
 
−
friedrich:      default        izwinger        0.0.0.0        UG    0      0        0 eth1
 
−
goeje:          default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
harris:        default        izwinger        0.0.0.0        UG    0      0        0 eth1
 
−
suda:          default        207.142.131.225 0.0.0.0        UG    0      0        0 eth0
 
−
tingxi:        default        izwinger        0.0.0.0        UG    0      0        0 eth0
 
−
will:          default        207.142.131.225 0.0.0.0        UG    0      0        0 eth0
 
−
zwinger:        default        207.142.131.225 0.0.0.0        UG    0      0        0 eth0
 
−
hypatia:        default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
humboldt:      default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
kluge:          default        izwinger        0.0.0.0        UG    0      0        0 eth1
 
−
khaldun:        default        207.142.131.193 0.0.0.0        UG    0      0        0 eth0
 
−
larousse:      default        207.142.131.225 0.0.0.0        UG    0      0        0 eth0
 
−
webster:        default        izwinger        0.0.0.0        UG    0      0        0 eth1
 
−
holbach:        default        izwinger        0.0.0.0        UG    0      0        0 eth1
 
−
benet:          default        izwinger        0.0.0.0        UG    0      0        0 eth0
 
−
ibiruni:        default        10.255.255.254  0.0.0.0        UG    0      0        0 eth0
 
−
irose:          default        izwinger        0.0.0.0        UG    0      0        0 eth0
 
−
ismellie:      default        izwinger        0.0.0.0        UG    0      0        0 eth0
 
−
ianthony:      default        izwinger        0.0.0.0        UG    0      0        0 eth0
 
−
ennael:        default        router-wikipedi 0.0.0.0        UG    0      0        0 eth0
 
−
chloe:          default        router-wikipedi 0.0.0.0        UG    0      0        0 eth0
 
−
bleuenn:        default        router-wikipedi 0.0.0.0        UG    0      0        0 eth0
 
−
</pre>
 
−
 
−
===Cables connected===
 
−
<pre>
 
−
[root@zwinger node_groups]# dsh -N working mii-tool
 
−
executing 'mii-tool'
 
−
albert:        SIOCGMIIPHY on 'eth0' failed: Operation not supported
 
−
albert:        SIOCGMIIPHY on 'eth1' failed: Operation not supported
 
−
albert:        no MII interfaces found
 
−
alrazi:        eth0: no link
 
−
alrazi:        eth1: negotiated 100baseTx-FD, link ok
 
−
ariel:          eth0: negotiated 100baseTx-FD, link ok
 
−
ariel:          eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
avicenna:      eth0: no link
 
−
avicenna:      eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
bacon:          eth0: negotiated 100baseTx-FD, link ok
 
−
bacon:          eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
bart:          eth0: negotiated 100baseTx-FD, link ok
 
−
bart:          eth1: no link
 
−
bayle:          eth0: negotiated 100baseTx-FD, link ok
 
−
bayle:          eth1: no link
 
−
browne:        eth0: negotiated 100baseTx-FD, link ok
 
−
browne:        eth1: no link
 
−
dalembert:      eth0: no link
 
−
dalembert:      eth1: negotiated 100baseTx-FD, link ok
 
−
diderot:        eth0: no link
 
−
diderot:        eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
friedrich:      eth0: no link
 
−
friedrich:      eth1: negotiated 100baseTx-FD, link ok
 
−
goeje:          eth0: no link
 
−
goeje:          eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
harris:        eth0: no link
 
−
harris:        eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
suda:          eth0: negotiated 100baseTx-FD, link ok
 
−
suda:          eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
tingxi:        eth0: negotiated 100baseTx-FD flow-control, link ok
 
−
tingxi:        eth1: no link
 
−
will:          eth0: negotiated 100baseTx-FD, link ok
 
−
will:          eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
zwinger:        eth0: negotiated 100baseTx-FD, link ok
 
−
zwinger:        eth1: negotiated 100baseTx-FD, link ok
 
−
hypatia:        eth0: no link
 
−
hypatia:        eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
humboldt:      eth0: no link
 
−
humboldt:      eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
kluge:          eth0: no link
 
−
kluge:          eth1: negotiated 100baseTx-FD, link ok
 
−
khaldun:        eth0: no link
 
−
khaldun:        eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
larousse:      eth0: negotiated 100baseTx-FD, link ok
 
−
larousse:      eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
webster:        eth0: negotiated 100baseTx-FD, link ok
 
−
webster:        eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
holbach:        eth0: negotiated 100baseTx-FD, link ok
 
−
holbach:        eth1: negotiated 100baseTx-FD flow-control, link ok
 
−
benet:          eth0: negotiated 100baseTx-FD, link ok
 
−
ibiruni:        eth0: negotiated 100baseTx-FD flow-control, link ok
 
−
ibiruni:        eth1: 10 Mbit, half duplex, no link
 
−
irose:          eth0: negotiated 100baseTx-FD, link ok
 
−
irose:          eth1: 10 Mbit, half duplex, no link
 
−
ismellie:      eth0: negotiated 100baseTx-FD, link ok
 
−
ismellie:      eth1: 10 Mbit, half duplex, no link
 
−
ianthony:      eth0: negotiated 100baseTx-FD, link ok
 
−
ianthony:      eth1: 10 Mbit, half duplex, no link
 
−
ennael:        eth0: negotiated 100baseTx-FD flow-control, link ok
 
−
ennael:        eth1: no link
 
−
chloe:          eth0: negotiated 100baseTx-FD flow-control, link ok
 
−
chloe:          eth1: no link
 
−
bleuenn:        eth0: negotiated 100baseTx-FD flow-control, link ok
 
−
bleuenn:        eth1: no link
 
−
</pre>
 
−
 
−
===IP addresses===
 
−
<pre>
 
−
[root@zwinger node_groups]# dsh -N working 'ifconfig | grep "inet addr" | grep -v 127.0.0.1'
 
−
executing 'ifconfig | grep "inet addr" | grep -v 127.0.0.1'
 
−
albert:                  inet addr:207.142.131.201  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
albert:                  inet addr:10.0.0.5  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
alrazi:                  inet addr:10.0.0.10  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
ariel:                    inet addr:207.142.131.244  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
ariel:                    inet addr:10.0.0.2  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
avicenna:                inet addr:207.142.131.249  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
avicenna:                inet addr:10.0.0.14  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
bacon:                    inet addr:207.142.131.200  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
bacon:                    inet addr:10.0.0.3  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
bart:                    inet addr:207.142.131.227  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
bayle:                    inet addr:207.142.131.228  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
browne:                  inet addr:207.142.131.229  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
dalembert:                inet addr:207.142.131.194  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
dalembert:                inet addr:10.0.0.13  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
diderot:                  inet addr:207.142.131.250  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
diderot:                  inet addr:10.0.0.7  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
friedrich:                inet addr:10.0.0.11  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
goeje:                    inet addr:207.142.131.198  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
goeje:                    inet addr:10.0.0.9  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
harris:                  inet addr:207.142.131.199  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
harris:                  inet addr:10.0.0.8  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
suda:                    inet addr:207.142.131.226  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
suda:                    inet addr:10.0.0.1  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
tingxi:                  inet addr:10.0.0.12  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
will:                    inet addr:207.142.131.243  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
will:                    inet addr:207.142.131.235  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.245  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.248  Bcast:207.142.131.255  Mask:255.255.255.255
 
−
will:                    inet addr:207.142.131.205  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.247  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.246  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.202  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.203  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.204  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.206  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:207.142.131.236  Bcast:207.142.131.255  Mask:255.255.255.0
 
−
will:                    inet addr:10.0.0.21  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
zwinger:                  inet addr:207.142.131.234  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
zwinger:                  inet addr:207.142.131.207  Bcast:207.142.131.255  Mask:255.255.255.255
 
−
zwinger:                  inet addr:207.142.131.216  Bcast:207.142.131.255  Mask:255.255.255.255
 
−
zwinger:                  inet addr:10.0.0.4  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
hypatia:                  inet addr:207.142.131.212  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
hypatia:                  inet addr:10.0.0.212  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
humboldt:                inet addr:207.142.131.213  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
humboldt:                inet addr:10.0.0.213  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
kluge:                    inet addr:10.0.0.214  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
khaldun:                  inet addr:207.142.131.218  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
khaldun:                  inet addr:10.0.0.218  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
larousse:                inet addr:207.142.131.208  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
larousse:                inet addr:10.0.0.22  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
larousse:                inet addr:10.10.0.1  P-t-P:10.10.0.2  Mask:255.255.255.255
 
−
webster:                  inet addr:207.142.131.221  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
webster:                  inet addr:10.0.0.23  Bcast:10.0.0.255  Mask:255.255.255.0
 
−
holbach:                  inet addr:207.142.131.224  Bcast:207.142.131.255  Mask:255.255.255.192
 
−
holbach:                  inet addr:10.0.0.24  Bcast:10.0.0.255  Mask:255.255.255.0
 
−
benet:                    inet addr:10.0.0.29  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
ibiruni:                  inet addr:10.0.0.25  Bcast:10.255.255.255  Mask:255.0.0.0
 
−
irose:                    inet addr:10.0.0.26  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
ismellie:                inet addr:10.0.0.27  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
ianthony:                inet addr:10.0.0.28  Bcast:10.0.255.255  Mask:255.255.0.0
 
−
ennael:                  inet addr:212.85.150.131  Bcast:212.85.150.135  Mask:255.255.255.248
 
−
ennael:                  inet addr:192.168.0.30  Bcast:192.168.0.255  Mask:255.255.255.0
 
−
chloe:                    inet addr:212.85.150.132  Bcast:212.85.150.135  Mask:255.255.255.248
 
−
chloe:                    inet addr:192.168.0.10  Bcast:192.168.0.255  Mask:255.255.255.0
 
−
chloe:                    inet addr:10.10.0.2  P-t-P:10.10.0.1  Mask:255.255.255.255
 
−
bleuenn:                  inet addr:212.85.150.133  Bcast:212.85.150.135  Mask:255.255.255.248
 
−
bleuenn:                  inet addr:192.168.0.20  Bcast:192.168.0.255  Mask:255.255.255.0
 
−
</pre>
 
  
  
Line 202: Line 17:
 
Wikimedia owns a Cisco C3560G-48TS switch, and three Netgear Gigabit [[switches]].
 
Wikimedia owns a Cisco C3560G-48TS switch, and three Netgear Gigabit [[switches]].
  
−
http://noc.wikimedia.org/~kate/network-design2.png
 
 
[[Category:Network]]
 
[[Category:Network]]

Revision as of 15:08, 18 May 2005

The purpose of this page is to give an overview of the current design of the network of the Wikimedia servers, and to provide a place to develop a new and improved network scheme.


Overall system design

The folowing is the general system design plan which the network layer must efficiently accommodate.

  • Databases in a central pool with each serving a subset of the wikis, so each has high cache efficiency and the total number needed to handle any query load is minimised. Database servers cost US$5,000-$8,000 each, depending on exact equipment.
  • A central pair of old text database servers (part of the long term storage growth plan for the databases, to move this high volume and seldom accessed data off costly and comparatively small disk systems).
  • Memcached caching spread on apaches across the whole cluster, producing one very large cache pool, accessible from any apache and stored on half or more of the apaches. Segmenting the pool would decrease the overall hit rate, increasing the number of apaches and database servers required for any given system load level.
  • Load balancing of squids and apaches, currently expected to use two or three systems between the internet and the squids and the same set between the squids and the apaches.

A key network systems design requirement is efficient access from any apache to any apache running memcached (expected to be more than half of all apaches) and efficient access from any apache to any database server. Losing this capability would dramatically increase overall system cost.

Current situation

Wikimedia servers reside in two racks along with Bomis servers, hosted at Candidhosting. Wikimedia/Bomis have a dedicated IP range, 207.142.131.192/26. There are two gateways: 207.142.131.193 and 207.142.131.225. Total burstable bandwidth is 1000 Mbit/s, delivered through one optic fiber 1000base-SX link.

Wikimedia owns a Cisco C3560G-48TS switch, and three Netgear Gigabit switches.

Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Network_design&oldid=2335"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox