|
|
| (29 intermediate revisions by 12 users not shown) |
| Line 1: |
Line 1: |
| − | The purpose of this page is to give an overview of the current '''design of the network''' of the Wikimedia servers, and to provide a place to develop a new and improved network scheme. | + | == AS 14907 == |
| | + | The US network. |
| | | | |
| − | ==Automatically generated information== | + | === 2011 === |
| | + | [[File:Eqiad logical.png|thumb|400px|AS14907 Eqiad in 2011]] |
| | + | [[File:Wikimedia pmtpa management network.png|thumb|400px|AS14907 in 2010]] |
| | | | |
| − | Generated 2005-02-24
| + | === Subnets === |
| | | | |
| − | ===Default gateway=== | + | ==== [[pmtpa]] ==== |
| | | | |
| − | <pre>
| + | ==== [[eqiad]] ==== |
| − | [root@zwinger node_groups]# dsh -N working "route | grep default" | + | {| class="wikitable" |
| − | executing 'route | grep default'
| + | |- |
| − | albert: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | !subnet !! vlan ID !! IPv4 !! IPv6 |
| − | ariel: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0
| + | |- |
| − | avicenna: default izwinger 0.0.0.0 UG 0 0 0 eth1
| + | | public1-a-eqiad || 1001 || 208.80.154.0/26 || 2620:0:861:1::/64 |
| − | bacon: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | |- |
| − | bart: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | | public1-b-eqiad || 1002 || 208.80.154.128/26 || 2620:0:861:2::/64 |
| − | bayle: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | |- |
| − | browne: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | | public1-c-eqiad || 1003 || || 2620:0:861:3::/64 |
| − | dalembert: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | |- |
| − | diderot: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0
| + | | public1-d-eqiad || 1004 || || 2620:0:861:4::/64 |
| − | friedrich: default izwinger 0.0.0.0 UG 0 0 0 eth1
| + | |- |
| − | goeje: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | | private1-a-eqiad || 1017 || 10.64.0.0/22 || 2620:0:861:101::/64 |
| − | harris: default izwinger 0.0.0.0 UG 0 0 0 eth1
| + | |- |
| − | suda: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0
| + | | private1-b-eqiad || 1018 || 10.64.16.0/22 || 2620:0:861:102::/64 |
| − | tingxi: default izwinger 0.0.0.0 UG 0 0 0 eth0
| + | |- |
| − | will: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0
| + | | private1-c-eqiad || 1019 || 10.64.32.0/22 || 2620:0:861:103::/64 |
| − | zwinger: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0
| + | |- |
| − | hypatia: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | | private1-d-eqiad || 1020 || 10.64.48.0/22 || 2620:0:861:104::/64 |
| − | humboldt: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | |} |
| − | kluge: default izwinger 0.0.0.0 UG 0 0 0 eth1
| + | |
| − | khaldun: default 207.142.131.193 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | larousse: default 207.142.131.225 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | webster: default izwinger 0.0.0.0 UG 0 0 0 eth1
| + | |
| − | holbach: default izwinger 0.0.0.0 UG 0 0 0 eth1
| + | |
| − | benet: default izwinger 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | ibiruni: default 10.255.255.254 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | irose: default izwinger 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | ismellie: default izwinger 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | ianthony: default izwinger 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | ennael: default router-wikipedi 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | chloe: default router-wikipedi 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | bleuenn: default router-wikipedi 0.0.0.0 UG 0 0 0 eth0
| + | |
| − | </pre>
| + | |
| | | | |
| − | ===Cables connected===
| |
| − | <pre>
| |
| − | [root@zwinger node_groups]# dsh -N working mii-tool
| |
| − | executing 'mii-tool'
| |
| − | albert: SIOCGMIIPHY on 'eth0' failed: Operation not supported
| |
| − | albert: SIOCGMIIPHY on 'eth1' failed: Operation not supported
| |
| − | albert: no MII interfaces found
| |
| − | alrazi: eth0: no link
| |
| − | alrazi: eth1: negotiated 100baseTx-FD, link ok
| |
| − | ariel: eth0: negotiated 100baseTx-FD, link ok
| |
| − | ariel: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | avicenna: eth0: no link
| |
| − | avicenna: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | bacon: eth0: negotiated 100baseTx-FD, link ok
| |
| − | bacon: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | bart: eth0: negotiated 100baseTx-FD, link ok
| |
| − | bart: eth1: no link
| |
| − | bayle: eth0: negotiated 100baseTx-FD, link ok
| |
| − | bayle: eth1: no link
| |
| − | browne: eth0: negotiated 100baseTx-FD, link ok
| |
| − | browne: eth1: no link
| |
| − | dalembert: eth0: no link
| |
| − | dalembert: eth1: negotiated 100baseTx-FD, link ok
| |
| − | diderot: eth0: no link
| |
| − | diderot: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | friedrich: eth0: no link
| |
| − | friedrich: eth1: negotiated 100baseTx-FD, link ok
| |
| − | goeje: eth0: no link
| |
| − | goeje: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | harris: eth0: no link
| |
| − | harris: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | suda: eth0: negotiated 100baseTx-FD, link ok
| |
| − | suda: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | tingxi: eth0: negotiated 100baseTx-FD flow-control, link ok
| |
| − | tingxi: eth1: no link
| |
| − | will: eth0: negotiated 100baseTx-FD, link ok
| |
| − | will: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | zwinger: eth0: negotiated 100baseTx-FD, link ok
| |
| − | zwinger: eth1: negotiated 100baseTx-FD, link ok
| |
| − | hypatia: eth0: no link
| |
| − | hypatia: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | humboldt: eth0: no link
| |
| − | humboldt: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | kluge: eth0: no link
| |
| − | kluge: eth1: negotiated 100baseTx-FD, link ok
| |
| − | khaldun: eth0: no link
| |
| − | khaldun: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | larousse: eth0: negotiated 100baseTx-FD, link ok
| |
| − | larousse: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | webster: eth0: negotiated 100baseTx-FD, link ok
| |
| − | webster: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | holbach: eth0: negotiated 100baseTx-FD, link ok
| |
| − | holbach: eth1: negotiated 100baseTx-FD flow-control, link ok
| |
| − | benet: eth0: negotiated 100baseTx-FD, link ok
| |
| − | ibiruni: eth0: negotiated 100baseTx-FD flow-control, link ok
| |
| − | ibiruni: eth1: 10 Mbit, half duplex, no link
| |
| − | irose: eth0: negotiated 100baseTx-FD, link ok
| |
| − | irose: eth1: 10 Mbit, half duplex, no link
| |
| − | ismellie: eth0: negotiated 100baseTx-FD, link ok
| |
| − | ismellie: eth1: 10 Mbit, half duplex, no link
| |
| − | ianthony: eth0: negotiated 100baseTx-FD, link ok
| |
| − | ianthony: eth1: 10 Mbit, half duplex, no link
| |
| − | ennael: eth0: negotiated 100baseTx-FD flow-control, link ok
| |
| − | ennael: eth1: no link
| |
| − | chloe: eth0: negotiated 100baseTx-FD flow-control, link ok
| |
| − | chloe: eth1: no link
| |
| − | bleuenn: eth0: negotiated 100baseTx-FD flow-control, link ok
| |
| − | bleuenn: eth1: no link
| |
| − | </pre>
| |
| | | | |
| − | ===IP addresses=== | + | == AS 43821 == |
| − | <pre>
| + | The European network. |
| − | [root@zwinger node_groups]# dsh -N working 'ifconfig | grep "inet addr" | grep -v 127.0.0.1'
| + | |
| − | executing 'ifconfig | grep "inet addr" | grep -v 127.0.0.1'
| + | |
| − | albert: inet addr:207.142.131.201 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | albert: inet addr:10.0.0.5 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | alrazi: inet addr:10.0.0.10 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | ariel: inet addr:207.142.131.244 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | ariel: inet addr:10.0.0.2 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | avicenna: inet addr:207.142.131.249 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | avicenna: inet addr:10.0.0.14 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | bacon: inet addr:207.142.131.200 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | bacon: inet addr:10.0.0.3 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | bart: inet addr:207.142.131.227 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | bayle: inet addr:207.142.131.228 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | browne: inet addr:207.142.131.229 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | dalembert: inet addr:207.142.131.194 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | dalembert: inet addr:10.0.0.13 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | diderot: inet addr:207.142.131.250 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | diderot: inet addr:10.0.0.7 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | friedrich: inet addr:10.0.0.11 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | goeje: inet addr:207.142.131.198 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | goeje: inet addr:10.0.0.9 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | harris: inet addr:207.142.131.199 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | harris: inet addr:10.0.0.8 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | suda: inet addr:207.142.131.226 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | suda: inet addr:10.0.0.1 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | tingxi: inet addr:10.0.0.12 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | will: inet addr:207.142.131.243 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | will: inet addr:207.142.131.235 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.245 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.248 Bcast:207.142.131.255 Mask:255.255.255.255
| + | |
| − | will: inet addr:207.142.131.205 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.247 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.246 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.202 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.203 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.204 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.206 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:207.142.131.236 Bcast:207.142.131.255 Mask:255.255.255.0
| + | |
| − | will: inet addr:10.0.0.21 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | zwinger: inet addr:207.142.131.234 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | zwinger: inet addr:207.142.131.207 Bcast:207.142.131.255 Mask:255.255.255.255
| + | |
| − | zwinger: inet addr:207.142.131.216 Bcast:207.142.131.255 Mask:255.255.255.255
| + | |
| − | zwinger: inet addr:10.0.0.4 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | hypatia: inet addr:207.142.131.212 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | hypatia: inet addr:10.0.0.212 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | humboldt: inet addr:207.142.131.213 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | humboldt: inet addr:10.0.0.213 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | kluge: inet addr:10.0.0.214 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | khaldun: inet addr:207.142.131.218 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | khaldun: inet addr:10.0.0.218 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | larousse: inet addr:207.142.131.208 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | larousse: inet addr:10.0.0.22 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | larousse: inet addr:10.10.0.1 P-t-P:10.10.0.2 Mask:255.255.255.255
| + | |
| − | webster: inet addr:207.142.131.221 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | webster: inet addr:10.0.0.23 Bcast:10.0.0.255 Mask:255.255.255.0
| + | |
| − | holbach: inet addr:207.142.131.224 Bcast:207.142.131.255 Mask:255.255.255.192
| + | |
| − | holbach: inet addr:10.0.0.24 Bcast:10.0.0.255 Mask:255.255.255.0
| + | |
| − | benet: inet addr:10.0.0.29 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | ibiruni: inet addr:10.0.0.25 Bcast:10.255.255.255 Mask:255.0.0.0
| + | |
| − | irose: inet addr:10.0.0.26 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | ismellie: inet addr:10.0.0.27 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | ianthony: inet addr:10.0.0.28 Bcast:10.0.255.255 Mask:255.255.0.0
| + | |
| − | ennael: inet addr:212.85.150.131 Bcast:212.85.150.135 Mask:255.255.255.248
| + | |
| − | ennael: inet addr:192.168.0.30 Bcast:192.168.0.255 Mask:255.255.255.0
| + | |
| − | chloe: inet addr:212.85.150.132 Bcast:212.85.150.135 Mask:255.255.255.248
| + | |
| − | chloe: inet addr:192.168.0.10 Bcast:192.168.0.255 Mask:255.255.255.0
| + | |
| − | chloe: inet addr:10.10.0.2 P-t-P:10.10.0.1 Mask:255.255.255.255
| + | |
| − | bleuenn: inet addr:212.85.150.133 Bcast:212.85.150.135 Mask:255.255.255.248
| + | |
| − | bleuenn: inet addr:192.168.0.20 Bcast:192.168.0.255 Mask:255.255.255.0
| + | |
| − | </pre>
| + | |
| | | | |
| | + | === 2008 === |
| | + | [[File:Knams-multihomed.png|thumb|400px|AS43821 in 2008]] |
| | | | |
| − | ==Overall system design==
| + | BGP default transit from AS1145 (Kennisnet), with some partial transit and peering over a 1 Gbps AMS-IX link. Everything on one core router/switch, csw1-knams (Foundry BigIron RX-8). |
| − | The folowing is the general system design plan which the network layer must efficiently accommodate.
| + | |
| | | | |
| − | *Databases in a central pool with each serving a subset of the wikis, so each has high cache efficiency and the total number needed to handle any query load is minimised. Database servers cost US$5,000-$8,000 each, depending on exact equipment.
| + | === 2009 === |
| − | *A central pair of old text database servers (part of the long term storage growth plan for the databases, to move this high volume and seldom accessed data off costly and comparatively small disk systems).
| + | [[File:AS43821 2009.png|thumb|400px|AS43821 in 2009]] |
| − | *Memcached caching spread on apaches across the whole cluster, producing one very large cache pool, accessible from any apache and stored on half or more of the apaches. Segmenting the pool would decrease the overall hit rate, increasing the number of apaches and database servers required for any given system load level.
| + | |
| − | *Load balancing of squids and apaches, currently expected to use two or three systems between the internet and the squids and the same set between the squids and the apaches.
| + | |
| | | | |
| − | A key network systems design requirement is efficient access from any apache to any apache running memcached (expected to be more than half of all apaches) and efficient access from any apache to any database server. Losing this capability would dramatically increase overall system cost.
| + | Temporary situation after the move from knams to esams. The network is split, with a new Foundry BigIron RX-4 as a pure router at knams for external connectivity, with Telia, DataHop, Init7 (partial) transit, and 2x 1 Gbps AMS-IX for peering. Connectivity between the two sites is supplied by a 10GBase-ER link over dark fiber, and a 3 Gbps MPLS backup link. A second dark fiber is being installed to form a ring. |
| | | | |
| − | == Current situation ==
| |
| − | Wikimedia servers reside in two racks along with Bomis servers, hosted at [http://www.candidhosting.com Candidhosting]. Wikimedia/Bomis have a dedicated IP range, <tt>207.142.131.192/26</tt>. There are two gateways: <tt>207.142.131.193</tt> and <tt>207.142.131.225</tt>. Total burstable bandwidth is 1000 Mbit/s, delivered through one optic fiber 1000base-SX link.
| |
| | | | |
| − | Wikimedia owns a Cisco 3560G-48TX switch, and three Netgear Gigabit [[switches]].
| + | === 2010 === |
| | + | [[File:AS43821 Q3 2010.png|thumb|400px|AS43821 late 2010]] |
| | + | |
| | + | The purchase of several Juniper EX4200s in a stack, for extra access ports for servers, also brings some opportunities w.r.t. the network topology. Since the EX4200s have excellent L3 support they can help create redundancy. |
| | + | |
| | + | The 2nd dark fiber is linked between [[br1-knams]] and [[csw2-esams]] to create a ring. [[csw1-esams]] and [[csw2-esams]] can then share responsibility as core switches, for inter-vlan routing and switching, using VRRP. Since an EX4200 can not install a full BGP routing table in FIB, it defaults to either of the two Foundry routers using OSPF. |
| | + | |
| | + | Toolserver can be connected redundantly as well, using (R)STP to both core switches and VRRP, or alternatively a LAG to the EX4200 stack. |
| | + | |
| | + | == Configuration guidelines == |
| | + | * Firewall filters, policies, prefix lists etc that are specific to a certain protocol family (e.g. only IPv4, or only IPv6) should have a '4' or '6' appended to their name. Filters, policies and prefix lists that are protocol family agnostic, should lack this suffix. |
| | + | |
| | + | == See also == |
| | + | * [[Multicast]] |
| | + | * [[TCP Tuning]] |
| | | | |
| − | http://noc.wikimedia.org/~kate/network-design2.png
| |
| | [[Category:Network]] | | [[Category:Network]] |
| | + | [[Category:knams cluster| *]] |
| | + | [[Category:Pmtpa cluster| *]] |
BGP default transit from AS1145 (Kennisnet), with some partial transit and peering over a 1 Gbps AMS-IX link. Everything on one core router/switch, csw1-knams (Foundry BigIron RX-8).
Temporary situation after the move from knams to esams. The network is split, with a new Foundry BigIron RX-4 as a pure router at knams for external connectivity, with Telia, DataHop, Init7 (partial) transit, and 2x 1 Gbps AMS-IX for peering. Connectivity between the two sites is supplied by a 10GBase-ER link over dark fiber, and a 3 Gbps MPLS backup link. A second dark fiber is being installed to form a ring.
The purchase of several Juniper EX4200s in a stack, for extra access ports for servers, also brings some opportunities w.r.t. the network topology. Since the EX4200s have excellent L3 support they can help create redundancy.
Toolserver can be connected redundantly as well, using (R)STP to both core switches and VRRP, or alternatively a LAG to the EX4200 stack.
