Network design
The purpose of this page is to give an overview of the current design of the network of the Wikimedia servers, and to provide a place to develop a new and improved network scheme.
Contents |
Current situation
Wikimedia servers reside in two racks along with Bomis servers, hosted at Candidhosting. Wikimedia/Bomis have a dedicated IP range, 207.142.131.192/26. There are two gateways: 207.142.131.193 and 207.142.131.225. Total burstable bandwidth is 200 Mbit/s, delivered through two separate 100BaseTx uplinks, connected from the same broadcast domain that is shared with other customers.
Wikimedia owns three switches. As the two uplinks are not allowed to create a loop, they must be connected to different switches that are not connected to eachother (when not using STP), which is not an ideal situation. A third switch is currently used to connect internal servers, that don't have public IPs and should not be accessible from the Internet. The IP range used for this internal network is 10.0.0.0/8.
Problems
The current network setup is not optimal in many ways, as will be described here.
Multiple uplinks
Recently, Wikimedia traffic spiked to 100Mbit/s multiple times, which is the limit of a single 100BaseTx connection. Also, average outging traffic at this moment is about 45 Mbit/s, so it is clear that Wikimedia was slowly becoming network limited. However, the colo provider charges $400 dollar per month just to provide us with a Gigabit uplink, unless we commit to 60 Mbit/s average traffic or higher. Instead, they decided to give us a second 100BaseTx for free.
This does pose some problems though. Because the two uplinks are connected from the same broadcast domain, we cannot connect them internally, or we would create a loop. One solution to this problem is to connect the uplinks to different switches that are not connected, but this means that hosts on the two different switches can only exchange traffic between eachother through the uplinks. This traffic is graphed and billed twice, and is a bottleneck, as it has to traverse both relatively slow uplinks.
