Day 3: Conference: Globally Distributed Content

Simon "Horms" Hormans, from VA Linux Systems, introduced his new supersparrow work.


Horms explained that http mirrors where users have to select their mirror aren't ideal (it's not always obvious for users which copy is the closest and fastest internet-wise), and that the solution is global load balancing.

Supersparrow is based on BGP and looks at a BGP table to get a view of where the requesting host is located and how far it is from the server.
One issue is that you have to determine who's going to answer the client rather quickly because you don't want the latency to be too high, or worse, for the client to time out.

Ping is not the right tool to find out how quickly we can send data to a client because it shows latency and not bandwidth (satellite is one example)
Looking at the domain name of the incoming request isn't great, but it works somewhat. One problem is that you have to reverse the IP address which can hang and many IPs do not reverse anymore (whoever sells IP connectivity without providing valid reverse DNS records for your IP addresses does not deserve your business or anyone else's BTW)

Another idea is to look at the routing information provided by BGP
Once you find out what the best remote site is, two ways to get clients to connect to it directly are DNS and http redirect.
For DNS, you can have a DNS server that gives out a different answer every time it's queried. It looks at the IP of the querying name server and assumes the remote client is sufficiently close to that name server and hands out the IP address that would be the closest to the querying name server.

Two other options are to use HTTP redirects, but this shows a www1 server name in the URL which isn't nice, so it's better to hit the main HTML page from the master server and have links to pictures and other high bandwidth content to other closer mirror servers (that's what Akamai does BTW)

Supersparrow talks to a BGP capable host, either through Zebra on a unix host or to a Cisco. Other routers will be added later.
For DNS, Supersparrow is based on dense, another DNS server that is easier to talk to than bind.
For http redirects, Supersparrow comes with a library, libsupersparrow, which can be plugged in Apache's mod_rewrite.

You can find Horms' supersparrow here and Simon's paper with the slide contents here

[library] Picture library [back] Back to Main Page [next] Next page

[ms free site] Email
Link to Home Page

2001/01/28 (15:12): Version 1.0