9.3 Current Architectural View of the Internet
9.3.1 Customers and Providers, Peering and Tiering, and Exchange Points
In the world of Internet routing connectivity, the term customer typically refers to an orga- nization that has an IP address block; it relies on a provider for Internet connectivity; note that owning an AS number is not necessary since you can have an address block and be a part of an existing AS number. For ease of discussion here, we will restrict to those cus- tomers that have their own AS numbers. The customer/provider relation is hierarchical and is sometimes described also as downstream ISP/upstream ISP relation. At the top of the hi- erarchy is tier 1 Internet service providers (tier 1 ISPs). Each tier 1 ISP has its own AS num- ber. It is certainly possible to have more than one AS number belong to an ISP, for exam- ple, due to the merger of companies. For simplicity, we will assume that each ISP has its own unique AS number. A tier 1 ISP provides a large network spanning a geographic re- gion such as the entire country, and sometimes across countries; such networks are often referred to as Internet backbone networks where link speeds can be in the order of 10 Gbps with the most advanced routers deployed. All tier 1 ISPs are at the same peering level.
Typically, tier 1 ISPs peer privately with each other at one or more points. It used to be the case that tier 1 ISPs meet at network access points (NAPs) to exchange traffic from one network to another. In Figure 9.6, we show a generic example with four tier 1 ISPs meet- ing at an NAP; note that this is not common any more, it is shown here for illustration only. It may be noted that NAPs are also known as Internet exchange points (XP, or, EP in short), or Metropolitan Area Exchanges (MAEs). Furthermore, such arrangements are known as public peering since they are neutral meeting points. First, it should be noted that ex- change points are operated by neutral entities that play the role of providers for traffic ex- change services to tier 1 ISP customers. During transition from the NSFNET, the notion of NAPs was conceived when it became clear that one core network would not be the carrier for all Internet traffic. Initially, there were four NAPs that were connected to the NSFNET during 1994–1995. Currently, there are more than 175 exchange points around the world.
There is also private peering between two tiernISPs where they connect directly to each other and agree to exchange traffic with each other; this then can serve as a bypass from congested exchange points, which some ISPs prefer. In Figure 9.6, we show that two tiern
F I G U R E 9.6 ISP connectivity through public peering at an exchange point and through private peering (left: used to be more common among tier 1 ISPs; right: now seen more commonly at other tiering (“tier-N”) levels).
ISPs are directly connected to each other through private peering while they are also part of the common exchange points with two other ISPs. For example, this would be a scenario where two tier n ISPs that have private peering as well as public peering would use the AS-path count to choose the private peering as the better path since they can use the exchange point as another AS in the path length between them. It may be noted that private network exchange points are also possible.
Exchange points provide physical connectivity to customers using technologies such as Gigabit Ethernet, ATM, and SONET, where customers’ routers for connectivity are collocated in the same physical facility. Mostly, exchange point provides a meeting place for layer 2 con- nectivity. Layer 2 connectivity can give the impression that a simple Ethernet environment with every ISP’s router attached to this Ethernet facility is probably sufficient. The difficulty is that the sheer volume of traffic each ISP generates is so high that such a simple environ- ment is not possible in practice. Thus, you see a combination of sophisticated technologies with functionalities for peer management at most of the exchange points. In any case, at an exchange point, each ISP’s BGP speaker can set up a BGP session to all other ISPs that have a BGP speaker collocated. In recent years, some exchange points have become popular for content delivery network providers since they can be directly connected to various major ISPs.
It is important to note that exchanges points have fairly well-defined policies while such policies can vary from one exchange point to another and certainly can evolve over time.
Some examples of policies are: (1) an ISP must have its own AS number and use BGP to become a member of a exchange point, (2) the exchange point cannot be used for transit, (3) the exchange point policy requires full peering among all parties, or, each ISP can choose a different policy from a set of acceptable policies. Depending on policy, some large ISPs might or might not want to joint an exchange point; for example, if some large ISPs do not want to peer with smaller ISPs, they might not join an exchange point that stipulates that they must peer with all parties. In some instances, ISPs of different tiers, including tier 1 ISPs, do meet at large exchange points that may not require that all parties must peer with everyone.
In such cases, each ISP has the option of not peering with everyone that is a member of this exchange point. Currently, Amsterdam Internet Exchange (AMS-IX) [14], considered the largest exchange point, has a flexible policy; it lets providers of different size to connect to its exchange point allowing each provider to set their own peering restrictions, including allowing private interconnects between two members.
In essence, an exchange is a giant traffic switching point. Some of the large exchange points push traffic in the order of 135 Gbps. It is not hard to imagine that such a high data push requirement can be taxing even with the top of the line inter-connecting hardware; in fact, this is no longer possible to do on a single hardware device. Thus, such exchange points must set up their own internal topology in such a way that multiple hardware devices are used for efficient traffic flow.
Now we move to consider multiple tiers. Tier 1 ISPs, in turn, provide connectivity to tier 2 ISPs; thus, in this case tier 2 ISPs are the customers and tier 1 ISPs are the providers. Tier 2 ISPs use tier 1 ISPs for transit service, but may peer with other tier 2 ISPs as well, for example, either through regional exchange points or private peering. Typically, tier 2 ISPs do not have international coverage—they are either at regional or national levels. It may be noted that a tier 1 ISP provides transit service to many tier 2 providers at certain meeting points; these
meeting points are commonly referred to as Points of Presences (PoPs). We will discuss PoPs more later in Section 9.6.
Tier 3 ISPs are the ones that seek transit service only from either tier 2 or tier 1 providers;
they are typically not involved in public peering, although they may do some private peering.
At the same time, tier 3 providers usually do not provide direct internet connectivity to users.
Beyond tier 3 ISPs, it becomes a bit murky in regard to the role of lower tiers or how many more tiers there are. To limit our discussion, we will stop at tier 4 ISPs by assuming that they provide local access to the Internet for institutions, enterprises, and residential users. Note that tier 4 ISPs require transit connectivity from tier 3 providers.
Although we have discussed various tier levels, there is no clear rule that indicates who is or is not a certain tier provider. Certainly, this is more clear in the case of a tier 1 ISP.
However, consider content delivery providers who want to be located close to tier 1 ISPs’
peering points. They usually have two options: (1) have their web servers hosted directly on one of the tier 1 ISPs; in this case, no AS number is necessary, or (2) have their series of web servers connected through routers to form a network with their own AS number, and then have peering with every major provider at major peering points or through private peering.
If they choose option 2, they do not exactly fall into one of the tiering providers—we label them as content delivery service (CDS) ISPs. Examples of CDS ISPs are Google, Yahoo!, and Akamai.
There is also some difference in peering arrangements which varies from one country to another. For example, private peering at tier 1 level is now common in US, while public peering in other countries often includes some tier 1 providers. The largest public peering point now is considered to be Amsterdam Internet Exchange, AMS-IX [14]. As of this writing, AMS-IX has about 250 members which includes some large tier 1 ISPs as well; the peak rate is 150 Gbps. London Internet exchange, LINX [419] has over 200 members with peak traffic of 130 Gbps and Japan Internet exchange, JPIX [338] has 100 members with the peak rate at approximately 65 Gbps.
Since the Internet is made up of many providers with different relations and tiers, the obvious question is: what possible traffic exchange and payment relation do ISPs agree on?
Here are a representative set of possible options [692]:
• Multilateral agreement: Several ISPs build/use shared facilities and share cost; for ex- ample, this agreement can be possible with public exchange points or private exchange points.
• Bilateral agreement: Two providers agree to exchange traffic if traffic is almost symmetric, or agree on a price, taking into account the imbalance in traffic swapped; for example, in a private peering setting.
• Unilateral agreement for transit: A customer pays its provider an “access” charge for car- rying traffic; for example, a tier 4 ISP would pay a charge to tier 3 ISP.
• Sender Keeps All (SKA): ISPs do not track or charge for traffic exchange; this is possible in private peering and in some public peering.
Along with agreements, especially the ones that involve payment, it is common to also write up service level agreements (SLAs). SLAs refer to an agreement on performance that is
to be met on a course time scale; for example, the average delay between entry and exit points not to exceed 20 millisec, when averaged over a certain time period such as a week or a month. Typically, SLAs do not include performance requirement on a short time window such as in seconds. Thus, SLAs can be thought of more as a coarse grain quality-of-service requirement than a fine grain quality-of-service requirement. Furthermore, SLAs may also cover issues such as demarcation points; this refers to the line that indicates who manages what on a day-to-day basis. When a customer connects to a provider, there are three points involved: the routers at each end (one for the customer and the other for the provider), and the physical connectivity that connects them, such as a physical wire or a layer-2 connectivity.
In some cases, the demarcation point is where the customer connects to a layer-2 switch in the physical connectivity part; in other cases, the customer’s router is completely located at the provider’s site; and yet in other cases, the provider’s access router may be physically at the customer’s site. Sometimes suitability of a demarcation point can be a factor for a customer in deciding which one to choose as a provider.