Tier-1 Internet Service Provider Research Articles

Exploring Network-Wide Flow Data With Flowyager

Many network operations, ranging from attack investigation and mitigation to traffic management, require answering network-wide flow queries in seconds. Although flow records are collected at each router, using available traffic capture utilities, querying the resulting datasets from hundreds of routers across sites and over time, remains a significant challenge due to the sheer traffic volume and distributed nature of flow records. In this paper, we investigate how to improve the response time for a priori unknown network-wide queries. We present Flowyager, a system that is built on top of existing traffic capture utilities. Flowyager generates and analyzes tree data structures, that we call Flowtrees, which are succinct summaries of the raw flow data available by capture utilities. Flowtrees are self-adjusted data structures that drastically reduce space and transfer requirements, by 75% to 95%, compared to raw flow records. Flowyager manages the storage and transfers of Flowtrees, supports Flowtree operators, and provides a structured query language for answering flow queries across sites and time periods. By deploying a Flowyager prototype at both a large Internet Exchange Point and a Tier-1 Internet Service Provider, we showcase its capabilities for networks with hundreds of router interfaces. Our results show that the query response time can be reduced by an order of magnitude when compared with alternative data analytics platforms. Thus, Flowyager enables interactive network-wide queries and offers unprecedented drill-down capabilities to, e.g., identify DDoS culprits, pinpoint the involved sites, and determine the length of the attack.

Efficient FIB caching using minimal non-overlapping prefixes

The size of the global Routing Information Base (RIB) has been increasing at an alarming rate. As a direct effect, the size of the global Forwarding Information Base (FIB) has experienced rapid growth. This increase raises serious concerns for Internet Service Providers (ISPs) as the FIB memory in line cards is much more expensive than regular memory modules, so frequently increasing this memory capacity for all the routers is prohibitively costly to an ISP. Previous research on Internet traffic indicates that a very small number of popular prefixes receive most of the Internet’s traffic, making “caching” a possible solution to reduce the FIB size. However, FIB caching may cause a cache-hiding problem where a packet’s longest-prefix match in the cache differs from that in the full FIB, and thus the packet will be forwarded to the wrong next hop. Motivated by these observations, we propose an efficient FIB caching scheme that stores only non-overlapping FIB entries into the fast memory (i.e., a FIB cache), while storing the complete FIB in slow memory. Our caching scheme achieves a considerably higher hit ratio than previous approaches while preventing the cache-hiding problem. It can also handle cache misses, cache replacement, and routing updates efficiently. Moreover, we have implemented the proposed caching scheme using the OpenFlow platform, which allows a local or remote route controller to manage routes in the cache. We use real traffic of a regional ISP and a Tier1 ISP to carry out our experiments. Our simulation results show that with only 20K prefixes in the cache (5.28% of the actual FIB size), the hit ratio of our scheme is higher than 99.95%. Our OpenFlow implementation achieves a hit ratio near 99.94%, which approaches the performance of the simulated results.

Line speed accurate superspreader identification using dynamic error compensation

A superspreader is a source that connects with a large number of distinct destinations during a measuring interval. Identifying superspreaders provides very important information for many other applications, such as port scanner and Distributed Denial of Service (DDoS) attack detection, worm propagation measurement, and hot spots localization in peer-to-peer (p2p) and Content Delivery Networks (CDN). There are several challenges remaining in the area of superspreader identification, especially for high speed networks, exhibited by the trade-offs among three critical requirements, which are (1) fast data processing speed, (2) low memory space, and (3) high accuracy. In this paper, we propose a new Superspreader Identification System, which uses a bitmap to identify new network flows and a counting bloom filter to record flow numbers for each source. Several error compensation mechanisms are designed to dynamically compensate the system (e.g., hash collisions) while its states are changing, and thus substantially improve its accuracy in superspreaders identification. Multiple light-weight data operation techniques are also introduced to expedite data processing and reduce required memory. The Superspreader Identification System requires only 256KB SRAM to effectively identify superspreaders from high-speed data streams (e.g., OC-192) in line speed with more than 95% identification accuracy. Theoretical performance analysis is conducted to evaluate false negative ratio, false positive ratio, and compensation efficacy of the SIS. Comprehensive experiments using real Internet traces from a tier-1 Internet Service Provider (ISP) network demonstrate that the Superspreader Identification System is superior to other related approaches in terms of its achieved trade-offs among required memory space, identification accuracy, and data processing efficiency and capacity.

Enabling content-aware traffic engineering

Today, a large fraction of Internet traffic is originated by Content Delivery Networks (CDNs). To cope with increasing demand for content, CDNs have deployed massively distributed infrastructures. These deployments pose challenges for CDNs as they have to dynamically map end-users to appropriate servers without being full+y aware of the network conditions within an Internet Service Provider (ISP) or the end-user location. On the other hand, ISPs struggle to cope with rapid traffic shifts caused by the dynamic server selection policies of the CDNs. The challenges that CDNs and ISPs face separately can be turned into an opportunity for collaboration. We argue that it is sufficient for CDNs and ISPs to coordinate only in server selection, not routing, in order to perform traffic engineering. To this end, we propose Content-aware Traffic Engineering (CaTE), which dynamically adapts server selection for content hosted by CDNs using ISP recommendations on small time scales. CaTE relies on the observation that by selecting an appropriate server among those available to deliver the content, the path of the traffic in the network can be influenced in a desired way. We present the design and implementation of a prototype to realize CaTE, and show how CDNs and ISPs can jointly take advantage of the already deployed distributed hosting infrastructures and path diversity, as well as the ISP detailed view of the network status without revealing sensitive operational information. By relying on tier-1 ISP traces, we show that CaTE allows CDNs to enhance the end-user experience while enabling an ISP to achieve several traffic engineering goals.

Scaling Ethernet speeds to 100 Gbits/s and beyond

Dramatic growth in Internet Protocol (IP) traffic demand is driving the need for new high-bandwidth IP interfaces. Today, router-to-router and router-to-transport system connections using Ethernet interfaces are limited to 10 Gbits/s (10GE) or slower. Although techniques, such as link aggregation, allow a limited degree of extensibility beyond 10 Gbits/s, they are limited in terms of scalability, introduce additional complexity, and reduce throughput efficiency. Discussion now centers on defining an Ethernet architecture that meets the needs of carriers and is conducive to implementation by the switch and server vendors. In this paper, we consider aggregation at the physical layer (APL) as a means to reuse existing 10GE physical layers (PHYs), while offering interface scalability to 100 Gbits/s and beyond. With APL, aggregation is performed at the PHY, whereby full Ethernet frames are transmitted across the aggregated PHYs in a parallel fashion. This ensures equal utilization of all links and allows aggregate bandwidth between nodes to scale with each new link added. We have demonstrated the applicability of such an approach by implementing a 100 Gbits/s interface using off-the-shelf components and running it over a live 4,000 km backbone network of a tier-1 Internet service provider.

Quantifying Path Exploration in the Internet

Previous measurement studies have shown the existence of path exploration and slow convergence in the global Internet routing system, and a number of protocol enhancements have been proposed to remedy the problem. However, existing measurements were conducted only over a small number of testing prefixes. There has been no systematic study to quantify the pervasiveness of Border Gateway Protocol (BGP) slow convergence in the operational Internet, nor any known effort to deploy any of the proposed solutions. In this paper, we present our measurement results that identify BGP slow convergence events across the entire global routing table. Our data shows that the severity of path exploration and slow convergence varies depending on where prefixes are originated and where the observations are made in the Internet routing hierarchy. In general, routers in tier-1 Internet service providers (ISPs) observe less path exploration, hence they experience shorter convergence delays than routers in edge ASs; prefixes originated from tier-1 ISPs also experience less path exploration than those originated from edge ASs. Furthermore, our data show that the convergence time of route fail-over events is similar to that of new route announcements and is significantly shorter than that of route failures. This observation is contrary to the widely held view from previous experiments but confirms our earlier analytical results. Our effort also led to the development of a path-preference inference method based on the path usage time, which can be used by future studies of BGP dynamics.

Revenge of the Bellheads: how the Netheads lost control of the Internet

Throughout their short, but volatile lives, Internet Service Providers (ISPs) have avoided having to satisfy traditional telecommunications regulatory requirements. ISPs do not operate as common carriers and do not have to pay access charges and make contributions to universal service funding. Having sponsored development of the Internet, many governments consider regulation anathema, despite the fact that ISPs no longer need incubation and many offer some services functionally equivalent to what regulated telephone companies provide. A dichotomy in regulatory treatment exists between ISPs and telecommunications carriers. A large difference in corporate cultures supports this dichotomy. Telephone company executives (Bellheads) may resent regulation, but they accept their fate and work creatively to exploit anomalies and opportunities to secure a regulation-conferred competitive advantage. Most ISP executives (Netheads) appear to embrace a libertarian attitude, strongly opposing any government involvement. Despite the clash of cultures, the telecommunications and Internet worlds have merged. Such convergence jeopardizes the ability of Netheads to avoid some degree of regulation, particularly when they offer services functionally equivalent to what their Bellhead counterparts offer. This paper will assess the regulatory consequences when telecommunications and Internet services converge in the marketplace and in terms of operating technologies. The paper identifies commercial developments in the Internet to support the view that the Internet has become more hierarchical and more like telecommunications networks. For example, the major Internet backbone providers, commonly referred to as Tier-1 ISPs now require smaller ISPs to pay for access rather than simply grant reciprocal “peering” rights. The paper concludes that ownership of the major Tier-1 ISPs and local facilities by incumbent telecommunications carriers provides the basis for management and control of the Internet. Additional opportunities for leadership results from incumbent Bellheads’ superior skill in working the regulatory process to their advantage. The paper suggests that Bellheads will outmaneuver Netheads particularly when the revenue siphoning effect of Internet-mediated services offsets the revenues generated from ISP leases of telecommunications transmission capacity.