Lookup Performance Research Articles

H‐LCFST: a hash lowest cost first search tree for flexible packet classification

SummaryPacket classification (PC) categorizes incoming packets into multiple forwarding classes in a router based on predefined filters. It is one of the most important enabling technologies for next generation network services, for example, firewall, quality of service routing, load balancing, and virtual private network. IPv6 technology has posed a great challenge on the wire‐speed router for the PC because of its longer IP addresses. In this paper, we propose a new algorithm to support both IPv4 and IPv6, called hash lowest cost first search tree (H‐LCFST) for the PC. The H‐LCFST combines the advantages of the hash tree and a novel LCFST to achieve a better lookup performance. More important, the H‐LCFST algorithm is able to utilize the features of distinct IPv4 and IPv6 classifiers and design corresponding data structure for different characteristics of the classifiers, with which it is able to avoid the performance degrade because of the longer address of the IPv6. The experiment results show that our proposed algorithm has only approximately seven times of memory accesses for the IPv4/IPv6 PC, which make it to be the fastest PC solutions so far. Moreover, it occupies extremely small memory and supports incremental update at the same time. Copyright © 2016 John Wiley & Sons, Ltd.

Robust dynamic network traffic partitioning against malicious attacks

The continual growth of network traffic rates leads to heavy packet processing overheads, and a typical solution is to partition traffic into multiple network processors for parallel processing especially in emerging software-defined networks. This paper is thus motivated to propose a robust dynamic network traffic partitioning scheme to defend against malicious attacks. After introducing the conceptual framework of dynamic network traffic partitioning based on flow tables, we strengthen its TCP connection management by building a half-open connection separation mechanism to isolate false connections in the initial connection table (ICT). Then, the lookup performance of the ICT table is reinforced by applying counting bloom filters to cope with malicious behaviors such as SYN flooding attacks. Finally, we evaluate the performance of our proposed traffic partitioning scheme with real network traffic traces and simulated malicious traffic by experiments. Experimental results indicate that our proposed scheme outperforms the conventional ones in terms of packet distribution performance especially robustness against malicious attacks.

CONSERT: Constructing optimal name-based routing tables

Name-based routing belongs to a routing category different from address-based routing, it is usually adopted by content-oriented networks [Sharma et al., 2014, Koponen et al., 2007, Rajahalme et al.,2011, Thaler et al.,1998, Hwang et al., 2010, Gritter et al., 2001, Caesar et al., 2006, Carzaniga et al., 2004, Koponen et al., 2007, Hwang et al., 2009 Singla et al., 2010, Detti et al., 2011, Jain et al., 2011 Xu et al., 2013, Katsaros et al., 2012. [1–15]] e.g., the recently proposed Named Data Networking (NDN). It populates routers with name-based routing tables, which are composed of name prefixesand their corresponding next hop(s). Name-based routing tables are believed to have much larger size than IP routing tables, because of the large amount of name prefixes and the unbounded length of each prefix. This paper presents CONSERT—an algorithm that, given an arbitrary name-based routing table as input, computes a routing table with the minimal number of prefixes, while keeping equivalent forwarding behavior. The optimal routing table also supports incremental update. We formulate the CONSERT algorithm and prove its optimality with an induction method. Evaluation results show that, CONSERT can reduce 18% to 45% prefixes in the synthetic routing tables depending on the distribution of the next hops, and meanwhile improve the lookup performance by more than 20%. Prior efforts usually focus on compact data structures and lookup algorithms so as to reduce memory consumption and expedite lookup speed of the routing table, while CONSERT compresses the routing table from another perspective: it removes the inherent “redundancy” in the routing table. Therefore, CONSERT is orthogonal to these prior efforts, thus the combination of CONSERT and a prior compressing method would further optimize the memory consumption and lookup speed of the routing table. E.g., we can first adopt CONSERT to achieve the optimal routing table, and afterwards apply NameFilter [Wang et al., 2013. [16], a two-stage-Bloom-filter method, to that optimal table. This combination diminishes the memory consumption of the routing table data structure by roughly 88%, and increases the lookup throughput by around 17% simultaneously. The joint method outperforms each individual method in terms of memory savings and absolute lookup throughout increase.

A seven-dimensional analysis of hashing methods and its implications on query processing

Hashing is a solved problem. It allows us to get constant time access for lookups. Hashing is also simple. It is safe to use an arbitrary method as a black box and expect good performance, and optimizations to hashing can only improve it by a negligible delta. Why are all of the previous statements plain wrong? That is what this paper is about. In this paper we thoroughly study hashing for integer keys and carefully analyze the most common hashing methods in a five-dimensional requirements space: (1) data-distribution, (2) load factor, (3) dataset size, (4) read/write-ratio, and (5) un/successful-ratio. Each point in that design space may potentially suggest a different hashing scheme, and additionally also a different hash function. We show that a right or wrong decision in picking the right hashing scheme and hash function combination may lead to significant difference in performance. To substantiate this claim, we carefully analyze two additional dimensions: (6) five representative hashing schemes (which includes an improved variant of Robin Hood hashing), (7) four important classes of hash functions widely used today. That is, we consider 20 different combinations in total. Finally, we also provide a glimpse about the effect of table memory layout and the use of SIMD instructions. Our study clearly indicates that picking the right combination may have considerable impact on insert and lookup performance, as well as memory footprint. A major conclusion of our work is that hashing should be considered a white box before blindly using it in applications, such as query processing. Finally, we also provide a strong guideline about when to use which hashing method.

PMST:一种面向高性能IP查找的双向平衡流水线结构

With the rapid development of network technology and the growing diversity of network applications, IP lookup, as a core function of routers, has been confronted with serious challenges, ranging from throughput, memory efficiency and update performance. Benefiting from the pipeline technology, the throughput of IP lookup has been improved significantly. However, unbalanced structure will not only lead to low memory efficiency and high update overhead, but also influence the lookup performance and the load balance of multi-pipeline architectures. While existing solutions to balance optimizing would bring some indispensable new problems, impending their applications in IPv6 or large-scale routing tables. In view of this, this paper proposes a balanced bidirectional pipeline architecture—pipelined multi-bit split tree (PMST). After splitting prefixes, reversing a sub-tree and a series of optimizations, PMST can achieve a desirable balance degree at the cost of only a few stages. What is more, PMST also realizes significant improvements on comprehensive performance. Large IPv6 routing data sets generated in some manner, as well as real-world IPv4/v6 routing data sets, are used to evaluate the performance of PMST comprehensively. According to the experimental results, when achieving a desirable balance degree, the number of stages cost in PMST decreases by 75%~85.7% in comparison to existing solutions. Meanwhile, the superiorities of PMST on pipeline latency, average memory accesses per lookup, on-chip memory efficiency, update overhead and the load balance of multi-pipeline architecture are also significant. Therefore, PMST has higher comprehensive performance and better scalability, which can better meet current and future demands.

I-sieve: An inline high performance deduplication system used in cloud storage

Data deduplication is an emerging and widely employed method for current storage systems. As this technology is gradually applied in inline scenarios such as with virtual machines and cloud storage systems, this study proposes a novel deduplication architecture called I-sieve. The goal of I-sieve is to realize a high performance data sieve system based on iSCSI in the cloud storage system. We also design the corresponding index and mapping tables and present a multi-level cache using a solid state drive to reduce RAM consumption and to optimize lookup performance. A prototype of I-sieve is implemented based on the open source iSCSI target, and many experiments have been conducted driven by virtual machine images and testing tools. The evaluation results show excellent deduplication and foreground performance. More importantly, I-sieve can co-exist with the existing deduplication systems as long as they support the iSCSI protocol.

HEAPO

In this work, we developed a Heap-Based Persistent Object Store (HEAPO) to manage persistent objects in byte-addressable Nonvolatile RAM (NVRAM). HEAPO defines its own persistent heap layout, the persistent object format, name space organization, object sharing and protection mechanism, and undo-only log-based crash recovery, all of which are effectively tailored for NVRAM. We put our effort into developing a lightweight and flexible layer to exploit the DRAM-like access latency of NVRAM. To address this objective, we developed (i) a native management layer for NVRAM to eliminate redundancy between in-core and on-disk copies of the metadata, (ii) an expandable object format, (iii) a burst trie-based global name space with local name space caching, (iv) static address binding, and (v) minimal logging for undo-only crash recovery. We implemented HEAPO at commodity OS (Linux 2.6.32) and measured the performance. By eliminating metadata redundancy, HEAPO improved the speed of creating, attaching, and expanding an object by 1.3×, 4.5×, and 3.8×, respectively, compared to memory-mapped file-based persistent object store. Burst trie-based name space organization of HEAPO yielded 7.6× better lookup performance compared to hashed B-tree-based name space of EXT4. We modified memcachedb to use HEAPO in maintaining its search structure. For hash table update, HEAPO-based memcachedb yielded 3.4× performance improvement against original memcachedb implementation which uses mmap() over ramdisk approach to maintain the key-value store in memory.

Guarantee IP lookup performance with FIB explosion

The Forwarding Information Base (FIB) of backbone routers has been rapidly growing in size. An ideal IP lookup algorithm should achieve constant, yet small, IP lookup time and on-chip memory usage. However, no prior IP lookup algorithm achieves both requirements at the same time. In this paper, we first propose SAIL, a Splitting Approach to IP Lookup. One splitting is along the dimension of the lookup process, namely finding the prefix length and finding the next hop, and another splitting is along the dimension of prefix length, namely IP lookup on prefixes of length less than or equal to 24 and IP lookup on prefixes of length longer than 24. Second, we propose a suite of algorithms for IP lookup based on our SAIL framework. Third, we implemented our algorithms on four platforms: CPU, FPGA, GPU, and many-core. We conducted extensive experiments to evaluate our algorithms using real FIBs and real traffic from a major ISP in China. Experimental results show that our SAIL algorithms are several times or even two orders of magnitude faster than well known IP lookup algorithms.

An effective single‐hop distributed hash table with high lookup performance and low traffic overhead

SUMMARYDistributed hash tables (DHTs) have been used in several applications, but most DHTs have opted to solve lookups with multiple hops, to minimize bandwidth costs while sacrificing lookup latency. This paper presents D1HT, an original DHT that has a peer‐to‐peer and self‐organizing architecture and maximizes lookup performance with reasonable maintenance traffic, and a Quarantine mechanism to reduce overheads caused by volatile peers. We implemented both D1HT and a prominent single‐hop DHT, and we performed an extensive and highly representative DHT experimental comparison, followed by complementary analytical studies. In comparison with current single‐hop DHTs, our results showed that D1HT consistently had the lowest bandwidth requirements, with typical reductions of up to one order of magnitude, and that D1HT could be used even in popular Internet applications with millions of users. In addition, we ran the first latency experiments comparing DHTs to directory servers, which revealed that D1HT can achieve latencies equivalent to or better than a directory server, and confirmed its greater scalability properties. Overall, our extensive set of results allowed us to conclude that D1HT can provide a very effective solution for a broad range of environments, from large‐scale corporate data centers to widely deployed Internet applications. Copyright © 2014 John Wiley & Sons, Ltd.

English

Data Deduplication describes approach that reduces the storage capacity needed to store data or the data has to be transfer on the network. Cloud storage has received increasing attention from industry as it offers infinite storage resources that are available on demand. Source Deduplication is useful in cloud backup that saves network bandwidth and reduces network space Deduplication is the process by breaking up an incoming stream into relatively large segments and deduplicating each segment against only a few of the most similar previous segments. To identify similar segments use block index technique The problem is that these schemes traditionally require a full chunk index, which indexes every chunk, in order to determine which chunks have already been stored unfortunately, it is impractical to keep such an index in RAM and a disk based index with one seek per incoming chunk is far too slow. In this paper we describes application based deduplication approach and indexing scheme contains block that preserved caching which maintains the locality of the fingerprint of duplicate content to achieve high hit ratio and to overcome the lookup performance and reduced cost for cloud backup services and increase dedulpication efficiency.

A Hybrid Hardware Architecture for High-Speed IP Lookups and Fast Route Updates

As network link rates are being pushed beyond 40 Gb/s, IP lookup in high-speed routers is moving to hardware. The ternary content addressable memory (TCAM)-based IP lookup engine and the static random access memory (SRAM)-based IP lookup pipeline are the two most common ways to achieve high throughput. However, route updates in both engines degrade lookup performance and may lead to packet drops. Moreover, there is a growing interest in virtual IP routers where more frequent updates happen. Finding solutions that achieve both fast lookup and low update overhead becomes critical. In this paper, we propose a hybrid IP lookup architecture to address this challenge. The architecture is based on an efficient trie partitioning scheme that divides the forwarding information base (FIB) into two prefix sets: a large disjoint leaf prefix set mapped into an external TCAM-based lookup engine and a small overlapping prefix set mapped into an on-chip SRAM-based lookup pipeline. Critical optimizations are developed on both IP lookup engines to reduce the update overhead. We show how to extend the proposed hybrid architecture to support virtual routers. Our implementation shows a throughput of 250 million lookups per second (equivalent to 128 Gb/s with 64-B packets). The update overhead is significantly lower than that of previous work, the memory consumption is reasonable, and the utilization ratio of most external TCAMs is up to 100%.

Bitmap-based Prefix Caching for Fast IP Lookup

IP address lookup is very crucial in performance of routers. Several works have been done on prefix caching to enhance the performance of IP address lookup. Since a prefix represents a range of IP addresses, a prefix cache shows better performance than an IP address cache. However, not every prefix is cacheable in itself. In a prefix cache it causes false hit to cache a non-leaf prefix because there is possibly the longer matching prefix in the routing table. Prefix expansion techniques such as complete prefix tree expansion (CPTE) make it possible to cache the non-leaf prefixes as the expanded forms, but it is hard to manage the expanded prefixes. The expanded prefixes sometimes incur a great deal of update overhead in a routing table. We propose a bitmap-based prefix cache (BMCache) to provide low update overhead as well as low cache miss ratio. The proposed scheme does not have any expanded prefixes in the routing table, but it can expand a non-leaf prefix using a bitmap on caching time. The trace-driven simulation shows that BMCache has very low miss ratio in spite of its low update overhead compared to other schemes.

Lookup Performance Deficiencies in the KAD Network

The KAD network is one of the largest distributed hash tables (DHT) deployed in the real world. DHT is designed to provide scalable peer-to-peer lookup service. Its real-world lookup performance, however, is recognized as inefficient. We ran wide-range measurements on the KAD network to evaluate this inefficiency by examining peer behaviors in publishing and retrieving information using routing tables and publishing tables. Our measurements show that even though routing tables are well maintained, the current refreshment scheme of publishing tables and lack of effective incentive policy cause lookup performance deficiency. Based on our findings, we propose three improvement schemes, analyze their strengths, and give suggestions to implementations.

Community-Based Caching for Enhanced Lookup Performance in P2P Systems

Large peer-to-peer systems exhibit the presence of communities based on user interests. Resources commonly shared within individual communities are in general relatively less popular and inconspicuous in the system-wide behavior. Hence, such communities are unable to benefit significantly from caching and replication that focus only on the most dominant queries. A community-based caching (CBC) solution that enhances both community-wide and system-wide lookup performance is proposed. CBC consists of a suboverlay formation scheme and a local-knowledge-based distributed caching (LKDC) algorithm. Suboverlays enable communities to forward queries through their members. While queries are forwarded, the LKDC algorithm causes members to identify and cache resources of interests to them, resulting in faster resolution of queries for popular resources within each community. Distributed local caching requires global information (e.g., hop count and popularity of contents) that is difficult and costly to obtain. However, by means of an analysis of globally optimal behavior and structural properties of the overlay, we developed the heuristic-based LKDC algorithm that not only relies on purely local information but also provides close-to-optimal caching performance. CBC is adaptive to changing popularity and user interests, works with any skewed distribution of queries, and introduces minimal modifications and overhead to the overlay network.

Scale-Free Networks to Search in Unstructured Peer-To-Peer Networks

Unstructured peer-to-peer file sharing networks are very popular in the market. They introduce network traffic. The resultant networks are not perform search efficiently because existing overlay topology formation algorithms are creates unstructured P2P networks without performance guarantees. In this paper, We compares structured and unstructured overlays, Showing through analytical and simulation results how an unstructured solution relying on a scale-free topology is an effective option to deploy for offering services based on equivalent servants. Based on this result, The EQUivalent servAnt locaTOR (EQUATOR) architecture, which overcomes the issues related to the deployment of a scale-free topology for service location in a real network, mainly due to the static nature of the ideal scale-free construction algorithm and the lack of a global knowledge of the participating peers. Simulation results confirmed the effectiveness of EQUATOR, showing how it offers good lookup performance in combination with low message overhead and high resiliency to node churn and failures. Some possible future works are introduced and are related to some complementary issues ranging from the proximity-aware selection of servants to the introduction of proper incentives to encourage nodes to join the EQUATOR overlay and offer their resources.

Spiralchord: a space-filling curve based location awareness, cross-layering P2P file sharing system in WMNs

The common characteristics of peer-to-peer (P2P) overlay networks and wireless multi-hop network, such as self-organization, decentralization, hop-by-hop message transmission mode and high degree of dynamicity, lead to research of operating wired P2P applications on wireless multi-hop networks. Wireless mesh network (WMN) as a relative static multi-hop wireless network which is extended from Ad-Hoc networks, has become one of the key technologies for providing increased network coverage of Internet infrastructures. This paper investigates the problem of enabling P2P file sharing in WMNs. A special chord algorithm—spiralchord is proposed to address the major problem in wireless file sharing system—how to efficiently find resources currently available. Spiralchord put forward an identifier (ID) assignment technique based on spiral space-filling curve to integrate location-awareness with cross-layering. Location awareness aims at alleviating the mismatch of physical network topology and overlay network topology, and requires close-by IDs in logical ring of neighboring peers, while cross-layering aims at speeding up resource lookup operations, requires faraway IDs of neighboring peers. Spiralchord uses spiral curve to assign peers' IDs which meet the contradictory requirements of location-awareness and cross-layering. The simulation results show spiralchord is effective in reducing message overhead, and increasing lookup performance with respect to basic chord.

An efficient IP address lookup algorithm based on a small balanced tree using entry reduction

Due to a tremendous increase in internet traffic, backbone routers must have the capability to forward massive incoming packets at several gigabits per second. IP address lookup is one of the most challenging tasks for high-speed packet forwarding. Some high-end routers have been implemented with hardware parallelism using ternary content addressable memory (TCAM). However, TCAM is much more expensive in terms of circuit complexity as well as power consumption. Therefore, efficient algorithmic solutions are essentially required to be implemented using network processors as low cost solutions. Among the state-of-the-art algorithms for IP address lookup, a binary search based on a balanced tree is effective in providing a low-cost solution. In order to construct a balanced search tree, the prefixes with the nesting relationship should be converted into completely disjointed prefixes. A leaf-pushing technique is very useful to eliminate the nesting relationship among prefixes [V. Srinivasan, G. Varghese, Fast address lookups using controlled prefix expansion, ACM Transactions on Computer Systems 17 (1) (1999) 1–40]. However, it creates duplicate prefixes, thus expanding the search tree. This paper proposes an efficient IP address lookup algorithm based on a small balanced tree using entry reduction. The leaf-pushing technique is used for creating the completely disjointed entries. In the leaf-pushed prefixes, there are numerous pairs of adjacent prefixes with similarities in prefix strings and output ports. The number of entries can be significantly reduced by the use of a new entry reduction method which merges pairs with these similar prefixes. After sorting the reduced disjointed entries, a small balanced tree is constructed with a very small node size. Based on this small balanced tree, a native binary search can be effectively used in address lookup issue. In addition, we propose a new multi-way search algorithm to improve a binary search for IPv4 address lookup. As a result, the proposed algorithms offer excellent lookup performance along with reduced memory requirements. Besides, these provide good scalability for large amounts of routing data and for the address migration toward IPv6. Using both various IPv4 and IPv6 routing data, the performance evaluation results demonstrate that the proposed algorithms have better performance in terms of lookup speed, memory requirement and scalability for the growth of entries and IPv6, as compared with other algorithms based on a binary search.

Cooperative caching versus proactive replication for location dependent request patterns

Today's trend to create and share personal content, such as digital photos and digital movies, results in an explosive growth of a user's personal content archive. Managing such an often distributed collection becomes a complex and time consuming task, indicating the need for a personal content management system that provides storage space transparently, is quality-aware, and is available at any time and at any place to end-users. A key feature of such a Personal Content Storage Service (PCSS) is the ability to search worldwide through the dataset of personal files. Due to the extremely large size of the dataset of personal content, a centralized solution is no longer feasible and an interesting approach for an efficient distributed PCSS implementation is to use a structured Peer-to-Peer network, and more in particular a Distributed Hash Table (DHT), providing a logarithmic lookup performance in the number of network nodes. In order to further increase the lookup performance, a caching layer is typically used between the application layer and the DHT. These caching strategies are location neutral, and usually do not exploit location dependence of request patterns. In this article we present our cooperative caching framework and introduce the cooperative Request Times Distance (RTD) caching algorithm. Since, lookup patterns in a PCSS typically have a power law popularity distribution and exhibit location dependent requests patterns, the proposed caching solution takes into account popularity and distance metrics. To enable cooperation between caches we introduce an update protocol, which only occasionally introduces one hop delay for a lookup operation. We present a systematic analysis of the caching framework and compare the cooperative caching algorithm to the state-of-the-art Beehive replication strategy. The cooperative RTD caching solution shows that when request patterns are more localized, the increase in lookup performance through cooperation is significantly better than Beehive.

Opportunistic P2P Communications in Delay-Tolerant Rural Scenarios

Opportunistic networking represents a promising paradigm for support of communications, specifically in infrastructureless scenarios such as remote areas communications. In principle in opportunistic environments, we would like to make available all the applications thought for traditional wired and wireless networks like file-sharing and content distribution. In this paper, we present a delay-tolerant scenario for file sharing applications in rural areas, where an opportunistic approach is exploited. In order to support communications, we compare two peer-to-peer (P2P) schemes initially conceived for wireless networks and prove their applicability and usefulness to a DTN scenario, where replication of resources can be used to improve the lookup performance and the network can be occasionally connected by means of a data mule. Simulation results show the suitability of the schemes and allow to derive interesting design guidelines on the convenience and applicability of such approaches.

Evaluating and improving the content access in KAD

We analyze in detail the content retrieval process in kad. kad implements content search (publish and retrieval) functions that use the Kademlia Distributed Hash Table for content routing. Node churn is quite common in peer-to-peer systems and results in information loss and stale routing table entries. To deal with node churn, kad issues parallel route requests and publishes multiple redundant copies of each piece of information. We identify the key design parameters in kad and present an analytical model to evaluate the impact of changes in the values of these parameters on the overall lookup latency and message overhead. Extensive measurements of the lookup performance using an instrumented client allow us to validate the model. The overall lookup latency is in most cases 5 s or larger. We elucidate the cause for such high lookup latencies and propose an improved scheme that significantly decreases the overall lookup latency without increasing the overhead.