Distributed Hash Table Systems Research Articles

HyCube: A distributed hash table based on a variable metric

Distributed Hash Tables (DHT) proved to be scalable decentralized systems providing efficient resource location. This paper concentrates on efficiency and resilience to node failures of DHT systems and presents a novel model of a distributed hash table based on a hierarchical hypercube geometry, called HyCube. The DHT geometry, the choice of the metric defining logical distances between nodes, and the routing algorithm have fundamental influence on routing efficiency and resilience. The use of the one-dimensional model (placing the nodes logically on a ring) allows the nodes to maintain sets of references called sequential neighbors - certain numbers of neighbors that are the closest existing nodes in both directions on the ring. Such a model yields a very high level of resilience to node failures. The new approach, presented in the paper, employs a variable multi-dimensional metric adopting the Steinhaus transform. Routing, lookup and search algorithms are discussed, as well as routing table nodes selection and self-organization techniques. It is shown that the new approach allows reaching a higher level of resilience to node failures, as well as a shorter average routing path length than with the use of the sequential neighbors sets.

Analyzing Generic Network Location Service

The main focus of this work is to describe specific fault-tolerance mechanisms which can be applied in a DHT (Distributed Hash Table) technology). We implement device translation solution among different identifications of devices based on the DHT (Distributed Hash Table) technology and present various algorithms ensuring fault-tolerance. Whereas nodes in DHT's are obviously used to store data identified by a single identification key and their possible failures are resolved by approaches of multiple data copies. We assumed the usage of stored value groups - records - for any possible accesses and functions. Records required for any kind of a specific identification set may be stored in more differently placed record copies in the DHT system. This approach supports the ability to solve failures of specific nodes in the DHT node set without direct multiplication of their storage space.Our solution was based on the DHT Chord implementation and the development of the Generic Network Location Service (GNLS) based on the Chord implementation. The systems of algorithms both proactive and reactive were used to solve a problem of node failures have been developed and compared in this research and selected algorithms have been tested by experimentation and simulations. The provided algorithms guarantee fault-tolerance in the presence of dynamism: they guarantee consistent lookup results in the presence of nodes failing and leaving. Finally, GNLS results are analysed. By practical and theoretical verification, it is concluded that reactive algorithms performed better than proactive algorithms in the GNLS framework and solved some of the most important problems of device location identifications.

An ID-based approach to the caching and distribution of peer-to-peer, proxy-based video content

The viewing of streamed video content has become second nature these days for many Internet users. With such copious amounts of data being transferred between hosts involved in this streaming, it is prudent to establish some method of effectively minimising the resultant outbound network traffic, whilst providing the client with a good Quality of Service (QoS). Although many approaches have attempted to address the aforementioned issues, most tend to focus on directly aiding the content provider, rather than the client and typically require a lookup mechanism, such as a distributed hash table (DHT), which engenders a storage and complexity overhead. In this paper we propose a novel infrastructure to the aim of enhancing QoS, whilst reducing outbound network traffic, for clients downloading YouTube video content. We achieve this without a reliance upon request flooding or large hash table overheads inherent to DHT systems. Augmenting this, we proffer two additional mechanisms, the first balances localised storage-peer (client) load and the second preemptively balances super-peer (proxy) load. Lastly, we provide a peer failure mechanism, which allows the proposed system to quickly and efficiently recover from peer failures. Results garnered from emulations indicate that the proposed architecture offers respectable cached video retrieval speeds, even under dynamic and high traffic conditions, thus precluding typical client video lag and/or buffering.

Resource Discovery Service while Minimizing Maintenance Overhead in Hierarchical DHT Systems

Despite hierarchical Distributed Hash Table (DHT) systems have addressed flat overlay system problems, most of existing solutions add a significant overhead to large scale systems. This not only increases the bandwidth consumption but also affect the routing efficiency. This paper deals with a resource discovery while minimizing maintenance overhead in hierarchical DHT systems. The considered resources are metadata describing data sources. In the solution, only one gateway in one overlay is attached to the superior level overlay. It aims to reduce both lookup and maintenance costs while minimizing the overhead added to the system. The authors present a cost analysis for a resource discovery process and discuss capabilities of the proposed protocol to reduce the overhead of maintaining the overlay network. The analysis result proved that our design decrease significantly the maintenance costs in such systems especially when nodes frequently join/leave the system.

Load balancing for two-tier Chord system

Compared to flat distributed hash table (DHT) systems, hierarchical DHT systems can use some powerful and stable peers (called superpeers) to achieve efficient look-up under churn. Several hierarchical DHT systems have been proposed so far. However, a crucial problem faced by all these systems is the load imbalance among superpeers. Proposed is a novel load balancing algorithm to efficiently solve this problem in a two-tier Chord system. Analysis and simulation results show that the algorithm can balance the load among superpeers in proportion to their capacity.

Data Recovery of Distributed Hash Table with Distributed-to-Distributed Data Copy

To realize huge-scale information services, many Distributed Hash Table (DHT) based systems have been proposed. For example, there are some proposals to manage item-level product traceability information with DHTs. In such an application, each entry of a huge number of item-level IDs need to be available on a DHT. To ensure data availability, the soft-state approach has been employed in previous works. However, this does not scale well against the number of entries on a DHT. As we expect 1010 products in the traceability case, the soft-state approach is unacceptable. In this paper, we propose Distributed-to-Distributed Data Copy (D3C). With D3C, users can reconstruct the data as they detect data loss, or even migrate to another DHT system. We show why it scales well against the number of entries on a DHT. We have confirmed our approach with a prototype. Evaluation shows our approach fits well on a DHT with a low rate of failure and a huge number of data entries.

Resilient and efficient load balancing in distributed hash tables

As a fundamental problem in distributed hash table (DHT)-based systems, load balancing is important to avoid performance degradation and guarantee system fairness. Among existing migration-based load balancing strategies, there are two main categories: (1) rendezvous directory strategy ( RDS) and (2) independent searching strategy ( ISS). However, none of them can achieve resilience and efficiency at the same time. In this paper, we propose a group multicast strategy ( GMS) for load balancing in DHT systems, which attempts to achieve the benefits of both RDS and ISS. GMS does not rely on a few static rendezvous directories to perform load balancing. Instead, load information is disseminated within the formed groups via a multicast protocol. Thus, each peer has enough information to act as the rendezvous directory and perform load balancing within its group. Besides intra-group load balancing, inter-group load balancing and emergent load balancing are also supported by GMS. In GMS, the position of the rendezvous directory is randomized in each round, which further improves system resilience. In order to have a better understanding of GMS, we also perform analytical studies on GMS in terms of its scalability and efficiency under churn. Finally, the effectiveness of GMS is evaluated by extensive simulation under different workload and churn levels.

Degree-Optimal Routing for P2P Systems

We define a family of Distributed Hash Table systems whose aim is to combine the routing efficiency of randomized networks—e.g. optimal average path length O(log 2 n/δlog δ) with δ degree—with the programmability and startup efficiency of a uniform overlay—that is, a deterministic system in which the overlay network is transitive and greedy routing is optimal. It is known that Ω(log n) is a lower bound on the average path length for uniform overlays with O(log n) degree (Xu et al., IEEE J. Sel. Areas Commun. 22(1), 151–163, 2004). Our work is inspired by neighbor-of-neighbor (NoN) routing, a recently introduced variation of greedy routing that allows us to achieve optimal average path length in randomized networks. The advantage of our proposal is that of allowing the NoN technique to be implemented without adding any overhead to the corresponding deterministic network. We propose a family of networks parameterized with a positive integer c which measures the amount of randomness that is used. By varying the value c, the system goes from the deterministic case (c=1) to an “almost uniform” system. Increasing c to relatively low values allows for routing with asymptotically optimal average path length while retaining most of the advantages of a uniform system, such as easy programmability and quick bootstrap of the nodes entering the system. We also provide a matching lower bound for the average path length of the routing schemes for any c.

An analytical study on optimizing the lookup performance of distributed hash table systems under churn

The phenomenon of system churn degrades the lookup performance of distributed hash table (DHT) systems greatly. To handle the churn, a number of approaches have been proposed to date. However, ther...

An analytical study on optimizing the lookup performance of distributed hash table systems under churn

AbstractThe phenomenon of system churn degrades the lookup performance of distributed hash table (DHT) systems greatly. To handle the churn, a number of approaches have been proposed to date. However, there is a lack of theoretical analysis to direct how to make design choices under different churn rates and how to configure their parameters optimally. In this paper, we analytically study three important aspects on optimizing DHT lookup performance under churn, i.e. lookup strategy, lookup parallelism and lookup key replication. Our objective is to build a theoretical basis for designers to make better design choices in the future. We first compare the performance of two representative lookup strategies—recursive routing and iterative routing—and explore the existence of better alternatives. Then we study the effectiveness of lookup parallelism in systems with different churn rates and show how to select the optimal degree of parallelism. Owing to the importance of key replication on lookup performance, we also analyze the reliability of the replicated key under two different replication policies, and show how to perform proper configuration. Besides the analytical study, our results are also validated by simulation, and Kad is taken as a case to show the meaningfulness of our analysis. Copyright © 2007 John Wiley & Sons, Ltd.

Locality in structured peer-to-peer networks

Distributed hash tables (DHTs), used in a number of structured peer-to-peer (P2P) systems, provide efficient mechanisms for resource placement and location. A key distinguishing feature of current DHT systems, such as Chord, Pastry, CAN and Tapestry, is the way they handle locality in the underlying network. Topology-based node identifier assignment, proximity routing, and proximity neighbor selection are examples of heuristics used to minimize message delays in the underlying network. While these heuristics are sometimes effective, they all rely on a single global overlay that may install the key of a popular object at a node far from most of the nodes accessing it. Furthermore, a response to a lookup message does not contain any locality information about the nodes holding a copy of the object. We address these issues in Plethora, a novel two-level overlay P2P network. A local overlay in Plethora acts as a locality-aware cache for the global overlay, grouping nodes close together in the underlying network. Local overlays are constructed by exploiting the organization of the Internet into autonomous systems (ASs). We present a detailed experimental study that demonstrates performance gains in response time of up to 60% compared to a single global Pastry overlay. We also present efficient distributed algorithms for maintaining local overlays in the presence of node arrivals and departures.

Improving lookup latency in distributed hash table systems using random sampling

Distributed hash table (DHT) systems are an important class of peer-to-peer routing infrastructures. They enable scalable wide-area storage and retrieval of information, and will support the rapid development of a wide variety of Internet-scale applications ranging from naming systems and file systems to application-layer multicast. DHT systems essentially build an overlay network, but a path on the overlay between any two nodes can be significantly different from the unicast path between those two nodes on the underlying network. As such, the lookup latency in these systems can be quite high and can adversely impact the performance of applications built on top of such systems. In this paper, we discuss a random sampling technique that incrementally improves lookup latency in DHT systems. Our sampling can be implemented using information gleaned from lookups traversing the overlay network. For this reason, we call our approach lookup-parasitic random sampling (LPRS). LPRS converges quickly, and requires relatively few modifications to existing DHT systems. For idealized versions of DHT systems like Chord, Tapestry, and Pastry, we analytically prove that LPRS can result in lookup latencies proportional to the average unicast latency of the network, provided the underlying physical topology has a power-law latency expansion. We then validate this analysis by implementing LPRS in the Chord simulator. Our simulations reveal that LPRS-Chord exhibits a qualitatively better latency scaling behavior relative to unmodified Chord. The overhead of LPRS is one sample per lookup hop in the worst case. Finally, we provide evidence which suggests that the Internet router-level topology resembles power-law latency expansion. This finding implies that LPRS has significant practical applicability as a general latency reduction technique for many DHT systems. This finding is also of independent interest since it might inform the design of latency-sensitive topology models for the Internet.

Incrementally improving lookup latency in distributed hash table systems

Distributed hash table (DHT) systems are an important class of peer-to-peer routing infrastructures. They enable scalable wide-area storage and retrieval of information, and will support the rapid development of a wide variety of Internet-scale applications ranging from naming systems and file systems to application-layer multicast. DHT systems essentially build an overlay network, but a path on the overlay between any two nodes can be significantly different from the unicast path between those two nodes on the underlying network. As such, the lookup latency in these systems can be quite high and can adversely impact the performance of applications built on top of such systems.In this paper, we discuss a random sampling technique that incrementally improves lookup latency in DHT systems. Our sampling can be implemented using information gleaned from lookups traversing the overlay network. For this reason, we call our approach lookup-parasitic random sampling (LPRS). LPRS is fast, incurs little network overhead, and requires relatively few modifications to existing DHT systems.For idealized versions of DHT systems like Chord, Tapestry and Pastry, we analytically prove that LPRS can result in lookup latencies proportional to the average unicast latency of the network, provided the underlying physical topology has a power-law latency expansion. We then validate this analysis by implementing LPRS in the Chord simulator. Our simulations reveal that LPRS-Chord exhibits a qualitatively better latency scaling behavior relative to unmodified Chord.Finally, we provide evidence which suggests that the Internet router-level topology resembles power-law latency expansion. This finding implies that LPRS has significant practical applicability as a general latency reduction technique for many DHT systems. This finding is also of independent interest since it might inform the design of latency-sensitive topology models for the Internet.