Consistent Hashing Research Articles

JumpBackHash: Say Goodbye to the Modulo Operation to Distribute Keys Uniformly to Buckets

ABSTRACTIntroductionDistributed data processing and storage systems require efficient methods to distribute keys across buckets. While simple and fast, the traditional modulo‐based mapping is unstable when the number of buckets changes, leading to spikes in system resource utilization, such as network or database requests. Consistent hash algorithms minimize remappings but are either significantly slower, require floating‐point arithmetic, or are based on a family of hash functions rarely available in standard libraries. This work introduces JumpBackHash, a consistent hash algorithm that overcomes those shortcomings.MethodologyJumpBackHash applies the concept of active indices borrowed from consistent weighted sampling, which inherently leads to consistency. It generates the active indices in reverse order, which avoids floating‐point operations, enables the minimization of consumed random values and the use of a standard pseudorandom generator, and finally leads to a very efficient algorithm.ResultsTheoretical analysis shows that JumpBackHash has an expected constant runtime. The expected value and the variance of the number of consumed random values perfectly agree with the experiments. Empirical tests also confirm the consistency.ConclusionJumpBackHash offers a fast and efficient solution for uniformly distributing keys across buckets in distributed systems. Its simplicity, performance, and the availability of a production‐ready Java implementation as part of the Hash4j open source library make it a viable replacement for the modulo‐based approach for improving assignment and system stability.

Decentralized collaborative machine learning for protecting electricity data

In recent years, there has been a noticeable surge in electric power load due to economic development and improved living standards. The growing need for smart power solutions, such as leveraging user electricity data to forecast power peaks and utilizing power data statistics to enhance end-user services, has been on the rise. However, the misuse and unauthorized access of data have prompted stringent regulations to safeguard data integrity. This paper presents a novel decentralized collaborative machine learning framework aimed at predicting peak power loads while protecting the privacy of users’ power data. In this scheme, multiple users engage in collaborative machine learning training within a peer-to-peer network free from a centralized server, with the objective of predicting peak power loads without compromising users’ local data privacy. The proposed approach leverages blockchain technology and advanced cryptographic techniques, including multi-key homomorphic encryption and consistent hashing. Key contributions of this framework include the development of a secure dual-aggregate node aggregation algorithm and the establishment of a verifiable process within a decentralized architecture. Experimental validation has been conducted to assess the feasibility and effectiveness of the proposed scheme, demonstrating its potential to address the challenges associated with predicting peak power loads securely and preserving user data privacy.

Semi-supervised Cross-modal Hashing with Joint Hyperboloid Mapping

By using a small amount of label information to achieve favorable performance, semi-supervised methods are more practical in real-world application scenarios. However, existing semi-supervised cross-modal retrieval methods mainly focus on preserving similarities and learning more consistent hash codes yet overlook the importance of constructing a joint abstract space shared by multi-modal embeddings. In this paper, we propose a novel Semi-supervised Cross-modal Hashing with Joint Hyperboloid Mapping (SCH-JHM). Firstly, we present a diffusion-based teacher model in SCH-JHM to learn the generalized semantic knowledge and output the pseudo-labels for unlabeled data. Secondly, SCH-JHM establishes a five-tuple plane, resembling an hourglass, for each retrieval task based on the queries, positive pairs, negative pairs, semi-supervised positive pairs, and semi-supervised negative pairs included in the semi-supervised cross-modal retrieval task. Furthermore, it projects the 12 tasks from the image, text, video, and audio modalities into a joint hyperboloid space. Finally, the student model in SCH-JHM is employed to explore the latent semantic relevance between filtered heterogeneous entities, which can be considered as a supervised process. Comprehensive experiments compared with state-of-the-art methods on three widely used datasets verify the effectiveness of our proposed approach.

MementoHash: A Stateful, Minimal Memory, Best Performing Consistent Hash Algorithm

MementoHash: A Stateful, Minimal Memory, Best Performing Consistent Hash Algorithm

Computation Offloading Based on a Distributed Overlay Network Cache-Sharing Mechanism in Multi-Access Edge Computing

Multi-access edge computing (MEC) enhances service quality for users and reduces computational overhead by migrating workloads and application data to the network edge. However, current solutions for task offloading and cache replacement in edge scenarios are constrained by factors such as communication bandwidth, wireless network coverage, and limited storage capacity of edge devices, making it challenging to achieve high cache reuse and lower system energy consumption. To address these issues, a framework leveraging cooperative edge servers deployed in wireless access networks across different geographical regions is designed. Specifically, we propose the Distributed Edge Service Caching and Offloading (DESCO) network architecture and design a decentralized resource-sharing algorithm based on consistent hashing, named Cache Chord. Subsequently, based on DESCO and aiming to minimize overall user energy consumption while maintaining user latency constraints, we introduce the real-time computation offloading (RCO) problem and transform RCO into a multi-player static game, prove the existence of Nash equilibrium solutions, and solve it using a multi-dimensional particle swarm optimization algorithm. Finally, simulation results demonstrate that the proposed solution reduces the average energy consumption by over 27% in the DESCO network compared to existing algorithms.

An Automatic Near-Duplicate Video Data Cleaning Method Based on a Consistent Feature Hash Ring

An Automatic Near-Duplicate Video Data Cleaning Method Based on a Consistent Feature Hash Ring

Improving byzantine fault tolerance based on stake evaluation and consistent hashing

Improving byzantine fault tolerance based on stake evaluation and consistent hashing

DxHash: A Memory-saving Consistent Hashing Algorithm

Consistent Hashing (CH) algorithms are widely adopted in networks and distributed systems for their ability to achieve load balancing and minimize disruptions. However, the rise of the Internet of Things (IoT) has introduced new challenges for existing CH algorithms, characterized by high memory usage and update overhead. This article presents DxHash, a novel CH algorithm based on repeated pseudo-random number generation. DxHash offers significant benefits, including a remarkably low memory footprint, high lookup throughput, and minimal update overhead. Additionally, we introduce a weighted variant of DxHash, enabling adaptable weight adjustments to handle heterogeneous load distribution. Through extensive evaluation, we demonstrate that DxHash outperforms AnchorHash, a state-of-the-art CH algorithm, in terms of the reduction of up to 98.4% in memory footprint and comparable performance in lookup and update.

Multi-level index construction method based on master-slave blockchains.

Master-slave blockchain is a novel information processing technology that is domain-oriented and uses efficient cryptography principles for trustworthy communication and storage of big data. Existing indexing methods primarily target the creation of a single-structured blockchain, resulting in extensive time and memory requirements. As the scale of domain data continues to grow exponentially, master-slave blockchain systems face increasingly severe challenges with regards to low query efficiency and extended traceback times. To address these issues, this paper propose a multi-level index construction method for the master-slave blockchain (MLI). Firstly, MLI introduces a weight matrix and partitions the entire master-slave blockchain based on the master chain structure, the weight of each partition is assigned. Secondly, for the master blockchain in each partition, a master chain index construction method based on jump consistent hash (JHMI) is proposed, which takes the key value of the nodes and the number of index slots as input and outputs the master chain index. Finally, a bloom filter is introduced to improve the column-based selection function and build a secondary composite index on the subordinate blockchain corresponding to each master block. Experimental results on three constraint conditions and two types of datasets demonstrate that the proposed method reduce the index construction time by an average of 9.28%, improve the query efficiency by 12.07%, and reduce the memory overhead by 24.4%.

Innovations in Cyber Security Algorithms for Databases Enhancing Data Retrieval and Management

The term Innovations in Cyber Security Algorithms for Databases Enhancing Data Retrieval and Management refers to a book that studies novel techniques for tackling problems related to digital data. The integration of three complicated methods—DQO, DSS, and RAI—is the major focus of attention in this piece of writing. DQO makes use of machine learning to optimize query processing on the fly to meet fluctuating workloads. This is done to accommodate such workloads. To address issues pertaining to the scale of distributed systems, distributed storage systems (DSS) convey data in an effective manner by utilizing consistent hashing. The RAI algorithm adjusts the index architecture in response to the query patterns to achieve real-time flexibility. In this way, the process of looking for information that is frequently asked about is sped up. The methodology that has been suggested is superior to six different ways that are often used in terms of its adaptability, scalability, and real-time capabilities. This article will give a thorough model for improving data management in computer systems. The objective of this essay is to present the model.

Supervised adaptive similarity consistent latent representation hashing

Cross-modal hashing has attracted significant attention in multimedia data similarity given its appealing computational cost and retrieval performance. Supervised hashing benefits from the auxiliary learning of a similarity matrix, which is usually predefined by inner product features or category labels. However, a predefined similarity matrix fails to reflect the real similarity relationship between image-text pairs. In addition, existing methods fix the weights to a value or update them by introducing sensitive dataset-related hyper-parameters. To overcome these problems, we propose a method to perform supervised adaptive similarity consistent latent representation hashing (SCLRH) that adaptively learns the similarity matrix during hashing learning. In SCLRH, we assume that multimodal data are observed and reconstructed from different perspectives of a common consistent latent representation. Instead of using a predefined similarity matrix, SCLRH adaptively learns this matrix to reflect the underlying manifold structure and describes the fine-grained similarity between consistent latent representations. In addition, SCLRH introduces a self-weighted learning strategy to update the weights based on the contributions of different modalities without involving additional hyper-parameters. Experimental results on three benchmark datasets demonstrate the superiority of the proposed SCLRH for cross-modal retrieval.

Exploration of Cultural Teaching Mechanisms of Ideology and Politics Courses under the Background of “Internet Plus”

Abstract As the cradle of cultivating and conveying talents, colleges and universities should innovate the education of ideological and political courses under the background of “Internet+”, focusing on cultivating students’ cultural self-confidence. This paper first explains blockchain-related technology in Internet technology, including the basic principle and structure of blockchain, the consistency Hash algorithm, the consensus mechanism and the credit mechanism of blockchain. Secondly, it discusses the fit between the blockchain and the cultural teaching mechanism of the Civics course, through which blockchain technology can effectively realize the interactive sharing of the cultural teaching resources of the Civics course. Based on the blockchain technology and MVC network structure, an interactive model of cultural teaching resources for Civic and Political Science courses is constructed, which further improves the way students access cultural teaching resources. Finally, a simulation test is conducted to evaluate the interactivity of the cultural teaching resources in the Civics and Political Science course and the effectiveness of the model. The results show that the average write bandwidth of the consistent Hash algorithm is 756.68MBps, and the average read bandwidth is 215.42MBps, and when the total amount of data exceeds 64GB, the relative standard deviation of the Hash algorithm decreases to about 0.15% and remains stable. The cultural teaching mechanism of the Civics course with the introduction of blockchain technology is based on a digital platform, and when the number of network nodes is 150, the delay of the consensus mechanism is 236.73ms, and the throughput reaches about 98 data interactions per second. The cultural teaching mechanism of the Civics and Politics course in the background of the Internet needs to be based on digitization, fully combining with information technology to realize the common construction and sharing of cultural teaching resources and cultivate new-age talents with higher cultural self-confidence for the development of society.

ACSarF: a DRL-based adaptive consortium blockchain sharding framework for supply chain finance

Blockchain technologies have been used to facilitate Web 3.0 and FinTech applications. However, conventional blockchain technologies suffer from long transaction delays and low transaction success rates in some Web 3.0 and FinTech applications such as Supply Chain Finance (SCF). Blockchain sharding has been proposed to improve blockchain performance. However, the existing sharding methods either use a static sharding strategy, which lacks the adaptability for the dynamic SCF environment, or are designed for public chains, which are not applicable to consortium blockchain-based SCF. To address these issues, we propose an adaptive consortium blockchain sharding framework named ACSarF, which is based on the deep reinforcement learning algorithm. The proposed framework can improve consortium blockchain sharding to effectively reduce transaction delay and adaptively adjust the sharding and blockout strategies to increase the transaction success rate in a dynamic SCF environment. Furthermore, we propose to use a consistent hash algorithm in the ACSarF framework to ensure transaction load balancing in the adaptive sharding system to further improve the performance of blockchain sharding in dynamic SCF scenarios. To evaluate the proposed framework, we conducted extensive experiments in a typical SCF scenario. The obtained experimental results show that the ACSarF framework achieves a more than 60% improvement in user experience compared to other state-of-the-art blockchain systems.

SCH: Symmetric Consistent Hashing for cross-modal retrieval

When handling large-scale multimodal data, hashing-based retrieval methods have gained significant attention due to their advantages in terms of low storage consumption and quick query speed. Despite the good performance of supervised hashing techniques, there are still certain limitations that need to be addressed. Firstly, they ignore valuable information such as reconstruction residuals when employing matrix factorization to learn the low-dimensional space. Secondly, most methods map heterogeneous data to the same feature space, which is unreasonable because different modalities possess distinct dimensions and distributions. Lastly, some methods relax constraints in order to optimize discrete hash codes, leading to considerable quantization errors. To solve the above problems, a novel two-step hashing-based technology called Symmetric Consistent Hashing (SCH) is proposed. Specifically, in the first step, two separate latent semantic spaces are learned by leveraging Principal Component Analysis to mitigate principal energy loss and Locality Preserving Projections to preserve local similarity. Then, the latent spaces of different modalities are aligned to facilitate the sharing of high-level semantic information by exploiting modality consistency. Next, hash codes are learned from distinct latent spaces, which are embedded in the second step for learning hash functions through Kernel Logistic Regression. To validate the effectiveness of SCH, extensive experiments on three publicly available benchmark datasets are accomplished. The results demonstrate the superiority of SCH over state-of-the-art hashing baselines, thus confirming the potency of the proposed approach.

Work Together: Correlation-Identity Reconstruction Hashing for Unsupervised Cross-Modal Retrieval

Unsupervised cross-modal hashing has attracted considerable attention to support large-scale cross-modal retrieval. Although promising progresses have been made so far, existing methods still suffer from limited capability on excavating and preserving the intrinsic multi-modal semantics. In this paper, we propose a Correlation-Identity Reconstruction Hashing (CIRH) method to alleviate this challenging problem. We develop a new unsupervised deep cross-modal hash learning framework to model and preserve the heterogeneous multi-modal correlation semantics into both hash codes and functions, and simultaneously, we involve both the hash codes and functions with the descriptive identity semantics. Specifically, we construct a multi-modal collaborated graph to model the heterogeneous multi-modal correlations, and jointly perform the intra-modal and cross-modal semantic aggregation on homogeneous and heterogeneous graph networks to generate a multi-modal complementary representation with correlation reconstruction. Furthermore, an identity semantic reconstruction process is designed to involve the generated representation with identity semantics by reconstructing the input modality representations. Finally, we propose a correlation-identity consistent hash function learning strategy to transfer the modelled multi-modal semantics into the neural networks of modality-specific deep hash functions. Experiments demonstrate the superior performance of the proposed method on both retrieval accuracy and efficiency. We provide our source codes and experimental datasets at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/XizeWu/CIRH</uri> .

Multi-Robot Distributed Communication in Heterogeneous Robotic Systems on 5G Networking

This work aims at new opportunities for the development of the robot communication network. Besides, a robot operation and decision control method driven by Digital Twins (DT) is designed to utilize lasers in a virtual environment. The multi-agent is integrated with the Proximal Policy Optimization (PPO) algorithm, forming the Muti-PPO algorithm for robot path planning. In addition, a consistent hashing algorithm is introduced and optimized as a Weighted Consistent Hash of Multiple Mappings (WCHMM) algorithm with multiple mappings. The experimental results indicate that the average task execution time of the WCHMM algorithm in the case of three bus nodes is 8.09s, which is 32.47 percent and 21.38 percent lower than the task execution time in the case of a single bus node and two bus nodes, respectively. The multi-robot distributed communication system based on Service Oriented Architecture (SOA) gives full play to the various inherent advantages of the multi-robot system.

Erasure Codes for Cold Data in Distributed Storage Systems

Replication and erasure codes are always used for storing large amounts of data in distributed storage systems. Erasure code technology can maximize the storage space of distributed storage systems as well as guaranteeing their availability and reliability, but it will decrease the performance of the system when encoding and decoding. Since cold data do not require high real-time data availability, we focus on the cold data using erasure codes. We propose a new erasure code process named NewLib code based on the Liberation code, which designs the data alignment after stripping the encoding data. The NewLib code improves the performance of reading and writing in the distributed storage systems. At the same time, we developed a node scheduling scheme called N-Schedule, which divides the data nodes into multiple virtual nodes according to the storage space and computing power. The virtual nodes are dispersed into a hash ring by a consistent hash to construct a fully symmetric and decentralized hash ring in order to achieve uniform data distribution and task scheduling. The experimental results show our scheme can improve the performance of the system.

ST-PBFT: An Optimized PBFT Consensus Algorithm for Intellectual Property Transaction Scenarios

For the current Intellectual Property (IP) transaction scenario, consensus nodes need to simultaneously consensus transactions of the same transaction type, resulting in low consensus efficiency, accuracy, and reliability, which seriously hinders the development of intellectual property. Based on the consortium chain, this paper proposes a secure and efficient blockchain-distributed consensus algorithm, ST-PBFT (Shard Transaction Practical Byzantine Fault Tolerance), applied to the IP transaction scenario. The main contributions of ST-PBFT include the following: first, a grouping method based on the principle of consistent hashing is proposed to group consensus nodes, and nodes group consensus, which reduces the complexity of communication. Second, the transaction consensus group can process IP transactions in parallel, which improves the throughput of the algorithm. Third, a node reputation evaluation model is proposed, which can prevent byzantine nodes from being repeatedly elected as primary nodes. The experimental results show that ST-PBFT can significantly improve the consensus efficiency and reliability and reduce consensus latency.

Data Utilization-Based Adaptive Data Management Method for Distributed Storage System in WAN Environment

Recently, research on a distributed storage system that efficiently manages a large amount of data has been actively conducted following data production and demand increase. Physical expansion limits exist for traditional standalone storage systems, such as I/O and file system capacity. However, the existing distributed storage system does not consider where data is consumed and is more focused on data dissemination and optimizing the lookup cost of data location. And this leads to system performance degradation due to low locality occurring in a Wide Area Network (WAN) environment with high network latency. This problem hinders deploying distributed storage systems to multiple data centers over WAN. It lowers the scalability of distributed storage systems to accommodate data storage needs. This paper proposes a method for distributing data in a WAN environment considering network latency and data locality to solve this problem and increase overall system performance. The proposed distributed storage method monitors data utilization and locality to classify data temperature as hot, warm, and cold. With assigned data temperature, the proposed algorithm adaptively selects the appropriate data center and places data accordingly to overcome the excess latency from the WAN environment, leading to overall system performance degradation. This paper also conducts simulations to evaluate the proposed and existing distributed storage methods. The result shows that our proposed method reduced latency by 38% compared to the existing method. Therefore, the proposed method in this paper can be used in large-scale distributed storage systems over a WAN environment to improve latency and performance compared to existing methods, such as consistent hashing.

Performance Enhancement of Distributed System Using HDFS Federation and Sharding

In today's world, 2.5 exabytes of data are generated and processed by the IT industry and different organizations. Processing and managing such a massive volume of Big data is challenging. Hadoop(HDFS) is widely used framework for processing Bigdata. The limitation of HDFS is it can scale only the datanodes and not namenode where the metadata is managed. There is no scaling of name nodes in the present Hadoop architecture, so managing the metadata of the exabyte scale becomes an essential and challenging leading to tightly coupled block storage, namespace scalability, and performance bottleneck issues. The proposed approach dynamic federated metadata management (DFMM) architecture manages the metadata in the name node. DFMM configures multiple name nodes using HDFS federation technique, and metadata is partitioned into numerous name nodes using sharding technique. Sharding manages the metadata using locality-preserving hashing and consistent hashing methods. Locality-preserving hashing stores metadata in the most appropriate name node using the file's parent directory, and metadata is dynamically distributed among Name nodes to maintain load balancing. DFMM also maintains replicas for the high availability and reliability of data. Results showcase the superiority of the proposed DFMM architecture combined with the sharding technique is highly scalable and manages metadata efficiently, and can store up to 1 billion files with a namespace size of 400 GB, enhancing the throughput by 21% and is highly scalable as compared to existing techniques. The comparative study analysis of existing HDFS, and proposed DFMM architecture combined with sharding enhances the performance and manages the metadata more efficiently.