Distributed Clustering Research Articles

A reliability-aware scheduling algorithm for parallel task executing on cloud computing system

As cloud computing is established on the massive cheap server clusters, it causes compute nodes' software and hardware to go wrong. Different computing nodes and communications links have different failure rate. For the parallel task scheduling problem that cloud users have requirements for deadlines and executing reliability, we put forward to generate all possible execution schemes of a parallel task on a cloud computing system. All the execution schemes are constructed into an execution scheme graph (ESG), in which a path from the start point to end point corresponds to an execution scheme of a parallel task. Based on ESG, we propose the maximum reliability execution scheme solving algorithm MRES that searches the execution schemes which have maximum reliability cost while meeting the parallel task's deadline requirement. The experimental results show that MRES algorithm can effectively improve the executing success rate.

A HDFS dynamic load balancing strategy using improved niche PSO algorithm in cloud storage

A Hadoop distributed file system (HDFS) NameNode dynamic load balancing strategy (NDLBT) using improved niche particle swarm optimisation (PSO) is proposed, which is used to solve the load balancing problem of urban surveillance video big data. Firstly, the relationship between bandwidth consumption and the bit rate of video files, the size of data blocks and access hotspots of online video files are analysed. Then, adaptive backup is realised by dynamic multiple copies of heterogeneous nodes. The dynamic distribution of metadata is implemented to ensure the performance of metadata server clusters. Finally, we propose a new improved niche PSO algorithm to achieve load balancing scheduling. In the experimental scenario where high bandwidth consumption video files are used as service access hotspots, the proposed method, which can reduce the bandwidth peak value of bottleneck nodes in data node clusters by 20%, is superior to the original load balancing method in 90% scenarios.

Parallel cuckoo search for cognitive wireless sensor networks

In cognitive wireless sensor networks (CWSNs), the limited energy of the sensor node is the core defect that restricts its comprehensive network performance. This paper proposes a parallel cuckoo search medoids (PCS-medoids) algorithm to manage the energy consumption in CWSNs efficiently. Firstly, a parallel cuckoo search algorithm (PCS) with communication is proposed to speed up the convergence of CS. Then, the PCS is applied to k-medoids to get cluster heads quickly. Finally, the PCS-medoids is presented to manage the consumption of sensor nodes. First experimental results illustrate that PCS tends to get optimal solutions quickly and accurately compared to CS and PSO. The other experimental results demonstrate that PCS-medoids has advantages over energy management in CWSNs compared to low-energy adaptive clustering hierarchy, LEACH-centralised, and hybrid energy-efficient distributed clustering. Besides, the ad-vantages are more obvious with the increase of sensor nodes in CWSNs.

Energy-Efficient and Fast Data Collection in UAV-Aided Wireless Sensor Networks for Hilly Terrains

Energy-constrained sensor nodes are often deployed in remote, hilly, and hard-to-reach areas for civilian and military purposes. In such wireless sensor networks (WSNs), an unmanned aerial vehicle (UAV) can be used to collect data from the sensor nodes. Low-altitude UAVs can be utilized to reduce the energy consumption of WSNs by optimizing the data collection position. In this study, we designed an energy-efficient and fast data collection (EFDC) scheme in UAV-aided WSNs for hilly areas with the help of a UAV as a data mule. First, we proposed a central bias hybrid energy-efficient distributed clustering algorithm for grouping the sensors. Then, we applied a modified tabu search algorithm to optimize the UAV position for collecting data from a cluster. To achieve fast data collection, we developed the traveling salesman problem with the derived data collection positions and solved it by applying a modified genetic algorithm. Based on our simulation results, the proposed EFDC scheme outperforms the conventional ones in terms of energy consumption, scalability, control overhead, delay, and load balancing.

Parallel cuckoo search for cognitive wireless sensor networks

In cognitive wireless sensor networks (CWSNs), the limited energy of the sensor node is the core defect that restricts its comprehensive network performance. This paper proposes a parallel cuckoo search medoids (PCS-medoids) algorithm to manage the energy consumption in CWSNs efficiently. Firstly, a parallel cuckoo search algorithm (PCS) with communication is proposed to speed up the convergence of CS. Then, the PCS is applied to k-medoids to get cluster heads quickly. Finally, the PCS-medoids is presented to manage the consumption of sensor nodes. First experimental results illustrate that PCS tends to get optimal solutions quickly and accurately compared to CS and PSO. The other experimental results demonstrate that PCS-medoids has advantages over energy management in CWSNs compared to low-energy adaptive clustering hierarchy, LEACH-centralised, and hybrid energy-efficient distributed clustering. Besides, the ad-vantages are more obvious with the increase of sensor nodes in CWSNs.

ИССЛЕДОВАНИЕ ПРОИЗВОДИТЕЛЬНОСТИ РАСПРЕДЕЛЕННОЙ ВИРТУАЛЬНОЙ ПРОГРАММНО-КОНФИГУРИРУЕМОЙ ИНФРАСТРУКТУРЫ ДЛЯ ЗАДАЧ ОБРАБОТКИ БОЛЬШИХ ДАННЫХ

Container-based virtual software-defined virtual infrastructures have become an integral part of the underlying cloud computing engine and are used in a variety of distributed, scalable, resilient systems. But big data analytics tasks are mostly handled by traditional distributed clusters that require initial deployment, careful upgrades, and skilled maintenance. The aim of the work is to study the effectiveness of using software- defined virtual infrastructures based on containers and methods for their rapid deployment according to cloud principles for the implementation of automation platforms for distributed processing of big data. The schemes of Hadoop and Spark architecture deployment based on Docker Swarm and Kubernetes clusters are given. The development of criteria for evaluating the performance of distributed computing for processing big data was also carried out and an experimental study of the work efficiency was carried out.

Federated Adaptive Asynchronous Clustering Algorithm for Wireless Mesh Networks

It is a challenge to generate an accurate machine learning model in a distributed network due to the increased concern in data privacy and high cost in gathering all raw data. This paper presents an adaptive asynchronous distributed clustering algorithm for agents in wireless network to learn the global models, while the privacy is protected. Moreover, the communication cost and clustering quality can be adaptively balanced. The proposed clustering algorithm does not require the number of clusters to be pre-defined. To improve the accuracy of the global model, we propose a bounding boxes based method to fully utilize the shape information of clusters. In addition, we consider different knowledge levels of agent and different requirements about the global model. In experiments on randomly generated network topologies, we demonstrate that methods which do more extensive clustering in each cycle, and which exchange descriptions of cluster shape and density instead of just centroids and data counts, achieve more consistent clustering, in significantly shorter elapsed time. We also show that the proposed methods can learn the same number of clusters as the ground truth when clusters are well separated from each other.

Experimental Setup for Investigating the Efficient Load Balancing Algorithms on Virtual Cloud

Cloud computing has emerged as the primary choice for developers in developing applications that require high-performance computing. Virtualization technology has helped in the distribution of resources to multiple users. Increased use of cloud infrastructure has led to the challenge of developing a load balancing mechanism to provide optimized use of resources and better performance. Round robin and least connections load balancing algorithms have been developed to allocate user requests across a cluster of servers in the cloud in a time-bound manner. In this paper, we have applied the round robin and least connections approach of load balancing to HAProxy, virtual machine clusters and web servers. The experimental results are visualized and summarized using Apache Jmeter and a further comparative study of round robin and least connections is also depicted. Experimental setup and results show that the round robin algorithm performs better as compared to the least connections algorithm in all measuring parameters of load balancer in this paper.

A New Clustering Routing Protocol for Homogeneous Wireless Sensor Networks Powered by Renewable Energy Sources

The technology of wireless sensor networks (WSNs) is in constant development and it made great progress in many applications. One of the most popular problems in WSNs is the limited energy storage power at every sensor node. This paper aims to propose and develop a new distributed clustering algorithm for energy harvesting wireless sensor networks denoted by DEH-WSN (Energy Harvesting for Distributed Clustering Wireless Sensor Networks Protocol) that relies on matching between clustering and energy harvesting in a distributed topology. DEH-WSN uses initial and residual energy capacity of the nodes to choose cluster heads. Simulation results prove that the proposed method increases network lifetime and the effective throughput.

Editorial for special issue on security, trust, and privacy in internet of things: Challenges and solutions

Editorial for special issue on security, trust, and privacy in internet of things: Challenges and solutions

IGenomics: Comprehensive DNA sequence analysis on your Smartphone.

BackgroundFollowing the miniaturization of integrated circuitry and other computer hardware over the past several decades, DNA sequencing is on a similar path. Leading this trend is the Oxford Nanopore sequencing platform, which currently offers the hand-held MinION instrument and even smaller instruments on the horizon. This technology has been used in several important applications, including the analysis of genomes of major pathogens in remote stations around the world. However, despite the simplicity of the sequencer, an equally simple and portable analysis platform is not yet available.ResultsiGenomics is the first comprehensive mobile genome analysis application, with capabilities to align reads, call variants, and visualize the results entirely on an iOS device. Implemented in Objective-C using the FM-index, banded dynamic programming, and other high-performance bioinformatics techniques, iGenomics is optimized to run in a mobile environment. We benchmark iGenomics using a variety of real and simulated Nanopore sequencing datasets of viral and bacterial genomes and show that iGenomics has performance comparable to the popular BWA-MEM/SAMtools/IGV suite, without necessitating a laptop or server cluster.ConclusionsiGenomics is available open source (https://github.com/stuckinaboot/iGenomics) and for free on Apple's App Store (https://apple.co/2HCplzr).

A Quadtree-based synchronization protocol for inter-server game state synchronization

Online games are a fundamental part of the entertainment industry but the current IP infrastructure does not satisfactorily fulfill the needs of these services. The novel networking architecture Named Data Networking (NDN) inherently supports network-level multicast and packet-level security and thereby introduces promising features for online games. In this paper, we propose an NDN-based approach to synchronize game state in a server cluster, a task necessary to allow multiple players in large numbers to play in the same game world. The proposed Quadtree Synchronization Protocol applies NDN’s data-centric nature to decouple the game world from the game servers hosting it. This means that requesting changes of a specific game world region becomes possible without knowing which game server is responsible for the requested region. We use a hierarchic game world structure when requesting data that allows the network to forward requests to the responsible game server without directly addressing it. This region-based naming scheme decouples world regions from servers which eases the management of the game server cluster and allows easier recovery after server failures. In addition, this decoupling allows exchanging information about a geographical region, such as a game world, without knowledge of the other participants changing the world. Such a region-based synchronization mode is not possible to implement with existing protocols. However, it allows building distributed systems that do not require a central server to work. Besides architectural benefits, network emulations show that our protocol increases the efficiency of data transport by utilizing network-level multicast. Our proposed approach can keep up with current protocols which can be used for inter-server game state synchronization.

Lightweight Coordination Patterns for Applications of the Internet of Things

Abstract Applications of the Internet of Things (IoT) are famously known for connecting devices via the internet. The main purpose of IoT systems (wireless or wired) is to connect devices together for data collection, buffering and data gateway. The collected large size of data is often captured from remote sources for automatic data analytics or for direct decision making by its users. This paper applies the programming pattern for Big Data in IoT systems that makes use of lightweight Java methods, introduced in the recently published work on ClientNet Distributed Cluster. Considering Big Data in IoT systems means the sensing of data from different resources, the network of IoT devices collaborating in data collection and processing; and the gateways servers where the resulting big data is supposed to be directed or further processed. This mainly involves resolving the issues of Big Data, i.e., the size and the network transfer speed along with many other issues of coordination and concurrency. The computer network that connects IoT may further include techniques such as Fog and Edge computing that resolve much of the network issues. This paper provides solutions to these problems that occur in wireless and wired systems. The talk is about the ClientNet programming model and its application in IoT systems for orchestration, such as coordination, data communication, device identification and synchronization between the gateway servers and devices. These devices include sensors attached with appliances (e.g., home automations, supply chain systems, light and heavy machines, vehicles, power grids etc.) or buildings, bridges and computers running data processing applications. As described in earlier papers, the introduced ClientNet techniques prevent from big data transfers and streaming that occupy more resources (hardware and bandwidth) and time. The idea is motivated by Big Data problems that make it difficult to collect it from different resources through small devices and then redirecting it. The proposed programming model of ClientNet Distributed Cluster stores Big Data on the nearest server coordinated by the nearest coordinator. The gateways and the systems that run analytics programs communicate by running programs from other computers when it is essentially required. This makes it possible to let Big Data rarely move across a communication network and allow only the source code to move around the network. The given programming model greatly simplifies data communication overheads, communication patterns among devices, networks and servers.

A Scalable Virtualized Server Cluster Providing Sensor Data Storage and Web Services

With the rapid development of the Internet of Things (IoT) technology, diversified applications deploy extensive sensors to monitor objects, such as PM2.5 air quality monitoring. The sensors transmit data to the server periodically and continuously. However, a single server cannot provide efficient services for the ever-growing IoT devices and the data they generate. This study bases on the concept of symmetry of architecture and quantities in system design and explores the load balancing issue to improve performance. This study uses the Linux Virtual Server (LVS) and virtualization technology to deploy a virtual machine (VM) cluster. It consists of a front-end server, also a load balancer, to dispatch requests, and several back-end servers to provide services. These receive data from sensors and provide Web services for browsing real-time sensor data. The Hadoop Distributed File System (HDFS) and HBase are used to store the massive amount of received sensor data. Because load-balancing is critical for resource utilization, this study also proposes a new load distribution algorithm for VM-based server clusters that simultaneously provide multiple services, such as sensor services and Web service. It considers the aggregate load of all back-end servers on the same physical server that provides multiple services. It also considers the difference between physical machines and VMs. Algorithms such as those for LVS, which do not consider these factors, can cause load imbalance between physical servers. The experimental results demonstrate that the proposed system is fault tolerant, highly scalable, and offers high availability and high performance.

Privacy-Preserving Distributed Clustering for Electrical Load Profiling

Electrical load profiling supports retailers in identifying consumer categories for customizing tariff design. However, each retailer only has access to the data of the customers it serves. Centralized joint clustering on retailers' union load dataset either enables the identification of more types of users that allows to design more customized retail plans, or informs whether each retailer already has a sufficiently broad customer base. However, the centralized clustering requires access to the confidential data of retailers. This may cause privacy issues among retailers, because retailers can not or do not want to share their confidential information with others. To tackle this issue, we propose a privacy-preserving distributed clustering framework by developing a privacy-preserving accelerated average consensus (PP-AAC) algorithm. Using the proposed framework, we modify several commonly used clustering methods, including k-means, fuzzy C-means, and Gaussian mixture model, to provide privacy-preserving distributed clustering methods. In this way, the clustering on retailers' union dataset can be achieved only by local calculations and information sharing between neighboring retailers without sacrificing privacy. The correctness, privacy-preserving property, time-saving feature, and robustness to random communication failures of the proposed methods are verified using a real-world Irish residential dataset.

Using independent resource allocation strategies to solve conflicts of Hadoop distributed architecture in virtualization

The strong computing power of Cloud computing enables rapid processing of mass data. Hadoop, the most extensively-distributed architecture among Cloud-computing platforms, uses the MapReduce programming model in a server cluster to separate the applications into many small parts in order to conduct arithmetic processing of big data. Virtualization technology renders the various computer physical resources into abstract virtual resources; thus, computer resources are used more flexibly to simplify management. In practice, Hadoop is frequently combined with virtualization technology to reduce handling costs; however, the virtualized Resource Pooling characteristic results in hardware access conflicts and performance degradation. This study revealed where virtualization degrades read and write capabilities by comparison of the substantialistic architecture and virtualized architecture, and strategies for solving conflicts in virtualization are presented. An optimized virtualized Hadoop architecture was strategically designed; conflicts were gradually solved, and the performance of the virtualized Hadoop was enhanced. The experimental results proved that the strategies presented solved the conflicts while maintaining the fault tolerance, processing performance, and expandability of Hadoop.

High-Throughput Bin Packing: Scheduling Jobs With Random Resource Demands in Clusters

We consider a natural scheduling problem which arises in many distributed computing frameworks. Jobs with diverse resource demands (e.g. memory requirements) arrive over time and must be served by a cluster of servers. To improve throughput and delay, the scheduler can pack as many jobs as possible in each server, however the sum of the jobs' resource demands cannot exceed the server's capacity. Motivated by the increasing complexity of workloads in shared clusters, we consider a setting where jobs' resource demands belong to a very large set of diverse types, or in the extreme case even infinitely many types, i.e. resource demands are drawn from a general unknown distribution over a possibly continuous support. The application of classical scheduling approaches that crucially rely on a predefined finite set of types is discouraging in this high (or infinite) type setting. We first characterize a fundamental limit on the maximum throughput in such setting. We then develop oblivious scheduling algorithms, based on Best-Fit and Universal Partitioning, that have low complexity and can achieve at least 1/2 and 2/3 of the maximum throughput respectively, without the knowledge of the resource demand distribution. Extensive simulation results, using both synthetic and real traffic traces, are presented to verify the performance of our algorithms.

Client-Aware Negotiation for Secure and Efficient Data Transmission

In Wireless Sensor Networks (WSNs), server clusters, and other systems requiring secure transmission, the overhead of data encryption and transmission is often not negligible. Unfortunately, a conflict exists between security and efficiency in processing data. Therefore, this paper proposes a strategy to overcome this conflict, called Client-Aware Negotiation for Secure and Efficient Data Transmission (CAN-SEAT). This strategy allows a client with different security transmission requirements to use the appropriate data security transmission without modifying the client. Two methods are designed for different clients. The first method is based on two-way authentication and renegotiation. After handshakes, the appropriate data security transmission scheme is selected according to the client requirements. Another method is based on redirection, which can be applied when the client does not support two-way authentication or renegotiation. For the characteristics of different architecture, this paper classifies and discusses symmetric key algorithms, asymmetric key algorithms, and hardware encryption instructions. In four application scenarios, the CAN-SEAT strategy is tested. Compared with the general transmission strategy, when only software encryption is used, the data processing and transmission cost can be reduced by 89.41% in the best case and by 15.40% in the worst case. When supporting hardware encryption, the cost can be reduced by 85.30% and 24.63%, respectively. A good effect was produced on the experimental platforms XiLinx, FT-2000+, and Intel processors. To the best of our knowledge, for Client-Aware Negotiation (CAN), this is the first method to be successfully deployed on a general system. CAN-SEAT can be easily combined with other energy-efficient strategies.

Tuning the Water Desalination Performance of Graphenic Layered Nanomaterials by Element Doping and Inter-Layer Spacing* *Supported by the National Natural Science Foundation of China (Grant No. 11874238), the Basic Research Project of Natural Science Foundation of Shandong Province (Grant No. ZR2018MA034), and Collaborative Innovation Funds of Shandong Normal University.

Through atomic molecular dynamics simulations, we investigate the performance of two graphenic materials, boron (BC3) and nitrogen doped graphene (C3N), for seawater desalination and salt rejection, and take pristine graphene as a control. Effects of inter-layer separation have been explored. When water is filtered along the transverse directions of three-layered nanomaterials, the optimal inter-layer separation is 0.7–0.9 nm, which results in high water permeability and salt obstruction capability. The water permeability is considerably higher than porous graphene filter, and is about two orders of magnitude higher than commercial reverse osmosis (RO) membrane. By changing the inter-layer spacing, the water permeability of three graphenic layered nanomaterials follows an order of C3N ≥ GRA > BC3 under the same working conditions. Amongst three nanomaterials, BC3 is more sensitive to inter-layer separation which offers a possibility to control the water desalination speed by mechanically changing the membrane thickness. This is caused by the intrinsic charge transfer inside BC3 that results in periodic distributed water clusters around the layer surface. Our present results reveal the high potentiality of multi-layered graphenic materials for controlled water desalination. It is hopeful that the present work can guide design and fabrication of highly efficient and tunable desalination architectures.

Generalized Cost-Based Job Scheduling in Very Large Heterogeneous Cluster Systems

We study job assignment in large, heterogeneous resource-sharing clusters of servers with finite buffers. This load balancing problem arises naturally in today's communication and big data systems, such as Amazon Web Services, Network Service Function Chains, and Stream Processing. Arriving jobs are dispatched to a server, following a load balancing policy that optimizes a performance criterion such as job completion time. Our contribution is a randomized Cost-Based Scheduling (CBS) policy in which the job assignment is driven by general cost functions of the server queue lengths. Beyond existing schemes, such as the Join the Shortest Queue (JSQ), the power of d or the SQ(d) and the capacity-weighted JSQ, the notion of CBS yields new application-specific policies such as hybrid locally uniform JSQ. As today's data center clusters have thousands of servers, exact analysis of CBS policies is tedious. In this article, we derive a scaling limit when the number of servers grows large, facilitating a comparison of various CBS policies with respect to their transient as well as steady state behavior. A byproduct of our derivations is the relationship between the queue filling proportions and the server buffer sizes, which cannot be obtained from infinite buffer models. Finally, we provide extensive numerical evaluations and discuss several applications including multi-stage systems.