Distributed Processing System Research Articles

GraphTune: An Efficient Dependency-Aware Substrate to Alleviate Irregularity in Concurrent Graph Processing

With the increasing need for graph analysis, massive Concurrent iterative Graph Processing (CGP) jobs are usually performed on the common large-scale real-world graph. Although several solutions have been proposed, these CGP jobs are not coordinated with the consideration of the inherent dependencies in graph data driven by graph topology. As a result, they suffer from redundant and fragmented accesses of the same underlying graph dispersed over distributed platform, because the same graph is typically irregularly traversed by these jobs along different paths at the same time. In this work, we develop GraphTune , which can be integrated into existing distributed graph processing systems, such as D-Galois, Gemini, PowerGraph, and Chaos, to efficiently perform CGP jobs and enhance system throughput. The key component of GraphTune is a dependency-aware synchronous execution engine in conjunction with several optimization strategies based on the constructed cross-iteration dependency graph of chunks. Specifically, GraphTune transparently regularizes the processing behavior of the CGP jobs in a novel synchronous way and assigns the chunks of graph data to be handled by them based on the topological order of the dependency graph so as to maximize the performance. In this way, it can transform the irregular accesses of the chunks into more regular ones so that as many CGP jobs as possible can fully share the data accesses to the common graph. Meanwhile, it also efficiently synchronizes the communications launched by different CGP jobs based on the dependency graph to minimize the communication cost. We integrate it into four cutting-edge distributed graph processing systems and a popular out-of-core graph processing system to demonstrate the efficiency of GraphTune. Experimental results show that GraphTune improves the throughput of CGP jobs by 3.1∼6.2, 3.8∼8.5, 3.5∼10.8, 4.3∼12.4, and 3.8∼6.9 times over D-Galois, Gemini, PowerGraph, Chaos, and GraphChi, respectively.

A Study on Aspect of Artificial Neural Networks for Machine Learning

Artificial Neural Network (ANN) uses the processing of the brain as a basis to develop algorithms that can be used to model complex patterns and prediction problems. In our brain, there are billions of cells called neurons, which processes information in the form of electric signals. External information/stimuli is received by the dendrites of the neuron, processed in the neuron cell body, converted to an output and passed through the Axon to the next neuron. The next neuron can choose to either accept it or reject it depending on the strength of the signal. Neural network architecture is presented as one approach to the design and implementation of intelligent control systems. Neural networks can be considered as massively parallel distributed processing systems with the potential for ever-improving performance through dynamical learning. The nomenclature and characteristics of neural networks are outlined. Two simple examples are presented to illustrate applications to control systems: one is fault isolation mapping, and the other involves optimization of a Hopfield network that defines a clock less analog-to-digital conversion. Deep learning is one of the types of artificial intelligence and machine learning; basically, deep learning behaves like a human brain, or we can say that it processes the data like the human brain. In deep learning, we collect data from different resources in different complex patterns, such as photos and text, and make predictions as well. As we are also able to automate the different tasks which require the human brain.

Cars logo recognition by using of backpropagation neural networks

Numerous developments in creating “intelligent” programs have been made, some of which have drawn inspiration from biological neural networks. Researchers are developing artificial neural networks (ANNs) from various scientific fields to address multiple issues in prediction, control, and optimization. Many s, densely connected processors make up artificial neural networks, which can be considered parallel and distributed processing systems. This paper uses a back propagation ANN algorithm to recognize cars' logos. Sixty-eight images are used, which represents the logo of vehicles and their distortion and rotation, to simulate human eye recognition. The proposed design is trained by using a supervised learning algorithm. The designed system successfully validates the car logos by 99.6%, which consider a high percentage in such a field. Finally, the implemented system presents a real-life application of ANN.

Exploring data merging methods for a distributed processing system

The ALICE experiment at the CERN LHC (Large Hadron Collider) is undertaking a major upgrade during the LHC Long Shutdown 2 in 2019-2021, which includes a new computing system called O2 (Online-Offline). The raw data input from the ALICE detectors will increase a hundredfold, up to 3.5 TB/s. By reconstructing the data online, it will be possible to compress the data stream down to 100 GB/s before storing it permanently.The O2 software is a message-passing system. It will run on approximately 500 computing nodes performing reconstruction, compression, calibration and quality control of the received data stream. As a direct consequence of having a distributed computing system, locally generated data might be incomplete and could require merging to obtain complete results.This paper presents the O2 Mergers, the software designed to match and combine partial data into complete objects synchronously to data taking. Based on a detailed study and results of extensive benchmarks, a qualitative and quantitative comparison of different merging strategies considered to reach the final design and implementation of the software is discussed.

An unsupervised learning-guided multi-node failure-recovery model for distributed graph processing systems

An unsupervised learning-guided multi-node failure-recovery model for distributed graph processing systems

Simulador para la evaluación de algoritmos para la gestión de recursos compartidos en sistemas distribuidos

In distributed processing systems, especially when processes constitute collaborative groups, it is necessary that, when allocating resources to processes, decisions are made based on agreements regarding access to resources; decisions may be related to the performance of a certain activity that requires or not the synchronization of processes, requiring the use of shared resources in the mode of mutual exclusion. Thus, the following question arises: What are the decision models and aggregation operators that will have to be generated by incorporating the global cognitive perspective to the classical models to make more intelligent decision making in the management of resources and groups of processes taking into account self-regulation? How will the algorithms of the different decision models be implemented? How to validate the new proposed algorithms by comparing them with each other and with traditional algorithms? For this, it is necessary to have a simulator that implements the traditional and the new proposed algorithms and allows to observe their behavior and results under different types of workloads. Classical models are considered for accessing shared resources in the mutual exclusion mode using critical regions to the centralized algorithm, the distributed algorithm of Lamport, Ricart and Agrawala, the token ring algorithm, among others. A prototype simulator is presented to evaluate the performance of the proposed new decision models and aggregation operators against the main traditional models.

Construction Means of Soil Microbial Synusiologic Network Based on ANN.

With the construction of synusiologic civilization and synusiologic environmental protection entering a new era driven by data, the breadth and depth of application of the DM technique in the domain of synusiologic environmental protection are constantly strengthened. If reasonable planning is not carried out in the process of social construction, it will cause unpredictable damage to the synusiologic environment. However, traditional synusiologic planning means too much human interference, and there are still some shortcomings in accuracy and operability, which means they cannot guide synusiologic construction well. In order to analyze the contribution of the soil nutrient data to soil fertility and dig out the knowledge describing soil fertility, this paper studies the construction means of soil microbial synusiologic network by ANN. By simulating the learning, memorizing, and processing problems of human brain neurons, the artificial network establishes a parallel distributed processing system computing DMG model with a large number of connections, which can quickly acquire knowledge from the outside world and store and process it and respond to the changes in the external environment in time. According to the research in this paper, the network performance of this algorithm is 18% better than that of the traditional algorithm, and it is suitable to be widely put into practice.

Wukong+G: Fast and Concurrent RDF Query Processing Using RDMA-Assisted GPU Graph Exploration

RDF graph has been increasingly used to store and represent information shared over the Web, including social graphs and knowledge bases. With the increasing scale of RDF graphs and the concurrency level of SPARQL queries, current RDF systems are confronted with inefficient concurrent query processing on massive data parallelism. The situation becomes more severe in the face of data-intensive queries (aka heavy query), which usually lead to suboptimal response time (latency) as well as throughput collapse. In this article, we present Wukong+G, the first graph-based distributed RDF query processing system that efficiently exploits the hybrid parallelism of CPU and GPU. Wukong+G is made fast and concurrent with four key designs. First, Wukong+G tames massive random memory accesses in graph exploration by efficiently mapping data between CPU and GPU for latency hiding, including a set of techniques like query-aware prefetching, pattern-aware pipelining and fine-grained swapping. Second, Wukong+G scales up by introducing a GPU-friendly RDF store to support RDF graphs exceeding GPU memory size, by using techniques like predicate-based grouping, pairwise caching and look-ahead replacing to narrow the gap between host and device memory scale. Third, Wukong+G scales out through a communication layer that decouples the transferring process for query metadata and intermediate results, and further leverages both native and GPUDirect RDMA to enable efficient communication on a CPU/GPU cluster. Finally, Wukong+G simultaneously runs multiple queries on a single GPU to improve overall throughput and fully exploits hardware heterogeneity (CPU/GPU) by scheduling a single query on CPU and GPU adaptively. We have implemented Wukong+G by extending a state-of-the-art distributed RDF store (i.e., Wukong) with distributed GPU support. Evaluation on a heterogeneous CPU/GPU cluster with RDMA-capable network shows that Wukong+G outperforms Wukong by up to 9.0× (from 2.3×) and scales well on 10 GPU cards for heavy queries. Wukong+G can also improve both latency and throughput by more than one order of magnitude when facing hybrid workloads.

Ship Image Classification Based on the Architecture of Imaging Spectral Data Distributed Processing System

Ship Image Classification Based on the Architecture of Imaging Spectral Data Distributed Processing System

Quality-related monitoring of distributed process systems using dynamic concurrent partial least squares

Process monitoring is critical to ensure process safety and maintain quality stabilization. Currently, a large-scale industrial process commonly consists of multiple operation units or manufacturing plants, and a huge number of variables that reflect operation conditions can be collected easily. The conventional monitoring methods usually utilize all process variables to model, which will submerge the correlation between the process and quality variables. In this paper, a new dynamic concurrent partial least squares (DCPLS) monitoring scheme based on variable importance in the projection (VIP) is proposed for large-scale processes. Firstly, process variables are firstly partitioned into quality-related and weakly quality-related space based on their VIP value. Then, DCPLS model is used to monitor abnormal events in these two spaces, respectively. Finally, fusing the statistics of these two spaces through support vector data description to provide an overall indication. The proposed distributed process monitoring scheme not only automatically realizes the division of variables but also characterizes the dynamic information of data. The monitoring results of the Tennessee Eastman process indicate that the proposed VIP-DCPLS method is more efficient than other monitoring methods.

II. Your Present Moment in Spacetime

This essay extends the argument begun in "Why Quantum Mechanics Makes Sense," exploring the conditions under which a physical world can define and communicate information. I argue that like the structure of quantum physics, the principles of Special and General Relativity can be understood as reflecting the requirements of a universe in which things are observable and measurable. I interpret the peculiar hyperbolic structure of spacetime not as the static, four-dimensional geometry of an unobservable "block universe", but as the background metric of an evolving web of communicated information that we, along with all our measuring instruments and recording devices, actually experience in our local "here and now." Our relativistic universe is conceived as a parallel distributed processing system, in which a common objective reality is constantly being woven out of many kinds of facts determined separately in countless local measurement-contexts.

Structured Data Processing Method and Distributed Processing System Construction

Structured Data Processing Method and Distributed Processing System Construction

TraNCE

Nested relational query languages have long been seen as an attractive tool for scenarios involving large hierarchical datasets. There has been a resurgence of interest in nested relational languages. One driver has been the affinity of these languages for large-scale processing platforms such as Spark and Flink. This demonstration gives a tour of TraNCE, a new system for processing nested data on top of distributed processing systems. The core innovation of the system is a compiler that processes nested relational queries in a series of transformations; these include variants of two prior techniques, shredding and unnesting, as well as a materialization transformation that customizes the way levels of the nested output are generated. The TraNCE platform builds on these techniques by adding components for users to create and visualize queries, as well as data exploration and notebook execution targets to facilitate the construction of large-scale data science applications. The demonstration will both showcase the system from the viewpoint of usability by data scientists and illustrate the data management techniques employed.

A Clustered Approach for Load Balancing in Distributed Systems

Distributed systems are increasingly becoming the dominant and rapidly expanding computational paradigm of tomorrow. A cluster is really a form of parallel or distributed processing system that consists of a set of intertwined stand-alone machines that function together like truly coherent computing and storage resources with a single system image (SSI) which means that perhaps the clusters are viewed as a single platform by the consumers. Global resource management, on the other hand, poses several concerns due to the sheer complexity and range of tools, as well as the need for user accountability. The possible advantages of load balancing in addressing the occasional congestion faced by some nodes when everyone else is idle or congested are widely agreed on a level of performance. This is also widely acknowledged that neither specific load balancing algorithm can adequately address evolving device characteristics and complex capacity management in a distributed ecosystem. To have a systematic approach and also in distributed systems, a proposed approach is created for a holistic view of element load balancing and also the qualities features of load balancing algorithms. The nomenclature has been expanded. In order for adaptive algorithms to understand the problem and manner of prefixing resilience along with different components in distributed systems, they must first recognize the concerns. In addition, a proposed approach is specified. The much more effective load balancing techniques and the modeling hypotheses used in prior load balancing experiments are established through a study of related research. We consider the most appropriate load balancing algorithm and optimum metrics for parameter estimation of the algorithm as a consequence of and output of this assessment for a range of formulations of resulting goals, distributed system features, and workload balancing framework.

RGraph: Asynchronous graph processing based on asymmetry of remote direct memory access

SummaryThe scale of real‐world graphs is constantly growing. To deal with large‐scale graphs, distributed graph processing has attracted much research efforts. Existing distributed graph processing systems are commonly built on traditional TCP/IP communication stack, which leads to network bottleneck because of low bandwidth and heavy kernel stack operations. Meanwhile, in real power‐law graphs, the average number of mirror vertices after graph partitioning is very large, resulting in significant communication overhead among nodes. The emerging high‐performance Remote Direct Memory Access (RDMA) network has the features of low latency, high bandwidth, and low CPU overhead, which brings new opportunities for distributed graph processing systems. Existing RDMA‐assisted graph processing systems focus on synchronous execution, which imposes barriers between consecutive iterations. Synchronous execution transfers bulk data among nodes and thus only needs a small number of network transfers. However, synchronous execution is usually less efficient than asynchronous execution because of bulk synchronization. Asynchronous execution accelerates graph processing by eliminating barriers, which in turn requires to transfer a large amount of small size data. In this paper, we propose RGraph, an RDMA‐assisted asynchronous distributed graph processing system. RGraph distributes edges into two parts to isolate master and mirror vertices. RGraph exploits the asymmetry of RDMA to accelerate the one‐to‐many communication between master and mirror vertices. We implement RGraph on top of PowerGraph and conduct comprehensive experiments with large‐scale real graphs to evaluate its performance. Results show that compared to existing designs, RGraph reduces the execution time by up to 81%.

Dynamic Scheduling Model of Rail-Guided Vehicle (RGV) Based on Genetic Algorithms in the Context of Mobile Computing

Track guidance vehicle (RGV) is widely used in logistics warehousing and intelligent workshop, and its scheduling effectiveness will directly affect the production and operation efficiency of enterprises. In practical operation, central information system often lacks flexibility and timeliness. By contrast, mobile computing can balance the central information system and the distributed processing system, so that useful, accurate, and timely information can be provided to RGV. In order to optimize the RGV scheduling problem in uncertain environment, a genetic algorithm scheduling rule (GAM) using greedy algorithm as the genetic screening criterion is proposed in this paper. In the experiment, RGV scheduling of two-step processing in an intelligent workshop is selected as the research object. The experimental results show that the GAM model can carry out real-time dynamic programming, and the optimization efficiency is remarkable before a certain threshold.

Estimating Delays in Switched Ethernet Networks on board of Aerospace Vehicles

Since the year 2000, several studies on bounding end-to-end delays in switched Ethernet topologies required in the design of distributed processing systems on board of aerospace vehicles were made public. These studies presented various approaches for dealing with the non-deterministic behavior of non-synchronized network traffic while striving to avoid overestimation. One of them described an effect observed in frame-based network transmissions and named it “serialization”. This effect is used in this paper with the objective of constructing a new low-complexity method for estimating the worst case delay experienced by a frame crossing a switched Ethernet. This method results from the combination of two new propositions which inherit concepts originally formulated by schedulability analysis methods for multitask processing systems. Its implementation is shown on a case study using a step-by-step procedure and the results obtained are favorably compared with one previously published work.

Coded Computing for Distributed Graph Analytics

Performance of distributed graph processing systems significantly suffers from 'communication bottleneck' as a large number of messages are exchanged among servers at each step of the computation. Motivated by graph based MapReduce, we propose a coded computing framework that leverages computation redundancy to alleviate the communication bottleneck in distributed graph processing. We develop a novel 'coding' scheme that systematically injects structured redundancy in computation phase to enable 'coded' multicasting opportunities during message exchange between servers, reducing communication load substantially in large-scale graph processing. For theoretical analysis, we consider random graph models, and prove that our proposed scheme enables an (asymptotically) inverse-linear trade-off between 'computation load' and 'average communication load' for two popular random graph models -- Erdos-Renyi model, and power law model. Particularly, for a given computation load r, (i.e. when each graph vertex is carefully stored at r servers), the proposed scheme slashes the average communication load by (nearly) a multiplicative factor of r. For the Erdos-Renyi model, our proposed scheme is optimal asymptotically as the graph size increases by providing an information-theoretic converse. To illustrate the benefits of our scheme in practice, we implement PageRank over Amazon EC2, using artificial as well as real-world datasets, demonstrating significant gains over conventional PageRank. We also specialize our scheme and extend our theoretical results to two other random graph models -- random bi-partite model, and stochastic block model. They asymptotically enable inverse-linear trade-offs between computation and communication loads in distributed graph processing for these popular random graph models as well. We complement the achievability results with converse bounds for both of these models.

Data Sampling methods in the ALICE O2 distributed processing system

The ALICE experiment at the CERN LHC focuses on studying the quark-gluon plasma produced by heavy-ion collisions. Starting from 2021, it will see its input data throughput increase a hundredfold, up to 3.5 TB/s. To cope with such a large amount of data, a new online-offline computing system, called O2, will be deployed. It will synchronously compress the data stream by a factor of 35 down to 100 GB/s before storing it permanently.One of the key software components of the system will be the data Quality Control (QC). This framework and infrastructure is responsible for all aspects related to the analysis software aimed at identifying possible issues with the data itself, and indirectly with the underlying processing done both synchronously and asynchronously. Since analyzing the full stream of data online would exceed the available computational resources, a reliable and efficient sampling will be needed. It should provide a few percent of data selected randomly in a statistically sound manner with a minimal impact on the main dataflow. Extra requirements include e.g. the option to choose data corresponding to the same collisions over a group of computing nodes.In this paper the design of the O2 Data Sampling software is presented. In particular, the requirements for pseudo-random number generators to be used for sampling decisions are highlighted, as well as the results of the benchmarks performed to evaluate different possibilities. Finally, a large scale test of the O2 Data Sampling is reported.

PStream: A Popularity-Aware Differentiated Distributed Stream Processing System

Real-world stream data with skewed distributions raises unique challenges to distributed stream processing systems. Existing stream workload partitioning schemes usually use a “one size fits all” design, which leverages either a shuffle grouping or a key grouping strategy for partitioning the stream workloads among multiple processing units, leading to notable problems of unsatisfied system throughput and processing latency. In this article, we show that the key grouping based schemes result in serious load imbalance and low computation efficiency in the presence of data skewness while the shuffle grouping schemes are not scalable in terms of memory space. We argue that the key to efficient stream scheduling is the popularity of the stream data. We propose PStream, a popularity-aware differentiated distributed stream processing system which assigns the hot keys using shuffle grouping while assigns rare ones using key grouping. PStream leverages a novel light-weighted probabilistic counting scheme for identifying the currently hot keys in dynamic real-time streams. The scheme is extremely efficient in computation and memory consumption, so that the predictor based on it can be well integrated into processing instances in the system. We further design an adaptive threshold configuration scheme, which can quickly adapt to the dynamical popularity changes in highly dynamical real-time streams. We implement PStream on top of Apache Storm and conduct comprehensive experiments using large-scale traces from real-world systems to evaluate the performance of this design. Results show that PStream achieves a 2.3× improvement in terms of processing throughput and reduces the processing latency by 64 percent compared to state-of-the-art designs.