Distributed Parallel Processing Research Articles

Low-frequency Radio Interferometric Array Imaging Pipeline Optimization Based on Distributed Execution Framework

The Square Kilometre Array (SKA) project is an international collaboration to build the world’s largest radio telescope, whose sensitivity and measurement speed will be an order of magnitude higher than those of all current radio telescopes. Radio continuum survey is one of the main observation mode of the SKA, and the establishment of a standard map of the survey area based on continuum imaging will provide an important foundation for subsequent astronomical science. The GaLactic and Extragalactic All-sky Murchison Widefield Array survey eXtended (GLEAM-X) is a project of the SKA pilot telescope Murchison Widefield Array (MWA) in 2018—2020. GLEAM-X is a new radio continuum survey project to be carried out with the MWA Phase II expansion array in 2018—2020. The experience of optimizing the imaging pipeline based on the distributed execution framework will help to solve the problem of massive data processing. In this paper, we describe the process steps of GLEAM-X imaging pipeline, integrate and improve it, and realize parallel processing of multiple pipelines on the China SKA Regional Centre Prototype (CSRC-P), and verify the deployment and test the correctness of the imaging pipeline system using GLEAM-X observation data. The GLEAM-X observations were used to validate the deployment of the imaging pipeline system and test its correctness. Then, to optimize the pipelines and improve the processing efficiency, the Data Activated Liu Graph Engine (DALiuGE) was used to integrate the imaging pipelines into the DALiuGE execution framework to automate the distributed parallel processing of the pipelines. Performance tests and results’ analysis show that the optimized imaging pipeline based on the DALiuGE execution framework has better performance, more flexible adaptability, and scalability than the traditional parallel approach, and can support future large-scale continuum imaging experiments during the first phase of SKA commissioning.

Optimizing Time Consumption for Smartphone-based Distributed Parallel Processing System

Optimizing Time Consumption for Smartphone-based Distributed Parallel Processing System

Takagi–Sugeno fuzzy based power system fault section diagnosis models via genetic learning adaptive GSK algorithm

To effectively deal with the operating uncertainties of protective relays and circuit breakers existing in the power system faults, an improved fault section diagnosis (FSD) method is proposed by using Takagi–Sugeno fuzzy neural networks (T–S FNN). In this method an optimal T–S FNN-based diagnosis model is built with the idea of distributed parallel processing for each section instead of the whole power system. To obtain accurate T–S FNN-based diagnosis models, a genetic learning adaptive gaining-sharing knowledge-based algorithm (GLAGSK) is designed to optimize their structure parameters and consequent parameters. GLAGSK combines an adaptive knowledge ratio and a genetic learning strategy to balance population diversity and convergence speed to boost the optimization ability. After a fault occurs, selective optimal T–S FNN-based diagnosis models are triggered according to the alarm information. They work in parallel to improve the fault diagnosis efficiency. Simulation results of three test systems including an actual fault event show that, compared with other peer algorithms, GLAGSK can obtain more accurate T–S FNN-based diagnosis models with faster global convergence. Besides, compared with the BP and RBF neural networks and other FSD methods, the proposed FSD method based on optimal T–S FNN-based diagnosis models can diagnose different complex faults successfully with higher fault credibility.

FAST Design for Large-Scale Satellite Image Processing

This study proposes a distributed parallel processing system, called the Fast Analysis System for remote sensing daTa(FAST), for large-scale satellite image processing and analysis. FAST is a system that designs jobs in vertices and sequences, and distributes and processes them simultaneously. FAST manages data based on the Hadoop Distributed File System, controls entire jobs based on Apache Spark, and performs tasks in parallel in multiple slave nodes based on a docker container design. FAST enables the high-performance processing of progressively accumulated large-volume satellite images. Because the unit task is performed based on Docker, it is possible to reuse existing source codes for designing and implementing unit tasks. Additionally, the system is robust against software/hardware faults. To prove the capability of the proposed system, we performed an experiment to generate the original satellite images as ortho-images, which is a pre-processing step for all image analyses. In the experiment, when FAST was configured with eight slave nodes, it was found that the processing of a satellite image took less than 30 sec. Through these results, we proved the suitability and practical applicability of the FAST design.

Big Data Analysis of Marketing User Intelligence Information Based on Deep Learning

To solve the problem that the explosive growth of data cannot be effectively analyzed, a big data analysis and prediction system based on deep learning is proposed. The systems proposed in this paper optimize the design of circulatory neural networks using an optimization processing strategy in addition to circulatory neural networks based on a distributed parallel processing framework that studies and analyzes disrupted neural networks during the in-depth study. The results are as follows: using the DCNNPS parallel strategy can effectively improve the training speed of the convolutional neural network to a small extent, and the speed can be improved by more than 50%. Compared with the other two, the network optimized by DCNNPS is less sensitive to data growth in the scenario of a large amount of data and is more suitable for processing a large amount of data. Moreover, with the increase of computing nodes, the acceleration effect is more obvious, the acceleration ratio is higher, and the improvement energy is about 70%. It is proved that the system proposed in this paper can process massive data very effectively and make a great contribution to the development of the industry.

Key Technologies of Media Big Data in-Depth Analysis System Based on 5G Platform

To meet the needs of large-scale users for personalized streaming media services with high speed, low delay, and high quality in a 5G mobile network environment, this paper studies the resource allocation mechanism of streaming media based on a 5G network from the perspective of user demand prediction, which can alleviate the pressure of mobile network, improve the utilization rate of streaming media resources and the quality of user service experience. The augmented reality visualization of large-scale social media data must rely on the computing power of distributed clusters. This paper constructs a distributed parallel processing framework in a high-performance cluster environment, which adopts a loosely coupled organizational structure. Each module can be combined, called, and expanded arbitrarily under the condition of following a unified interface. In this paper, the algebraic method of parallel computing algorithm is innovatively proposed to describe parallel processing tasks and organize and call large-scale data-parallel processing operators, which effectively supports the business requirements of large-scale parallel processing of large-scale spatial social media data and solves the bottleneck of large-scale spatial social media data-parallel processing.

Spatio-Temporal Coding-Based Helicopter Trajectory Planning for Pulsed Neural Membrane System.

For the trajectory planning problem under the nonlinear and strongly coupled characteristics of unmanned helicopters, membrane computing with distributed parallel processing capability is introduced for unmanned helicopter trajectory planning. The global and local spatial information is temporally characterized; the temporal characterization algorithm under mapping information is designed; the hierarchical discriminant regression algorithm is designed based on incremental principal component analysis to realize the process of building and identifying trees in trajectory planning; and the pulsed neural membrane system (PNMS) with spatio-temporal coding function under membrane computing is constructed. Compared with the RRT algorithm in two experimental environments, the original path length, the trimmed path length, the time used to plan the trajectory, and the number of search nodes have different levels of improvement; the feasibility and effectiveness of the PNMS in unmanned helicopter trajectory planning are verified. It expands the theoretical research of membrane computing in the field of optimal control and provides theoretical support for the subsequent application practice.

Supply network design with uncertain demand: Computational cooperative game theory approach using distributed parallel programming

In this work we investigate a supply chain design problem with uncertain final demands for the end products produced by a set of manufacturers. This network of manufacturers has the organizational decisions made internally by pooling resources in a cooperative manner and externally by determining the dominant strategic actors that are characterized by the retailer. A non-linear production game (NLPG) is formulated as a mathematical programming problem to describe coalition formation among the manufacturers based on initial contractual agreements. We show that NLPG is a grand coalition game when demand distribution has an increasing generalized failure rate. The conditions that impact the profit allocation in the game including core set, fairness, stability, superadditivity, least-core and ∊-core are defined. A core allocation solution is generated by using an algorithmic approach. This algorithmic solution is tested in a distributed work station network and resulted in attaining strong computational results for the proposed mathematical programming problem; optimization results with 1000 players are determined in 6 min 40 s on a university computer network using parallel programming with 52 powerful work stations. The numerical results indicated O(log(-N|)) complexity of the algorithmic solution up to 800 players. To the best of our knowledge, this work is the first of its kind in which distributed parallel processing is implemented in a university work station network with distributed parallel programming and processing for solving a production cooperative game.

Analysis method of big data based on K-means algorithm

Efficient and accurate clustering of massive data is the basic requirement of deep excavation for internal knowledge of data in the information era. Firstly, in the distributed parallel processing framework of Hadoop, the Map Reduce model is applied to operate the K-means clustering algorithm, then the genetic algorithm is utilized to transform the clustering problem of K-means algorithm into the solution of extremum to solve the randomness confirmed in cluster center. After testing, this method has achieved the ideal effect in terms of clustering efficiency and clustering precision, and plays certain role in deeply excavating the internal knowledge rule of data.

Development of Distributed Parallel Processing System Using P2P

Development of Distributed Parallel Processing System Using P2P

DiPLIP: Distributed Parallel Processing Platform for Stream Image Processing Based on Deep Learning Model Inference

Recently, as the amount of real-time video streaming data has increased, distributed parallel processing systems have rapidly evolved to process large-scale data. In addition, with an increase in the scale of computing resources constituting the distributed parallel processing system, the orchestration of technology has become crucial for proper management of computing resources, in terms of allocating computing resources, setting up a programming environment, and deploying user applications. In this paper, we present a new distributed parallel processing platform for real-time large-scale image processing based on deep learning model inference, called DiPLIP. It provides a scheme for large-scale real-time image inference using buffer layer and a scalable parallel processing environment according to the size of the stream image. It allows users to easily process trained deep learning models for processing real-time images in a distributed parallel processing environment at high speeds, through the distribution of the virtual machine container.

IOT and cloud computing based parallel implementation of optimized RBF neural network for loader automatic shift control

One of the key issues in automatic shift control of V-type cyclical loaders is determining how to find the best gear for the current conditions according to a certain mapping relation, but this complex and nonlinear mapping is difficult to express by a mathematical relation. However, to solve such nonlinear problems, a radial basis function (RBF) neural network is the best choice. In this paper, a certain type of wheel loader is taken as the research object, and an RBF neural network algorithm based on an improved genetic algorithm (GA) optimization is proposed. The global search ability of the GA is improved by adaptively adjusting the crossover probability and mutation probability. The RBF neural network expansion coefficient is optimized by an improved GA. Using industrial IOT technology, an optimized RBF neural network based on Map-Reduce on a cloud computing cluster is designed. The diesel engine computer and transmission computer on the loader are integrated to achieve dual-processor distributed parallel data processing and calculation. Then the loader automatic variable speed control algorithm model of improved GA optimized RBF neural network based on IOT cloud computing is established. The network model is trained and simulated using real vehicle automatic shift test data. The simulation results show that the improved GA-RBF neural network algorithm can achieve a correct recognition rate of 97.92%. The error matrix norm reaches the minimum value when the algorithm is iterated to the 17th generation. The improved algorithm has the advantages of a high gear recognition rate, fast convergence speed and strong real-time shift performance and is an effective new shift control method. The test results show that the shift boost time is less than 0.15 s and has a certain gradient. Compared with the manual shift process performed in the past, some improvements are achieved in the optimal shift time, shift response speed and shift quality. Compared with the traditional single computer based on serial training RBF neural network learning algorithm, whether it is Great progress has been made in convergence speed, training time, recognition rate, and data processing capabilities. Through the simulation and test, the validity of the intelligent shift control method of the improved GA optimized RBF neural network based on IOT cloud computing is verified. It has better engineering application value.

Dps-MuSyQ: A Distributed Parallel Processing System for Multi-Source Data Synergized Quantitative Remote Sensing Products Producing

With the development of earth observation technologies and the construction of earth observation systems, an increasing amount of remote sensing data are being obtained. These provide the datasets required for research on remote sensing monitoring across large areas. To compensate for the shortcomings of global and large-area temporal monitoring data, synergized computing using multi-source remote sensing data can improve the accuracy and temporal resolution of remote sensing monitoring. However, remote sensing data are drawn from multiple sources and multiple scales, and have a complex structure and large volume; in addition, the nested system architecture of multi-source synergized remote sensing products makes the design of large-scale multi-source synergized remote sensing monitoring systems difficult. In this paper, we describe the design and implementation of a distributed parallel processing system for multi-source data synergized quantitative remote sensing based on a distributed cluster platform. The system integrates the algorithms normalizing more than 30 kinds of data sources and producing 40 quantitative remote sensing products. The system also connects a number of centers for satellite data, serves for several applications, and implements dynamic expansion integration for highly efficient quantitative remote sensing products. The system has produced approximately 50 TB of quantitative remote sensing products, and the application of these data to agriculture, forestry, the environment, and water conservation has resulted in very positive effects.

A Study of Real Time Raster Graphic Visualization Based on Big Data Technology

The grid sizes were designed to provide best quality for raster graphics. The grid data was regionally grouped into blocks and stored by partition, providing a data structure suitable for distributed parallel processing. The whole procedure from data query to grid block sub-graphic generation was fully parallel processed by an independently developed cluster, achieving high performance visualization of pixel-level raster graphics under full-HD screen resolution. The solution provides reference for the construction of similar systems.

DEVELOPMENT OF A HIGH-SPEED VIDEOGRAMMETRIC MEASUREMENT SYSTEM WITH APPLICATION IN LARGE-SCALE SHAKING TABLE TEST

Abstract. The high-speed videogrammetric measurement system, which provides a convenient way to capture three-dimensional (3D) dynamic response of moving objects, has been widely used in various applications due to its remarkable advantages including non-contact, flexibility and high precision. This paper presents a distributed high-speed videogrammetric measurement system suitable for monitoring of large-scale structures. The overall framework consists of hardware and software two parts, namely observation network construction and data processing. The core component of the observation network is high-speed cameras to provide multiview image sequences. The data processing part automatically obtains the 3D structural deformations of the key points from the captured image sequences. A distributed parallel processing framework is adopted to speed up the image sequence processing. An empirical experiment was conducted to measure the dynamics of a double-tube five-layer building structure on the shaking table using the presented videogrammetric measurement system. Compared with the high-accuracy total station measurement, the presented system can achieve a sub-millimeter level of coordinates discrepancy. The 3D deformation results demonstrate the potential of the non-contact high-speed videogrammetric measurement system in dynamic monitoring of large-scale shake table tests.

Ultralow Power Dual Gated Sub-Threshold Oxide Neuristors: An Enabler for Higher Order Neuronal Temporal Correlations

Inspired by neural computing, the pursuit of ultralow power neuromorphic architectures with highly distributed memory and parallel processing capability has recently gained more traction. However, emulation of biological signal processing via artificial neuromorphic architectures does not exploit the immense interplay between local activities and global neuromodulations observed in biological neural networks, and hence are unable to mimic complex biologically plausible adaptive functions like heterosynaptic plasticity and homeostasis. Here, we demonstrate emulation of complex neuronal behaviours like heterosynaptic plasticity, homeostasis, association, correlation and coincidence in a single neuristor via a novel dual-gated architecture. This multiple gating approach allows one gate to capture the effect of local activity correlations and the second gate to represent global neuromodulations, allowing additional modulations which augment their plasticity and enabling higher order temporal correlations at a unitary level. Moreover, the dual-gate operation extends the available dynamic range of synaptic conductance while maintaining symmetry in the weight-update operation, expanding the number of accessible memory states. Finally, operating neuristors in the sub-threshold regime enables synaptic weight changes with high gain, while maintaining ultralow power consumption of the order of femto-Joules.

Ultralow Power Dual-Gated Subthreshold Oxide Neuristors: An Enabler for Higher Order Neuronal Temporal Correlations.

Inspired by neural computing, the pursuit of ultralow power neuromorphic architectures with highly distributed memory and parallel processing capability has recently gained more traction. However, emulation of biological signal processing via artificial neuromorphic architectures does not exploit the immense interplay between local activities and global neuromodulations observed in biological neural networks and hence are unable to mimic complex biologically plausible adaptive functions like heterosynaptic plasticity and homeostasis. Here, we demonstrate emulation of complex neuronal behaviors like heterosynaptic plasticity, homeostasis, association, correlation, and coincidence in a single neuristor via a dual-gated architecture. This multiple gating approach allows one gate to capture the effect of local activity correlations and the second gate to represent global neuromodulations, allowing additional modulations which augment their plasticity, enabling higher order temporal correlations at a unitary level. Moreover, the dual-gate operation extends the available dynamic range of synaptic conductance while maintaining symmetry in the weight-update operation, expanding the number of accessible memory states. Finally, operating neuristors in the subthreshold regime enable synaptic weight changes with high gain while maintaining ultralow power consumption of the order of femto-Joules.

Fibonacci Series based Virtual Machine Selection for Load Balancing in Cloud Computing

The rapid advancement of the internet has given birth to many technologies. Cloud computing is one of the most emerging technology which aim to process large scale data by using the computational capabilities of shared resources. It gives support to the distributed parallel processing. Using cloud computing, we can process data by paying according to its uses which eliminates the requirement of device by individual users. As cloud computing grows, more users get attracted towards it. However, providing an efficient execution time and load distribution is a major challenging issue in the distributed systems. In our approach, weighted round robin algorithm is used and benefits of Fibonacci sequence is combined which results in better execution time than static round robin. Relevant virtual machines are chosen and jobs are assigned to them. Also, number of resources being utilized concurrently is reduced, which leads to resource saving thereby reducing the cost. There is no need to deploy new resources as resources such as virtual machines are already available.

Automatic text summarization within big data frameworks

Data increasing at unabated rates will prove to be challenging for individuals trying to effectively leverage it. This paper describes an approach to automatically summarize text documents by utili...

Scalable distributed data cube computation for large-scale multidimensional data analysis on a Spark cluster

A data cube is a powerful analytical tool that stores all aggregate values over a set of dimensions. It provides users with a simple and efficient means of performing complex data analysis while assisting in decision making. Since the computation time for building a data cube is very large, however, efficient methods for reducing the data cube computation time are needed. Previous works have developed various algorithms for efficiently generating data cubes using MapReduce, which is a large-scale distributed parallel processing framework. However, MapReduce incurs the overhead of disk I/Os and network traffic. To overcome these MapReduce limitations, Spark was recently proposed as a memory-based parallel/distributed processing framework. It has attracted considerable research attention owing to its high performance. In this paper, we propose two algorithms for efficiently building data cubes. The algorithms fully leverage Spark’s mechanisms and properties: Resilient Distributed Top-Down Computation (RDTDC) and Resilient Distributed Bottom-Up Computation (RDBUC). The former is an algorithm for computing the components (i.e., cuboids) of a data cube in a top-down approach; the latter is a bottom-up approach. The RDTDC algorithm has three key functions. (1) It approximates the size of the cuboid using the cardinality without additional Spark action computation to determine the size of each cuboid during top-down computation. Thus, one cuboid can be computed from the upper cuboid of a smaller size. (2) It creates an execution plan that is optimized to input the smaller sized cuboid. (3) Lastly, it uses a method of reusing the result of the already computed cuboid by top-down computation and simultaneously computes the cuboid of several dimensions. In addition, we propose the RDBUC bottom-up algorithm in Spark, which is widely used in computing Iceberg cubes to maintain only cells satisfying a certain condition of minimum support. This algorithm incorporates two primary strategies: (1) reducing the input size to compute aggregate values for a dimension combination (e.g., A, B, and C) by removing the input, which does not satisfy the Iceberg cube condition at its lower dimension combination (e.g., A and B) computed earlier. (2) We use a lazy materialization strategy that computes every combination of dimensions using only transformation operations without any action operation. It then stores them in a single action operation. To prove the efficiency of the proposed algorithms using a lazy materialization strategy by employing only one action operation, we conducted extensive experiments. We compared them to the cube() function, a built-in cube computation library of Spark SQL. The results showed that the proposed RDTDC and RDBUC algorithms outperformed Spark SQL cube().