Distributed Programming Model Research Articles

Comparative Analysis of Skew-Join Strategies for Large-Scale Datasets with MapReduce and Spark

In the era of data deluge, Big Data gradually offers numerous opportunities, but also poses significant challenges to conventional data processing and analysis methods. MapReduce has become a prominent parallel and distributed programming model for efficiently handling such massive datasets. One of the most elementary and extensive operations in MapReduce is the join operation. These joins have become ever more complex and expensive in the context of skewed data, in which some common join keys appear with a greater frequency than others. Some of the reduction tasks processing these join keys will finish later than others; thus, the benefits of parallel computation become meaningless. Some studies on the problem of skew joins have been conducted, but an adequate and systematic comparison in the Spark environment has not been presented. They have only provided experimental tests, so there is still a shortage of representations of mathematical models on which skew-join algorithms can be compared. This study is, therefore, designed to provide the theoretical and practical basics for evaluating skew-join strategies for large-scale datasets with MapReduce and Spark—both analytically with cost models and practically with experiments. The objectives of the study are, first, to present the implementation of prominent skew-join algorithms in Spark, second, to evaluate the algorithms by using cost models and experiments, and third, to show the advantages and disadvantages of each one and to recommend strategies for the better use of skew joins in Spark.

Data Analysis Method of Intelligent Analysis Platform for Big Data of Film and Television

Based on cloud computing and statistics theory, this paper proposes a reasonable analysis method for big data of film and television. The method selects Hadoop open source cloud platform as the basis, combines the MapReduce distributed programming model and HDFS distributed file storage system and other key cloud computing technologies. In order to cope with different data processing needs of film and television industry, association analysis, cluster analysis, factor analysis, and K-mean + association analysis algorithm training model were applied to model, process, and analyze the full data of film and TV series. According to the film type, producer, production region, investment, box office, audience rating, network score, audience group, and other factors, the film and television data in recent years are analyzed and studied. Based on the study of the impact of each attribute of film and television drama on film box office and TV audience rating, it is committed to the prediction of film and television industry and constantly verifies and improves the algorithm model.

DisCANTree: A Distributed Algorithm for Incremental Frequent Itemset Mining based on MapReduce

Frequent itemset mining is one of the most important data mining tasks. Classical frequent itemset mining algorithms need to store data in a centralized way and run in a batch way, which cannot meet the requirements of fast updating big data mining. In this paper, we propose a distributed incremental frequent itemset mining algorithm, DisCANTree, which uses CANTree to store the conditional database, achieves the load balance between nodes by grouping all items, updates the new transaction to the existing CANTree to avoid the load of tree reconstruction, and uses the efficient FPGrowth algorithm to mine CANTree to generate frequent itemsets. The popular distributed programming model MapReduce and its open source system Hadoop are used to implement the DisCANTree algorithm. The experimental results show that the DisCANTree algorithm has more advantages than the most popular PFP algorithm in performance as well as the number of transferred records between nodes, and especially suits for the fast updating sparse big data.

Programmable Logic Controllers in the Context of Industry 4.0

Programmable logic controllers (PLCs) are an established platform, widely used throughout industrial automation but poorly understood among researchers. This article gives an overview of the state of the practice, explaining why this settled technology persists throughout industry and presenting a critical analysis of the strengths and weaknesses of the dominant programming styles for today's PLC-based automation systems. We describe the software execution patterns that are standardized loosely in IEC 61131-3. We identify opportunities for improvements that would enable increasingly complex industrial automation applications while strengthening safety and reliability. Specifically, we propose deterministic, distributed programming models that embrace explicit timing, event-triggered computation, and improved security.

Optimizing parallel particle tracking in Brownian motion using machine learning

In this paper, we present a generic, scalable and adaptive load balancing parallel Lagrangian particle tracking approach in Wiener type processes such as Brownian motion. The approach is particularly suitable in problems involving particles with highly variable computation time, like deposition on boundaries that may include decay, when particle lifetime obeys exponential distribution. At first glance, Lagranginan tracking is highly suitable for a distributed programming model due to the independence of motion of separate particles. However, the commonly employed Decomposition Per Particle (DPP) method, where each process is in charge of a certain number of particles, actually displays poor parallel efficiency due to the high particle lifetime variability when dealing with a wide set of deposition problems that optionally include decay. The proposed method removes DPP defects and brings a novel approach to discrete particle tracking. The algorithm introduces master/slave model dubbed Partial Trajectory Decomposition (PTD), in which a certain number of processes produce partial trajectories and put them into the shared queue, while the remaining processes simulate actual particle motion using previously generated partial trajectories. Our approach also introduces meta-heuristics for determining the optimal values of partial trajectory length, chunk size and the number of processes acting as producers/consumers, for the given total number of participating processes (Optimized Partial Trajectory Decomposition, OPTD). The optimization process employs a surrogate model to estimate the simulation time. The surrogate is based on historical data and uses a coupled machine learning model, consisting of classification and regression phases. OPTD was implemented in C, using standard MPI for message passing and benchmarked on a model of 220 Rn progeny in the diffusion chamber, where particle motion is characterized by an exponential lifetime distribution and Maxwell velocity distribution. The speedup improvement of OPTD is approximatelly 320% over standard DPP, reaching almost ideal speedup on up to 256 CPUs.

Parallel Computation of Rough Set Approximations in Information Systems with Missing Decision Data

The paper discusses the use of parallel computation to obtain rough set approximations from large-scale information systems where missing data exist in both condition and decision attributes. To date, many studies have focused on missing condition data, but very few have accounted for missing decision data, especially in enlarging datasets. One of the approaches for dealing with missing data in condition attributes is named twofold rough approximations. The paper aims to extend the approach to deal with missing data in the decision attribute. In addition, computing twofold rough approximations is very intensive, thus the approach is not suitable when input datasets are large. We propose parallel algorithms to compute twofold rough approximations in large-scale datasets. Our method is based on MapReduce, a distributed programming model for processing large-scale data. We introduce the original sequential algorithm first and then the parallel version is introduced. Comparison between the two approaches through experiments shows that our proposed parallel algorithms are suitable for and perform efficiently on large-scale datasets that have missing data in condition and decision attributes.

The MapReduce Model on Cascading Platform for Frequent Itemset Mining

The implementation of parallel algorithms is very interesting research recently. Parallelism is very suitable to handle large-scale data processing. MapReduce is one of the parallel and distributed programming models. The implementation of parallel programming faces many difficulties. The Cascading gives easy scheme of Hadoop system which implements MapReduce model.Frequent itemsets are most often appear objects in a dataset. The Frequent Itemset Mining (FIM) requires complex computation. FIM is a complicated problem when implemented on large-scale data. This paper discusses the implementation of MapReduce model on Cascading for FIM. The experiment uses the Amazon dataset product co-purchasing network metadata.The experiment shows the fact that the simple mechanism of Cascading can be used to solve FIM problem. It gives time complexity O(n), more efficient than the nonparallel which has complexity O(n2/m).

Distributing relational model transformation on MapReduce

MDE has been successfully adopted in the production of software for several domains. As the models that need to be handled in MDE grow in scale, it becomes necessary to design scalable algorithms for model transformation (MT) as well as suitable frameworks for storing and retrieving models efficiently. One way to cope with scalability is to exploit the wide availability of distributed clusters in the Cloud for the parallel execution of MT. However, because of the dense interconnectivity of models and the complexity of transformation logic, the efficient use of these solutions in distributed model processing and persistence is not trivial.This paper exploits the high level of abstraction of an existing relational MT language, ATL, and the semantics of a distributed programming model, MapReduce, to build an ATL engine with implicitly distributed execution. The syntax of the language is not modified and no primitive for distribution is added. Efficient distribution of model elements is achieved thanks to a distributed persistence layer, specifically designed for relational MT. We demonstrate the effectiveness of our approach by making an implementation of our solution publicly available and using it to experimentally measure the speed-up of the transformation system while scaling to larger models and clusters.

A strategy to load balancing for non-connectivity MapReduce job

MapReduce has been widely used in large scale and complex datasets as a kind of distributed programming model. Original Hash partitioning function in MapReduce often results the problem of data skew when data distribution is uneven. To solve the imbalance of data partitioning, we proposes a strategy to change the remaining partitioning index when data is skewed. In Map phase, we count the amount of data which will be distributed to each reducer, then Job Tracker monitor the global partitioning information and dynamically modify the original partitioning function according to the data skew model, so the Partitioner can change the index of these partitioning which will cause data skew to the other reducer that has less load in the next partitioning process, and can eventually balance the load of each node. Finally, we experimentally compare our method with existing methods on both synthetic and real datasets, the experimental results show our strategy can solve the problem of data skew with better stability and efficiency than Hash method and Sampling method for non-connectivity MapReduce task.

MixHeter: A global scheduler for mixed workloads in heterogeneous environments

As data centers and applications grow more heterogeneous, allocating the proper resources to various applications increasingly depends on understanding the tradeoffs between different allocations, because mixed workloads may benefit from different resources e.g. GPU, Solid State Drives(as SSD). However, traditional distributed programming models are designed and improved for homogeneous environments and have poor performances in current heterogeneous environments. Thus we reconsider the problem in this paper and make three contributions: (1) Through analysis of experimental results, we summarize the main reasons of poor performances are unreasonable allocation of tasks between heterogeneous nodes and improper allocation of resources to mixed workloads; (2) To resolve them, we propose a global scheduler MixHeter based on or-constraints. Or-constraints imbibes advantages of no-constraints and hard-constraints, which satisfy applications’ resource preferences when all the resources are available and do not waste the non-preferred resources when the preferred resources are occupied. The model of or-constraints is based on utility function, which can associate utility with different resource requests to represent resource preferences and maximize overall utility to improve system efficiency. (3) Finally, we prove MixHeter can greatly decrease execution time than capacity scheduler of Hadoop 2.7.3 and capacity scheduler with label-based scheduling up to 15%–60% in heterogeneous environments, especially in the condition of mixed workloads with different resource preferences.

SparkBench: a spark benchmarking suite characterizing large-scale in-memory data analytics

Spark has been increasingly employed by industries for big data analytics recently, due to its resilience, scalability and efficient in-memory distributed programming model. Meanwhile, the rapid growing community is also actively incubating a rich ecosystem around Spark to tackle various big data challenges. The current benchmarks fall short in providing guidance of development, optimization, configuration and deployment of Spark. To this end, we introduce SparkBench, a Spark specific benchmarking suite. It selectively embraces a set of representative applications to identify various performance bottlenecks and reveals the resource consumption behaviors across execution phases. Overall, SparkBench covers four critical usage patterns of Spark, including machine learning, graph processing, stream computations and SQL query processing. We present comprehensive characterization of resource consumptions, data flows and timing information under different execution patterns and demonstrate that SparkBench can effectively guide the optimization of data analytic platforms to better suit for various workloads.

Efficient Pairwise Document Similarity Computation in Big Datasets

Support Vector Machine (SVM) is extremely powerful and widely accepted classifier in the field of machine learning due to its better generalization capability. However, SVM is not suiTable for large scale dataset due to its high computational complexity. The computation and storage requirement increases tremendously for large dataset. In this paper, we have proposed a MapReduce based SVM for large scale data. MapReduce is a distributed programming model which works on large scale dataset by dividing the huge datasets in smaller chunks. MapReduce distribution model works on several frame works like Hadoop Twister and so on. In this paper, we have analyzed the impact of penalty and kernel parameters on the performance of parallel SVM. The experimental result shows that the number of support vectors and predictive accuracy of SVM is affected by the choice of these parameters. From experimental results, it is also analyzed that the computation time taken by the SVM with multi-node cluster is less as compared to the single node cluster for large dataset.

Management of Possible Roles for Distributed Software Projects Using Layer Architecture

Several members are involved in development and management of the Distributed Software Projects. Each member needs to know the responsibilities of each other for proper management of the activities of such distributed projects to produce coherent outcomes. Distribution middleware software has higher-level distributed programming models whose reusable APIs (application programming interface) and components automate and extend native operating system capabilities. Software management tools like Work break-down structure (WBS), Gantt chart, Critical Path Method, and Critical Chain Method etc. does not fully help the managers to manage the member's responsibilities during the development of distributed applications. The layered architecture can help to do so. This style not only gives the layer level description of the activity involved, it also defines and directs the group of workforce. By listing the groups of workforce, the development team as well as the customer can know the activity and the member involved to work on those specific activities. This layered architecture is much benefited to development team and also to numbers of stakeholder of the large distributed project. The extended new approach of layer pattern with 'Responsibility Index' adds extra value to manage all the members' responsibilities. Managers, stakeholders and others can have an easy management system. The request or complaint from the customer can be passed to appropriate team without much delay. Most importantly this will give facility to collect timely feedback from all levels of customers.

English

Recent advances in information technology have led to an increase in volumes of data thereby exceeding beyond petabytes. Clustering distributed document sets from a central location is difficult due to the massive demand of computational resources. So there is a need for distributed document clustering algorithms to cluster documents using distributed resources. The greatest challenge in this area of distributed document clustering is the clustering quality and speedup associated with increase in document sets. The proposed clustering algorithm uses a hybrid algorithm which comprises of Particle Swarm Optimization (PSO), K-Means clustering and Latent Semantic Indexing (LSI) algorithm (PKMeansLSI), and uses MapReduce framework for distributed computation. The resultant of this is that it ultimately promotes clustering quality of the algorithm. The MapReduce framework and its corresponding implementation Hadoop is used as a distributed programming model which stresses on the improvement factor of the speedup of algorithm. The execution time is dramatically reduced as the dimensionality of documents is reduced. Experiment results show improved quality and effectiveness of the hybrid algorithm with varying increase in document size.   Key words: Distributed document clustering, Hadoop, K-Means, particle swarm optimization (PSO), latent semantic indexing (LSI), MapReduce.

Performance Comparison of OpenMP, MPI, and MapReduce in Practical Problems

With problem size and complexity increasing, several parallel and distributed programming models and frameworks have been developed to efficiently handle such problems. This paper briefly reviews the parallel computing models and describes three widely recognized parallel programming frameworks: OpenMP, MPI, and MapReduce. OpenMP is the de facto standard for parallel programming on shared memory systems. MPI is the de facto industry standard for distributed memory systems. MapReduce framework has become the de facto standard for large scale data-intensive applications. Qualitative pros and cons of each framework are known, but quantitative performance indexes help get a good picture of which framework to use for the applications. As benchmark problems to compare those frameworks, two problems are chosen: all-pairs-shortest-path problem and data join problem. This paper presents the parallel programs for the problems implemented on the three frameworks, respectively. It shows the experiment results on a cluster of computers. It also discusses which is the right tool for the jobs by analyzing the characteristics and performance of the paradigms.

Performance Improvement of MapReduce Framework in Heterogeneous Context using Reinforcement Learning

Abstract MapReduce is presently established as an important distributed and parallel programming model with wide acclaim for large scale computing. Intelligent scheduling decisions can help in reducing the overall runtime of the jobs. MapReduce performance is currently limited by its default scheduler, which does not adapt well in heterogeneous environments. Heterogeneous environments were considered in Longest Approximate Time to End scheduler. This too has several shortcomings due to the static manner in which it computes progress of tasks. The lack of adequate approach to heterogeneous environments is currently being taken up in recent research. In this paper, we propose a novel MapReduce scheduler in heterogeneous environments based on Reinforcement learning called MapReduce Reinforcement Learning scheduler, which observes the system state of task execution and suggests speculative re-execution of the slower tasks to other available nodes in the cluster for faster execution. The proposed approach adapts to the heterogeneous environment and no prior knowledge of the environmental characteristics are required. It is expected that over a few runs the system would be able to better map the computing requirements to the resources available in a heterogeneous cluster and minimizes the overall job completion time.

The Optimization and Improvement of MapReduce in Web Data Mining

Extracting and mining social networks information from massive Web data is of both theoretical and practical significance. However, one of definite features of this task was a large scale data processing, which remained to be a great challenge that would be addressed. MapReduce is a kind of distributed programming model. Just through the implementation of map and reduce those two functions, the distributed tasks can work well. Nevertheless, this model does not directly support heterogeneous datasets processing, while heterogeneous datasets are common in Web. This article proposes a new framework which improves original MapReduce framework into a new one called Map-Reduce-Merge. It adds merge phase that can efficiently solve the problems of heterogeneous data processing. At the same time, some works of optimization and improvement are done based on the features of Web data.

Triolet

Functional algorithmic skeletons promise a high-level programming interface for distributed-memory clusters that free developers from concerns of task decomposition, scheduling, and communication. Unfortunately, prior distributed functional skeleton frameworks do not deliver performance comparable to that achievable in a low-level distributed programming model such as C with MPI and OpenMP, even when used in concert with high-performance array libraries. There are several causes: they do not take advantage of shared memory on each cluster node; they impose a fixed partitioning strategy on input data; and they have limited ability to fuse loops involving skeletons that produce a variable number of outputs per input. We address these shortcomings in the Triolet programming language through a modular library design that separates concerns of parallelism, loop nesting, and data partitioning. We show how Triolet substantially improves the parallel performance of algorithms involving array traversals and nested, variable-size loops over what is achievable in Eden, a distributed variant of Haskell. We further demonstrate how Triolet can substantially simplify parallel programming relative to C with MPI and OpenMP while achieving 23--100% of its performance on a 128-core cluster.

Utility Mining Algorithm for High Utility Item sets from Transactional Databases

The discovery of item sets with high utility like profits is referred by mining high utility item sets from a transactional database. Although in recent years a number of relevant algorithms have been proposed, for high utility item sets the problem of producing a large number of candidate item sets is incurred. The mining performance is degraded by such a large number of candidate item sets in terms of execution time and space requirement. When the database contains lots of long transactions or long high utility item sets the situation may become worse. Internet purchasing and transactions is increased in recent years, mining of high utility item sets especially from the big transactional databases is required task to process many day to day operations in quick time. There are many methods presented for mining the high utility item sets from large transactional datasets are subjected to some serious limitations such as performance of this methods needs to be investigated in low memory based systems for mining high utility itemsets from large transactional datasets and hence needs to address further as well. Another limitation is these proposed methods cannot overcome the screenings as well as overhead of null transactions; hence, performance degrades drastically. During this paper, we are presenting the new approach to overcome these limitations. We presented distributed programming model for mining business-oriented transactional datasets by using an improved Map Reduce framework on Hadoop, which overcomes not only the single processor and main memory-based computing, but also highly scalable in terms of increasing database size. We have used this approach with existing UP-Growth and UP-Growth+ with aim of improving their performances further. In experimental studies we will compare the performances of existing algorithms UP-Growth and UP-Growth+ against the improve UP-Growth and UP-Growth+ with Hadoop.

Graph-based parallel distributed genetic programming model

Graph-based parallel distributed genetic programming model