Distributed Computing Framework Research Articles

Multi-Server Multi-Function Distributed Computation.

The work here studies the communication cost for a multi-server multi-task distributed computation framework, as well as for a broad class of functions and data statistics. Considering the framework where a user seeks the computation of multiple complex (conceivably non-linear) tasks from a set of distributed servers, we establish the communication cost upper bounds for a variety of data statistics, function classes, and data placements across the servers. To do so, we proceed to apply, for the first time here, Körner's characteristic graph approach-which is known to capture the structural properties of data and functions-to the promising framework of multi-server multi-task distributed computing. Going beyond the general expressions, and in order to offer clearer insight, we also consider the well-known scenario of cyclic dataset placement and linearly separable functions over the binary field, in which case, our approach exhibits considerable gains over the state of the art. Similar gains are identified for the case of multi-linear functions.

Multiple-Task Coded Computing for Distributed Computation Framework: Modeling and Delay Analysis

Multiple-Task Coded Computing for Distributed Computation Framework: Modeling and Delay Analysis

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Distributed computing frameworks are the fundamental component of distributed computing systems. They provide an essential way to support the efficient processing of big data on clusters or cloud. The size of big data increases at a pace that is faster than the increase in the big data processing capacity of clusters. Thus, distributed computing frameworks based on the MapReduce computing model are not adequate to support big data analysis tasks which often require running complex analytical algorithms on extremely big data sets in terabytes. In performing such tasks, these frameworks face three challenges: computational inefficiency due to high I/O and communication costs, non-scalability to big data due to memory limit, and limited analytical algorithms because many serial algorithms cannot be implemented in the MapReduce programming model. New distributed computing frameworks need to be developed to conquer these challenges. In this paper, we review MapReduce-type distributed computing frameworks that are currently used in handling big data and discuss their problems when conducting big data analysis. In addition, we present a non-MapReduce distributed computing framework that has the potential to overcome big data analysis challenges.

A Semantic Data-Based Distributed Computing Framework to Accelerate Digital Twin Services for Large-Scale Disasters.

As natural disasters become extensive, due to various environmental problems, such as the global warming, it is difficult for the disaster management systems to rapidly provide disaster prediction services, due to complex natural phenomena. Digital twins can effectively provide the services using high-fidelity disaster models and real-time observational data with distributed computing schemes. However, the previous schemes take little account of the correlations between environmental data of disasters, such as landscapes and weather. This causes inaccurate computing load predictions resulting in unbalanced load partitioning, which increases the prediction service times of the disaster management agencies. In this paper, we propose a novel distributed computing framework to accelerate the prediction services through semantic analyses of correlations between the environmental data. The framework combines the data into disaster semantic data to represent the initial disaster states, such as the sizes of wildfire burn scars and fuel models. With the semantic data, the framework predicts computing loads using the convolutional neural network-based algorithm, partitions the simulation model into balanced sub-models, and allocates the sub-models into distributed computing nodes. As a result, the proposal shows up to 38.5% of the prediction time decreases, compared to the previous schemes.

Collaborative Learning Based Straggler Prevention in Large-Scale Distributed Computing Framework

Modern big data applications tend to prefer a cluster computing approach as they are linked to the distributed computing framework that serves users jobs as per demand. It performs rapid processing of tasks by subdividing them into tasks that execute in parallel. Because of the complex environment, hardware and software issues, tasks might run slowly leading to delayed job completion, and such phenomena are also known as stragglers. The performance improvement of distributed computing framework is a bottleneck by straggling nodes due to various factors like shared resources, heavy system load, or hardware issues leading to the prolonged job execution time. Many state-of-the-art approaches use independent models per node and workload. With increased nodes and workloads, the number of models would increase, and even with large numbers of nodes. Not every node would be able to capture the stragglers as there might not be sufficient training data available of straggler patterns, yielding suboptimal straggler prediction. To alleviate such problems, we propose a novel collaborative learning-based approach for straggler prediction, the alternate direction method of multipliers (ADMM), which is resource-efficient and learns how to efficiently deal with mitigating stragglers without moving data to a centralized location. The proposed framework shares information among the various models, allowing us to use larger training data and bring training time down by avoiding data transfer. We rigorously evaluate the proposed method on various datasets with high accuracy results.

Vietnamese Short Text Classification via Distributed Computation

Social networking has been growing rapidly in Vietnam. The sharing information is diverse and circulates in many forms. It requires user-friendly solutions such as topic sorting and perspectives analysis in analyzing community trends, advertisements or anticipating and monitoring the spread of bad news. Unfortunately, Vietnamese is highly different from other languages and little research has been conducted in the literature on messages classification. The implementation of machine learning models on Vietnamese has not been thoroughly investigated and these models’ performance is unknown when applying in a different language. Vietnamese text is a serialization of syllables, hence, word boundary identification is not trivial. This research portrays our endeavor to construct an effective distributed framework for addressing the task of classification of short Vietnamese texts on social networks using the idea of probability categorization. The authors argue that addressing the task sharps the successful combination of machine learning, nat-ural language processing, and ambient intelligence. The proposed framework is effective and enables fast calculation, suitable for implementation in Apache Spark, meeting the demand for dealing with large amounts of textual data on the current social networks. Our data has been collected from several online text sources of 12412 short messages classified into 5 different topics. The evaluation shows that our approach has achieved an average of 82.73% classification accuracy. Thoughtfully learning the literature, we could state that this is the first attempt to classify short Vietnamese messages under a distributed computation framework.

Enhanced optimization of volumetric modulated arc therapy plans using Monte Carlo generated beamlets.

A treatment planning system (TPS) produces volumetric modulated arc therapy (VMAT) plans by applying an optimization process to an objective function, followed by an accurate calculation of the final, deliverable dose. However, during the optimization step, a rapid dose calculation algorithm is required, which reduces its accuracy and its representation of the objective function space. Monte Carlo (MC) routines, considered the gold standard in accuracy, are currently too slow for practical comprehensive VMAT optimization. Therefore, we propose a novel approach called enhanced optimization (EO), which employs the TPS VMAT plan as a starting point, and applies small perturbations to nudge the solution closer to a true objective minimum. The perturbations consist of beamlet dose matrices, calculated using MC routines on a distributed-computing framework. DICOM files for clinical VMAT plans files are exported from the TPS and used to generate input files for the EGSnrc MC toolkit. Beamlet doses are calculated using the MC routines, each corresponding to a single multileaf collimator leaf from a single control point traveling 0.5cm in or out of the field. A typical VMAT plan requires 5000 to 10000 beamlets, which may be calculated overnight. This results in a ternary-valued objective function, which may use the same clinical objectives as the original VMAT plan. A simple greedy search algorithm is applied to minimize this function and determine the optimal set of ternary variables. The resulting modified control point parameters are imported into the TPS to calculate the final, deliverable dose, and to compare the EO plan with the original. EO was evaluated retrospectively on seven VMAT plans (two adult brain, one pediatric brain, two head and neck, and two prostate). Additionally, the use of stricter objectives was investigated for two of the cases: the left cochlea planning organ at risk (OAR) volume objective for the pediatric brain case, and the rectum objective for a prostate case. EO produced improved objective scores (by 6% to 60%) and dose-volume histograms (DVH) for the brain plans and the head and neck plans. For each of these plans, the target dose minimum and homogeneity were preserved, while one or more of the OAR DVH's was reduced. Although EO also reduced the objective scores for the prostate plans (by 46% and 79%), their absolute score and DVH improvements were not substantial. The stricter objective on the pediatric brain case resulted in lower dose to the OAR without compromising the target dose. However, the rectum dose in the prostate case could not be improved without reducing dose homogeneity to the planning target volume, suggesting that VMAT prostate cases may already be highly optimized by the TPS. We have developed a novel approach to improving the dose distribution of VMAT plans, which relies on MC calculations to provide small modifications to the control points. This method may be particularly useful for complex treatments in which a certain OAR is of concern and it is difficult for the treatment planner to obtain an acceptable solution with the TPS. Further development will reduce the beamlet computation time and result in more sophisticated EO treatment planning methods.

Parallel graph algorithms in constant adaptive rounds

We study fundamental graph problems such as graph connectivity, minimum spanning forest (MSF), and approximate maximum (weight) matching in a distributed setting. In particular, we focus on the Adaptive Massively Parallel Computation (AMPC) model, which is a theoretical model that captures MapReduce-like computation augmented with a distributed hash table. We show the first AMPC algorithms for all of the studied problems that run in a constant number of rounds and use only O ( n ϵ ) space per machine, where 0 < ϵ < 1. Our results improve both upon the previous results in the AMPC model, as well as the best-known results in the MPC model, which is the theoretical model underpinning many popular distributed computation frameworks, such as MapReduce, Hadoop, Beam, Pregel and Giraph. Finally, we provide an empirical comparison of the algorithms in the MPC and AMPC models in a fault-tolerant distributed computation environment. We empirically evaluate our algorithms on a set of large real-world graphs and show that our AMPC algorithms can achieve improvements in both running time and round-complexity over optimized MPC baselines.

Performance Analysis of Distributed Computing Frameworks for Big Data Analytics: Hadoop Vs Spark

In the last one decade, the tremendous growth in data emphasizes big data storage and management issues with the highest priorities. For providing better support to software developers for dealing with big data problems, new programming platforms are continuously developing and Hadoop MapReduce is a big game-changer followed by Spark which sets the world of big data on fire with its processing speed and comfortable APIs. Hadoop framework emerged as a leading tool based on the MapReduce programming model with a distributed file system. Spark is on the other hand, recently developed big data analysis and management framework used to explore unlimited underlying features of Big Data. In this research work, a comparative analysis of Hadoop MapReduce and Spark has been presented on the basis of working principle, performance, cost, ease of use, compatibility, data processing, failure tolerance, and security. Experimental analysis has been performed to observe the performance of Hadoop MapReduce and Spark for establishing their suitability under different constraints of the distributed computing environment.

Big data time series forecasting based on pattern sequence similarity and its application to the electricity demand

This work proposes a novel algorithm to forecast big data time series. Based on the well-established Pattern Sequence-based Forecasting algorithm, this new approach has two major contributions to the literature. First, the improvement of the original algorithm with respect to the accuracy of predictions, and second, its transformation into the big data context, having reached meaningful results in terms of scalability. The algorithm uses the Apache Spark distributed computation framework and it is a ready-to-use application with few parameters to adjust. Physical and cloud clusters have been used to carry out the experimentation, which consisted in applying the algorithm to real-world data from Uruguay electricity demand.

Local-Global Modeling and Distributed Computing Framework for Nonlinear Plant-Wide Process Monitoring With Industrial Big Data.

Industrial big data and complex process nonlinearity have introduced new challenges in plant-wide process monitoring. This article proposes a local-global modeling and distributed computing framework to achieve efficient fault detection and isolation for nonlinear plant-wide processes. First, a stacked autoencoder is used to extract dominant representations of each local process unit and establish the local inner monitor. Second, mutual information (MI) is used to determine the neighborhood variables of a local unit. Afterward, a joint representation learning is then performed between the local unit and the neighborhood variables to extract the outer-related representations and establish the outer-related monitor for the local unit. Finally, the outer-related representations from all process units are used to establish global monitoring systems. Given that the modeling of each unit can be performed individually, the computation process can be efficiently completed with different CPUs. The proposed modeling and monitoring method is applied to the Tennessee Eastman (TE) and laboratory-scale glycerol distillation processes to demonstrate the feasibility of the method.

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

As data volumes grow rapidly, distributed computations are widely employed in data-centers to provide cheap and efficient methods to process large-scale parallel datasets. Various computation models have been proposed to improve the abstraction of distributed datasets and hide the details of parallelism. However, most of them follow the single-layer partitioning method, which limits developers to express a multi-level partitioning operation succinctly. To overcome the problem, we present the NDD (Nested Distributed Dataset) data model. It is a more compact and expressive extension of Spark RDD (Resilient Distributed Dataset), in order to remove the burden on developers to manually write the logic for multi-level partitioning cases. Base on the NDD model, we develop an open-source framework called Bigflow, which serves as an optimization layer over computation engines from most widely used processing frameworks. With the help of Bigflow, some advanced optimization techniques, which may only be applied by experienced programmers manually, are enabled automatically in a distributed data processing job. Currently, Bigflow is processing about 3 PB data volumes daily in the data-centers of Baidu. According to customer experience, it can significantly save code length and improve performance over the intuitive programming style.

Unified Fast Decoupled Load Flow in a Parallel Distributed Computation Framework

This research proposes a parallel and distributed computation approach for the fast decoupled load flow (FDLF) analysis applicable to both transmission and distribution networks of large electrical power systems. In this approach, the overall repetitive FDLF computations are performed by the respective control centers identified for each sub-system, the sub-systems being formed by partitioning the system through a network tearing procedure splitting the identified buses. The data of each subsystem are stored at their own control centers and hence the subsystems are solved using message passing distributed memory architecture. The computations performed by the control centers in various sub-systems are coordinated from a regional control center. The investigations on IEEE 118 bus standard test system indicate that the proposed approach provides faster solution than the conventional approach, without affecting the solution accuracy. The effectiveness of the proposed technique over the conventional approach become more visible when it is applied for the load flow analysis of large scale systems spread over a wide geographical area.

GraphMap: scalable iterative graph processing using NoSQL

Despite having several distributed graph processing frameworks, scalable iterative processing of large graphs is a challenging problem since the graph and intermediate data need a global view of the graph topology in distributed memory. Although some systems support out-of-core iterative computations, they use a single machine and often require fast storage. In this paper, we present a new distributed iterative graph computation framework, called GraphMap, that utilizes a disk-based NoSQL database system for scalable graph processing while ensuring competitive performance. Extensive experiments on several real-world graphs show that GraphMap is more scalable and often faster than existing distributed memory-based systems for various graph processing workloads.

Distributed Balanced Partitioning via Linear Embedding †

Balanced partitioning is often a crucial first step in solving large-scale graph optimization problems, for example, in some cases, a big graph can be chopped into pieces that fit on one machine to be processed independently before stitching the results together, leading to certain suboptimality from the interaction among different pieces. In other cases, links between different parts may show up in the running time and/or network communications cost, hence the desire to have small cut size. We study a distributed balanced-partitioning problem where the goal is to partition the vertices of a given graph into k pieces so as to minimize the total cut size. Our algorithm is composed of a few steps that are easily implementable in distributed computation frameworks such as MapReduce. The algorithm first embeds nodes of the graph onto a line, and then processes nodes in a distributed manner guided by the linear embedding order. We examine various ways to find the first embedding, for example, via a hierarchical clustering or Hilbert curves. Then we apply four different techniques including local swaps, and minimum cuts on the boundaries of partitions, as well as contraction and dynamic programming. As our empirical study, we compare the above techniques with each other, and also to previous work in distributed graph algorithms, for example, a label-propagation method, FENNEL and Spinner. We report our results both on a private map graph and several public social networks, and show that our results beat previous distributed algorithms: For instance, compared to the label-propagation algorithm, we report an improvement of 15–25% in the cut value. We also observe that our algorithms admit scalable distributed implementation for any number of partitions. Finally, we explain three applications of this work at Google: (1) Balanced partitioning is used to route multi-term queries to different replicas in Google Search backend in a way that reduces the cache miss rates by ≈ 0.5 % , which leads to a double-digit gain in throughput of production clusters. (2) Applied to the Google Maps Driving Directions, balanced partitioning minimizes the number of cross-shard queries with the goal of saving in CPU usage. This system achieves load balancing by dividing the world graph into several “shards”. Live experiments demonstrate an ≈ 40 % drop in the number of cross-shard queries when compared to a standard geography-based method. (3) In a job scheduling problem for our data centers, we use balanced partitioning to evenly distribute the work while minimizing the amount of communication across geographically distant servers. In fact, the hierarchical nature of our solution goes well with the layering of data center servers, where certain machines are closer to each other and have faster links to one another.

A Distributed Indexing Method for Timeline Similarity Query

Timelines have been used for centuries and have become more and more widely used with the development of social media in recent years. Every day, various smart phones and other instruments on the internet of things generate massive data related to time. Most of these data can be managed in the way of timelines. However, it is still a challenge to effectively and efficiently store, query, and process big timeline data, especially the instant recommendation based on timeline similarities. Most existing studies have focused on indexing spatial and interval datasets rather than the timeline dataset. In addition, many of them are designed for a centralized system. A timeline index structure adapting to parallel and distributed computation framework is in urgent need. In this research, we have defined the timeline similarity query and developed a novel timeline index in the distributed system, called the Distributed Triangle Increment Tree (DTI-Tree), to support the similarity query. The DTI-Tree consists of one T-Tree and one or more TI-Trees based on a triangle increment partition strategy with the Apache Spark. Furthermore, we have provided an open source timeline benchmark data generator, named TimelineGenerator, to generate various timeline test datasets for different conditions. The experiments for DTI-Tree’s construction, insertion, deletion, and similarity queries have been executed on a cluster with two benchmark datasets that are generated by TimelineGenerator. The experimental results show that the DTI-tree provides an effective and efficient distributed index solution to big timeline data.

A Study on Distributed Computing Framework Hadoop, Spark and Storm

International Journal of Computer Sciences and Engineering (A UGC Approved and indexed with DOI, ICI and Approved, DPI Digital Library) is one of the leading and growing open access, peer-reviewed, monthly, and scientific research journal for scientists, engineers, research scholars, and academicians, which gains a foothold in Asia and opens to the world, aims to publish original, theoretical and practical advances in Computer Science,Information Technology, Engineering (Software, Mechanical, Civil, Electronics & Electrical), and all interdisciplinary streams of Computing Sciences. It intends to disseminate original, scientific, theoretical or applied research in the field of Computer Sciences and allied fields. It provides a platform for publishing results and research with a strong empirical component. It aims to bridge the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research.

MCRS: A course recommendation system for MOOCs

With the popularization development of MOOC platform, the number of online courses grows rapidly. Efficient and appropriate course recommendation can improve learning efficiency. Traditional recommendation system is applied to the closed educational environment in which the quantity of courses and users is relatively stable. Recommendation model and algorithm cannot directly be applied to MOOC platform efficiently. With the light of the characteristics of MOOC platform, MCRS proposed in this paper has made great improvement in the course recommendation model and recommendation algorithm. MCRS is based on distributed computation framework. The basic algorithm of MCRS is distributed association rules mining algorithm, which based on the improvement of Apriori algorithm. In addition, it is useful to mine the hidden courses rules in course enrollment data. Firstly, the data is pre-processed into a standard form by Hadoop. It aims to improve the efficiency of the basic algorithm. Then it mines association rules of the standard data by Spark. Consequently, course recommendation information is transferred into MySQL through Sqoop, which makes timely feedback and improves user’s courses retrieval efficiency. Finally, to validate the efficiency of MCRS, a series of experiments are carried out on Hadoop and Spark, and the results shows that MCRS is more efficient than traditional Apriori algorithm and Apriori algorithm based on Hadoop, and the MCRS is suitable for current MOOC platform.

An investigation of pulsar searching techniques with the fast folding algorithm

Here we present an in-depth study of the behaviour of the Fast Folding Algorithm, an alternative pulsar searching technique to the Fast Fourier Transform. Weaknesses in the Fast Fourier Transform, including a susceptibility to red noise, leave it insensitive to pulsars with long rotational periods (P > 1 s). This sensitivity gap has the potential to bias our understanding of the period distribution of the pulsar population. The Fast Folding Algorithm, a time-domain based pulsar searching technique, has the potential to overcome some of these biases. Modern distributed-computing frameworks now allow for the application of this algorithm to all-sky blind pulsar surveys for the first time. However, many aspects of the behaviour of this search technique remain poorly understood, including its responsiveness to variations in pulse shape and the presence of red noise. Using a custom CPU-based implementation of the Fast Folding Algorithm, ffancy, we have conducted an in-depth study into the behaviour of the Fast Folding Algorithm in both an ideal, white noise regime as well as a trial on observational data from the HTRU-S Low Latitude pulsar survey, including a comparison to the behaviour of the Fast Fourier Transform. We are able to both confirm and expand upon earlier studies that demonstrate the ability of the Fast Folding Algorithm to outperform the Fast Fourier Transform under ideal white noise conditions, and demonstrate a significant improvement in sensitivity to long-period pulsars in real observational data through the use of the Fast Folding Algorithm.

Exploiting the untapped potential of mobile distributed computing via approximation

Mobile computing is one of the largest untapped reservoirs in today’s pervasive computing world as it has the potential to enable a variety of in-situ, real-time applications. Yet, this computing paradigm suffers when the available resources–such as energy in the network, CPU cycles, memory, I/O data rate–are limited. In this article, the new paradigm of approximate computing is proposed to harness such potential and to enable real-time computation-intensive mobile applications in resource-limited and uncertain environments. A reduction in time and energy consumed by an application is obtained via approximate computing by decreasing the amount of computation needed; such improvement, however, comes with the potential loss in accuracy. Hence, a Mobile Distributed Computing framework, is introduced to determine offline the ‘approximable’ tasks in an application and a light-weight online algorithm is devised to select the approximate version of the tasks in an application during run time. The effectiveness of the proposed approach is validated through extensive simulation and testbed experiments by comparing approximate versus exact-computation performance.