Distributed Processing Framework Research Articles

Construction and Application of Trajectory Data Analysis Model Based on Big Data and Stochastic Gradient Descent Algorithm

This paper studies the model construction of computing and storage resource management system framework based on Hadoop and the implementation of trajectory data analysis function under big data. Relying on the cloud platform infrastructure, in order to support the rapid data growth and massive data processing needs, it provides a mixed storage and analysis platform for structured and unstructured data, and uses big data technology to build a highly scalable and distributed data processing framework. The distributed computation, overall frame model of the memory system, and function module have been built with the aim of constructing the system in consideration. Second, by using Hadoop to preprocess the original data and concentrating on the data hierarchical design model and key technology analysis of big data systems, the design model, functional modules, technological solutions, and SGD algorithm are suggested, along with the detailed implementation procedure. Lastly, by merging the data of running vehicles, the system accomplishes the data analysis of vehicle trajectory, empty and load cars, and load and unload people.

A Demonstration of DLBD: Database Logic Bug Detection System

Database management systems (DBMSs) are prone to logic bugs that can result in incorrect query results. Current debugging tools are limited to single table queries and struggle with issues like lack of ground-truth results and repetitive query space exploration. In this paper, we demonstrate DLBD, a system that automatically detects logic bugs in databases. DLBD offers holistic logic bug detection by providing automatic schema and query generation and ground-truth query result retrieval. Additionally, DLBD provides minimal test cases and root cause analysis for each bug to aid developers in reproducing and fixing detected bugs. DLBD incorporates heuristics and domain-specific knowledge to efficiently prune the search space and employs query space exploration mechanisms to avoid the repetitive search. Finally, DLBD utilizes a distributed processing framework to test database logic bugs in a scalable and efficient manner. Our system offers developers a reliable and effective way to detect and fix logic bugs in DBMSs.

Efficient query evaluation techniques over large amount of distributed linked data

As RDF becomes more widely established and the amount of linked data is rapidly increasing, the efficient querying of large amount of data becomes a significant challenge. In this paper, we propose a family of algorithms for querying large amount of linked data in a distributed manner. These query evaluation algorithms are independent of the way the data is stored, as well as of the particular implementation of the query evaluation. We then use the MapReduce paradigm to present a distributed implementation of these algorithms and experimentally evaluate them, although the algorithms could be straightforwardly translated into other distributed processing frameworks. We also investigate and propose multiple query decomposition approaches of Basic Graph Patterns (subclass of SPARQL queries) that are used to improve the overall performance of the distributed query answering. A deep analysis of the effectiveness of these decomposition algorithms is also provided.

RDMA-Enabled Concurrency Control Protocols for Transactions in the Cloud Era

Online transaction processing (OLTP) is widely used on modern cloud infrastructures to complete important businesses such as payments and stock exchanges. Remote Direct Memory Access (RDMA) is a technology that enables ultra-low inter-server memory access latency - which is critical for implementing high-performance concurrency control protocols in distributed OLTP. In this article, we develop RCC, the first unified and comprehensive RDMA-enabled distributed transaction processing framework containing six serializable concurrency control protocols—not only the classical protocols <inline-formula><tex-math notation="LaTeX">${{\sf NOWAIT}}$</tex-math></inline-formula> , <inline-formula><tex-math notation="LaTeX">${{\sf WAITDIE}}$</tex-math></inline-formula> , <inline-formula><tex-math notation="LaTeX">${{\sf OCC}}$</tex-math></inline-formula> , but also more advanced <inline-formula><tex-math notation="LaTeX">${{\sf MVCC}}$</tex-math></inline-formula> , <inline-formula><tex-math notation="LaTeX">${{\sf SUNDIAL}}$</tex-math></inline-formula> , and <inline-formula><tex-math notation="LaTeX">${{\sf CALVIN}}$</tex-math></inline-formula> — the deterministic protocol. Our goal is to unbiasedly compare protocols on OLTP workloads in a common execution environment with the concurrency control protocol being the only changeable component. From RCC, we obtained new insights on building RDMA-based protocols. We analyzed stage-wise latency breakdown to develop more efficient hybrid implementations. Moreover, RCC can enumerate all stage-wise hybrid designs under a given workload characteristic. Our results show that throughput-wise hybrid designs are better than RPC or one-sided counterparts by 32.2% and up to 67%; three hybrid designs are better than their pure counterparts by up to 17.8%. RCC can provide performance insights and be used as the common open-sourced infrastructure for fast prototyping new implementations.

A Novel Reinforcement Learning Approach for Spark Configuration Parameter Optimization.

Apache Spark is a popular open-source distributed data processing framework that can efficiently process massive amounts of data. It provides more than 180 configuration parameters for users to manually select the appropriate parameter values according to their own experience. However, due to the large number of parameters and the inherent correlation between them, manual tuning is very tedious. To solve the problem of tuning through personal experience, we designed and implemented a reinforcement-learning-based Spark configuration parameter optimizer. First, we trained a Spark application performance prediction model with deep neural networks, and verified the accuracy and effectiveness of the model from multiple perspectives. Second, in order to improve the search efficiency of better configuration parameters, we improved the Q-learning algorithm, and automatically set start and end states in each iteration of training, which effectively improves the agent’s poor performance in exploring better configuration parameters. Lastly, comparing our proposed configuration with the default configuration as the baseline, experimental results show that the optimized configuration gained an average performance improvement of 47%, 43%, 31%, and 45% for four different types of Spark applications, which indicates that our Spark configuration parameter optimizer could efficiently find the better configuration parameters and improve the performance of various Spark applications.

Media Production Using Cloud and Edge Computing: Recent Progress and NBMP-Based Implementation

Media production is facing tremendous challenges during processing and transmission, due to large data volume, high computational complexity and real-time transmission requirement. Hence, more and more media servers tend to spread media content in a distributed manner, over e.g., cloud platform or edge computing nodes, which are closer to client side and/or have adequate computing capacity. Benefiting from cloud or edge computing, users can receive various media service with high quality and low latency. Network-based Media Processing (NBMP) established by International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) standard organization-Moving Picture Experts Group (MPEG), is one of these representative distributed architectures, which has attracted a wide range of attention. This paper focuses on emerging cloud and edge computation for media production to highlight the necessity of adopting a distributed computing framework. We take NBMP architecture as an example, and introduce the basic components and workflow in distributed processing frameworks. Moreover, we also provide related application instances based on NBMP with the hope of promoting future development of cloud and edge computation.

Distributed D-core decomposition over large directed graphs

Given a directed graph G and integers k and l , a D-core is the maximal subgraph H ⊆ G such that for every vertex of H , its in-degree and out-degree are no smaller than k and l , respectively. For a directed graph G , the problem of D-core decomposition aims to compute the non-empty D-cores for all possible values of k and l. In the literature, several peeling-based algorithms have been proposed to handle D-core decomposition. However, the peeling-based algorithms that work in a sequential fashion and require global graph information during processing are mainly designed for centralized settings, which cannot handle large-scale graphs efficiently in distributed settings. Motivated by this, we study the distributed D-core decomposition problem in this paper. We start by defining a concept called anchored coreness , based on which we propose a new H-index-based algorithm for distributed D-core decomposition. Furthermore, we devise a novel concept, namely skyline coreness , and show that the D-core decomposition problem is equivalent to the computation of skyline corenesses for all vertices. We design an efficient D-index to compute the skyline corenesses distributedly. We implement the proposed algorithms under both vertex-centric and block-centric distributed graph processing frameworks. Moreover, we theoretically analyze the algorithm and message complexities. Extensive experiments on large real-world graphs with billions of edges demonstrate the efficiency of the proposed algorithms in terms of both the running time and communication overhead.

Genetic Algorithm Based Fuzzy Local Informationc-Means (Gaflicm) Clustering Algorithm And Hybrid Kernel Convolution Neural Network (Hkcnn) With Distributed Processing Framework For Brain Mri Images

Cancer is the large important origin of casualty in today world. Among many cancers, brain cancer has been becomes one of the lowest living rate. It is formed based on the brain tumor. However brain tumors are able to have diverse categories based on their shape, texture, and position. Appropriate identification and retrieval of the tumor types create potential the doctor to compose the right cure option and assist keep the patient's life. Image processing has gained wide attention in medical analysis and health in recently. In general image processing methods, brain Magnetic Resonance Imaging (MRI) image collections cannot be processed efficiently on one computer due to large collection sizes and high computational costs. Hence, parallel computing and distributed system has been performed increasingly for brain MRI images in recently. In this paper, a novel Medical Image Cloud Processing (MICP) based distributed processing framework is proposed for brain MRI images by lesser computational time. In this work, image preprocessing is done by using the Adaptive Median Filtering (AMF) and image enhancement by Histogram Equalization (HE). The proposed MICP framework includes of Static Medical Image Cloud Processing (SMICP) and Dynamic Medical Image Cloud Processing (DMICP). In MICP framework, SMICP consists of two methods called Pure-Image and Big-Image. These methods are integrated to Genetic Algorithm based Fuzzy Local Information C-Means-MapReduce (GAFLICM-MR) algorithm to attain more optimized design and higher effectiveness. The core design of GAFLICM-MR framework is to make use of the rich computing resources given by means of the distributed system consequently as to apply efficient parallel processing. GAFLICM algorithm is also used as brain tumor segmentation in MRI images. In MICP framework, DMICP is developed via a parallel processing process of the distributed system. For retrieval and detection of brain Magnetic MRIimages into normal and tumor, Hybrid Kernel Convolution Neural Network (HKCNN) is developed in this work. Finally the results of the HKCNN classifier are compared to other previous works like precision, recall, f-measure, accuracy, time and memory.

Distributed frameworks for detecting distributed denial of service attacks: A comprehensive review, challenges and future directions

AbstractA distributed denial of service (DDoS) attack is a significant threat to web‐based applications and hindering legitimate traffic (denies access to benign users) by overwhelming the victim system or its infrastructure (service, bandwidth, networking devices, etc.) with a large volume of attack traffic. It leads to a delay in responses or sometimes a crash victim system. Even a few moments of pause in web‐based applications lead to a huge monetary loss and a bad reputation in the market. Several approaches available in the literature to protect websites from different types of DDoS attacks. However, incidents and volume sizes of DDoS attacks are growing quarter by quarter. Further, various challenges in the traditional framework based defense mechanisms: itself becoming a victim of attacks while analyzing a massive amount of traffic, require more time for detection process, no coordination among the modules, etc. This paper presents a comprehensive DDoS defense deployment taxonomy and critically reviewed existing distributed frameworks based DDoS attack detection systems. Further, characterized several existing distributed processing frameworks to select an appropriate one for deploying DDoS attack detection mechanisms. Finally, several evaluation metrics, open issues, discussion on available datasets including their limitations, and future directions are presented.

AEML: An Acceleration Engine for Multi-GPU Load-balancing in Distributed Heterogeneous Environment

For the rapid growth computation requirements in big data and artificial intelligence area, CPU-GPU heterogeneous clusters can provide more powerful computing capacity compared to CPU clusters. The number of GPUs on single computing node is scalable, which greatly improves the computing capacity of the cluster under the condition of limited cluster size. However, there is a lack of the effective load-balancing scheduling model in multi-GPU hardware environment. This paper proposes AEML, an acceleration engine for multi-GPU load-balancing in distributed heterogeneous environment. AEML can effectively integrate GPUs into distributed processing framework and achieve great load-balance among multiple heterogeneous GPUs. We propose a heterogeneous task execution model based on multiple GPUs and multiple streams (MGMS), which can effectively balance the workload of multiple GPUs. MGMS model utilizes four core techniques: a fine-grained task mapping mechanism, a device resource unified management scheme, a novel resource-aware GPU task scheduling strategy and a feedback-based streams adjustment scheme. The implementation of AEML system is based on Spark 2.4.1 and NVIDIA CUDA 10.0. We comprehensively evaluate the performance of AEML with multiple typical benchmarks. Experimental results show that AEML can fully exploit the computing power of GPUs and achieve great load-balance among multiple heterogeneous GPUs.

A Unifying Framework for Parallel and Distributed Processing in R using Futures

A future is a programming construct designed for concurrent and asynchronous evaluation of code, making it particularly useful for parallel processing. The future package implements the Future API for programming with futures in R. This minimal API provides sufficient constructs for implementing parallel versions of well-established, high-level map-reduce APIs. The future ecosystem supports exception handling, output and condition relaying, parallel random number generation, and automatic identification of globals lowering the threshold to parallelize code. The Future API bridges parallel frontends with parallel backends following the philosophy that end-users are the ones who choose the parallel backend while the developer focuses on what to parallelize. A variety of backends exist and third-party contributions meeting the specifications, which ensure that the same code works on all backends, are automatically supported. The future framework solves several problems not addressed by other parallel frameworks in R.

Design of Distributed Processing Framework Based on H-RTGL One-class Classifier for Big Data

Design of Distributed Processing Framework Based on H-RTGL One-class Classifier for Big Data

An Analysis on Task Migration Strategy of Big Data Streaming Storm Computing Framework for Distributed Processing

In this modern era, a large volume of data is generated regularly, which needs to be processed for gaining profits of latent information. The processing of big data is composed of a challenge termed as communication overhead. In order to minimize communication overhead on the premise of various resource constraints, a task migration strategy under heterogeneous storm environment is proposed. The proposed strategy is based on the establishment and demonstration of storm resource constraint model, optimal communication overhead model, and task migration model. Where the source node selection algorithm adds the nodes beyond the threshold to the source node set according to the load of CPU, memory, and network bandwidth of each working node in the cluster and the priority order of various resources. The experiment shows that compared with the existing research, computing storm can effectively reduce the delay and communication overhead between nodes, and the execution overhead is small.

Research on Intelligent Prediction of Urban Short-term Traffic Flow Based on Big Data

Traffic congestion has been the scale of major cities. Urban highway is an important carrier, which is to ensure the normal operation of the whole city. Therefore, we must establish intelligent transportation system, which is the most effective measure for intelligent prediction (hereinafter referred to as IP) of urban short-term traffic flow (hereinafter referred to as STTF). Among them, STTF prediction and big data are the key technologies, which require us to improve the accuracy and real-time performance of the prediction. This prediction model and intelligent system can be realized according to the idea of IP. The existing STTF prediction research is often based on the shallow model method, which can’t fully reflect the characteristics of STTF. Based on the IP method, this paper proposes a prediction model based on the distributed processing framework of MapReduce and BP neural network in Hadoop environment. Finally, this paper analyzes the basic process of the two runtime.

Configuring Parallelism for Hybrid Layouts Using Multi-Objective Optimization.

Modern organizations typically store their data in a raw format in data lakes. These data are then processed and usually stored under hybrid layouts, because they allow projection and selection operations. Thus, they allow (when required) to read less data from the disk. However, this is not very well exploited by distributed processing frameworks (e.g., Hadoop, Spark) when analytical queries are posed. These frameworks divide the data into multiple partitions and then process each partition in a separate task, consequently creating tasks based on the total file size and not the actual size of the data to be read. This typically leads to launching more tasks than needed, which, in turn, increases the query execution time and induces significant waste of computing resources. To allow a more efficient use of resources and reduce the query execution time, we propose a method that decides the number of tasks based on the data being read. To this end, we first propose a cost-based model for estimating the size of data read in hybrid layouts. Next, we use the estimated reading size in a multi-objective optimization method to decide the number of tasks and computational resources to be used. We prototyped our solution for Apache Parquet and Spark and found that our estimations are highly correlated (0.96) with the real executions. Further, using TPC-H we show that our recommended configurations are only 5.6% away from the Pareto front and provide 2.1 × speedup compared with default solutions.

Automatic ICA detection in online social networks with PageRank

Nowadays, online social networks have become an essential part of humans. However, there are some dark side to this widespread use of online social networks. One of them is the fact that many attackers have succeeded to clone celebrities’ profiles and have attracted hundreds or thousands of followers. This type of forging has caused many problems for famous people. This phenomenon is commonly known as Identity Cloning Attack which is abbreviated to ICA in the literature. ICA occurs when a malicious user selects one of the famous users of a social network as his victim. The attacker then creates a user account similar to the victim’s profile and embarks on various malicious social activities. In this paper, we have proposed an automatic method to identify cloned profiles. This method consists of three main steps and is implemented on Hadoop framework using the MapReduce programming model. In the first step, we count the number of followers of each user and store it as an attribute for their profile. In the second step, the network users are clustered based on their profile attributes and their number of followers. Subsequently, we move all the profiles within the same cluster as the victim’s profile to the next step and consider them as suspicious profiles. The victim’s profile is a profile of a celebrity, where the proposed method is conducted to verify its authenticity. In the third step, we eventually select the profile with the highest rank as the valid profile. This method of ranking the profiles is based on the outcome of PageRank algorithm in the first step. This method is easily applicable and does not require any additional information for identifying the original user account. Furthermore, this method employs a distributed processing framework, limits the search space, and decreases the required computation by clustering the profiles. We have applied the suggested method to a dataset that we collected from Instagram. Our findings were quite promising, and in some situations, we were able to identify all the cloned profiles with a 100% accuracy. The results are comparable to the best ones in this area of study.

A Scalable Synthesis of Multiple Models of Geo Big Data Interpretation

The paper proposes a scalable fuzzy approach for mapping the status of the environment integrating several distinct models exploiting geo big data. The process is structured into two phases: the first one can exploit products yielded by distinct models of remote sensing image interpretation defined in the scientific literature, and knowledge of domain experts, possibly ill-defined, for computing partial evidence of a phenomenon. The second phase integrates the partial evidence maps through a learning mechanism exploiting ground truth to compute a synthetic Environmental Status Indicator (ESI) map. The proposal resembles an ensemble approach with the difference that the aggregation is not necessarily consensual but can model a distinct decision attitude in between pessimistic and optimistic. It is scalable and can be implemented in a distributed processing framework, so as to make feasible ESI mapping in near real time to support land monitoring. It is exemplified to map the presence of standing water areas, indicator of water resources, agro-practices or natural hazard from remote sensing by considering different models.

TACC: Topology-Aware Coded Computing for Distributed Graph Processing

This article proposes a coded distributed graph processing framework to alleviate the communication bottleneck in large-scale distributed graph processing. In particular, we propose a topology-aware coded computing (TACC) algorithm that has two novel salient features: (i) a topology-aware graph allocation strategy, and (ii) a coded aggregation scheme that combines the intermediate computations for graph processing while constructing coded messages. The proposed setup results in a trade-off between computation and communication, in that increasing the computation load at the distributed parties can in turn reduce the communication load. We demonstrate the effectiveness of the TACC algorithm by comparing the communication load with existing setups on both Erdös-Rényi and Barabási-Albert type random graphs, as well as real-world Google web graph for PageRank computations. In particular, we show that the proposed coding strategy can lead to up to 82% reduction in communication load and up to 46% reduction in overall execution time, when compared to the state-of-the-art and implemented on the Amazon EC2 cloud compute platform.

Distributed Hypergraph Processing Using Intersection Graphs

The advent of online applications such as social networks has led to an unprecedented scale of data and complex relationships among data. Hypergraphs are introduced to represent complex relationships that may involve more than two entities. A hypergraph is a generalized form of a graph, where edges are generalized to hyperedges. Each hyperedge may consist of any number of vertices. The flexibility of hyperedges also brings challenges in distributed hypergraph processing. In particular, a hypergraph is more difficult to be partitioned and distributed among k workers with balanced partitions. In this paper, we propose to convert a hypergraph into an intersection graph before partitioning by leveraging the inherent shared relationships among hypergraphs. We explore the intersection graph construction method and the corresponding partition strategy which can achieve the goal of evenly distributing vertices and hyperedges across workers, while yielding a significant communication reduction. We also design a distributed processing framework named Hyraph that can directly run hypergraph analysis algorithms on our intersection graphs. Experimental results on real datasets confirm the effectiveness of our techniques and the efficiency of the Hyraph framework.

GraphMap: scalable iterative graph processing using NoSQL

Despite having several distributed graph processing frameworks, scalable iterative processing of large graphs is a challenging problem since the graph and intermediate data need a global view of the graph topology in distributed memory. Although some systems support out-of-core iterative computations, they use a single machine and often require fast storage. In this paper, we present a new distributed iterative graph computation framework, called GraphMap, that utilizes a disk-based NoSQL database system for scalable graph processing while ensuring competitive performance. Extensive experiments on several real-world graphs show that GraphMap is more scalable and often faster than existing distributed memory-based systems for various graph processing workloads.