Big Data Processing Framework Research Articles

Atomic Predicates-Based Data Plane Properties Verification in Software Defined Networking Using Spark

Software-Defined Networking (SDN) is an innovational network architecture which gives network administrators the ability to directly control the whole network by programming on a centralized controller. Due to network complexity, networks are unlikely to be bug-free. The ability to verify data plane properties will make network management easier for network administrators in SDN. In this paper, we present a novel atomic predicates based data plane properties verification method for SDN using Spark which is a big data processing framework. First, we verify packet reachability which is a fundamental data plane property. Then, we verify other data plane properties such as loop-freedom and nonexistence of black holes. In addition, the proposed method can detect a security threat existing in SDN called firewall bypass threat with packet reachability verification. By adopting atomic predicates, we achieve less computational and storage overhead. We implement the methods and study the performance. The results of experiments show that we can efficiently and accurately detect loops, black holes and firewall bypass threats.

Low Latency Big Data Processing without Prior Information

Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. Simple job scheduling policies, such as Fair and FIFO scheduling, do not consider job sizes and may degrade the performance when jobs of varying sizes arrive. More elaborate job scheduling policies make the convenient assumption that jobs are recurring, and complete information about their sizes is available from their prior runs. In this paper, we design and implement an efficient and practical job scheduler for big data processing systems to achieve better performance even without prior information about job sizes. The superior performance of our job scheduler originates from the design of multiple level priority queues, where jobs are demoted to lower priority queues if the amount of service consumed so far reaches a certain threshold. In this case, jobs in need of a small amount of service can finish in the topmost several levels of queues, while jobs that need a large amount of service to complete are moved to lower priority queues to avoid head-of-line blocking. Our new job scheduler can effectively mimic the shortest job first scheduling policy without knowing the job sizes in advance. To demonstrate its performance, we have implemented our new job scheduler in YARN, a popular resource manager used by Hadoop/Spark, and validated its performance with experiments in both real testbeds including Amazon EC2 and large-scale trace-driven simulations. Our experimental and simulation results have strongly confirmed the effectiveness of our design: our new job scheduler can reduce the average job response time of the Fair scheduler by up to 45 percent and achieve better fairness at the same time.

Ignis: An efficient and scalable multi-language Big Data framework

Most of the relevant Big Data processing frameworks (e.g., Apache Hadoop, Apache Spark) only support JVM (Java Virtual Machine) languages by default. In order to support non-JVM languages, subprocesses are created and connected to the framework using system pipes. With this technique, the impossibility of managing the data at thread level arises together with an important loss in the performance. To address this problem we introduce Ignis, a new Big Data framework that benefits from an elegant way to create multi-language executors managed through an RPC system. As a consequence, the new system is able to execute natively applications implemented using non-JVM languages. In addition, Ignis allows users to combine in the same application the benefits of implementing each computational task in the best suited programming language without additional overhead. The system runs completely inside Docker containers, isolating the execution environment from the physical machine. A comparison with Apache Spark shows the advantages of our proposal in terms of performance and scalability.

A configurable and executable model of Spark Streaming on Apache YARN

Streams of data are produced today at an unprecedented scale. Efficient and stable processing of these streams requires a careful interplay between the parameters of the streaming application and of the underlying stream processing framework. Today, finding these parameters happens by trial and error on the complex, deployed framework. This paper shows that high-level models can help to determine these parameters by predicting and comparing the performance of streaming applications running on stream processing frameworks with different configurations. To demonstrate this approach, this paper considers Spark Streaming, a widely used framework to leverage data streams on the fly and provide real-time stream processing. Technically, we develop a configurable and executable model to simulate both the streaming applications and the underlying Spark stream processing framework. Furthermore, we model the deployment of Spark Streaming on Apache YARN, which is a popular open-source distributed software framework for big data processing. We show that the developed model provides a satisfactory accuracy for predicting performance by means of empirical validation.

A configurable and executable model of Spark Streaming on Apache YARN

Streams of data are produced today at an unprecedented scale. Efficient and stable processing of these streams requires a careful interplay between the parameters of the streaming application and of the underlying stream processing framework. Today, finding these parameters happens by trial and error on the complex, deployed framework. This paper shows that high-level models can help to determine these parameters by predicting and comparing the performance of streaming applications running on stream processing frameworks with different configurations. To demonstrate this approach, this paper considers Spark Streaming, a widely used framework to leverage data streams on the fly and provide real-time stream processing. Technically, we develop a configurable and executable model to simulate both the streaming applications and the underlying Spark stream processing framework. Furthermore, we model the deployment of Spark Streaming on Apache YARN, which is a popular open-source distributed software framework for big data processing. We show that the developed model provides a satisfactory accuracy for predicting performance by means of empirical validation.

ATCS: Auto-Tuning Configurations of Big Data Frameworks Based on Generative Adversarial Nets

Big data processing frameworks (e.g., Spark, Storm) have been extensively used for massive data processing in the industry. To improve the performance and robustness of these frameworks, developers provide users with highly-configurable parameters. Due to the high-dimensional parameter space and complicated interactions of parameters, manual tuning of parameters is time-consuming and ineffective. Building performance-predicting models for big data frameworks is challenging for several reasons: (1) the significant time required to collect training data and (2) the poor accuracy of the prediction model when training data are limited. To meet this challenge, we proposes an auto-tuning configuration parameters system (ATCS), a new auto-tuning approach based on Generative Adversarial Nets (GAN). ATCS can build a performance prediction model with less training data and without sacrificing model accuracy. Moreover, an optimized Genetic Algorithm (GA) is used in ATCS to explore the parameter space for optimum solutions. To prove the effectiveness of ATCS, we select five frequently-used workloads in Spark, each of which runs on five different sized data sets. The results demonstrate that ATCS improves the performance of five frequently-used Spark workloads compared to the default configurations. We achieved a performance increase of 3.5× on average, with a maximum of 6.9×. To obtain similar model accuracy, experiment results also demonstrate that the quantity of ATCS training data is only 6% of Deep Neural Network (DNN) data, 13% of Support Vector Machine (SVM) data, 18% of Decision Tree (DT) data. Moreover, compared to other machine learning models, the average performance increase of ATCS is 1.7× that of DNN, 1.6× that of SVM, 1.7× that of DT on the five typical Spark programs.

Cost-efficient dynamic scheduling of big data applications in apache spark on cloud

Job scheduling is one of the most crucial components in managing resources, and efficient execution of big data applications. Specifically, scheduling jobs in a cloud-deployed cluster are challenging as the cloud offers different types of Virtual Machines (VMs) and jobs can be heterogeneous. The default big data processing framework schedulers fail to reduce the cost of VM usages in the cloud environment while satisfying the performance constraints of each job. The existing works in cluster scheduling mainly focus on improving job performance and do not leverage from VM types on the cloud to reduce cost. In this paper, we propose efficient scheduling algorithms that reduce the cost of resource usage in a cloud-deployed Apache Spark cluster. In addition, the proposed algorithms can also prioritise jobs based on their given deadlines. Besides, the proposed scheduling algorithms are online and adaptive to cluster changes. We have also implemented the proposed algorithms on top of Apache Mesos. Furthermore, we have performed extensive experiments on real datasets and compared to the existing schedulers to showcase the superiority of our proposed algorithms. The results indicate that our algorithms can reduce resource usage cost up to 34% under different workloads and improve job performance.

Cloud accelerated alignment and assembly of full-length single-cell RNA-seq data using Falco

BackgroundRead alignment and transcript assembly are the core of RNA-seq analysis for transcript isoform discovery. Nonetheless, current tools are not designed to be scalable for analysis of full-length bulk or single cell RNA-seq (scRNA-seq) data. The previous version of our cloud-based tool Falco only focuses on RNA-seq read counting, but does not allow for more flexible steps such as alignment and read assembly.ResultsThe Falco framework can harness the parallel and distributed computing environment in modern cloud platforms to accelerate read alignment and transcript assembly of full-length bulk RNA-seq and scRNA-seq data. There are two new modes in Falco: alignment-only and transcript assembly. In the alignment-only mode, Falco can speed up the alignment process by 2.5–16.4x based on two public scRNA-seq datasets when compared to alignment on a highly optimised standalone computer. Furthermore, it also provides a 10x average speed-up compared to alignment using published cloud-enabled tool for read alignment, Rail-RNA. In the transcript assembly mode, Falco can speed up the transcript assembly process by 1.7–16.5x compared to performing transcript assembly on a highly optimised computer.ConclusionFalco is a significantly updated open source big data processing framework that enables scalable and accelerated alignment and assembly of full-length scRNA-seq data on the cloud. The source code can be found at https://github.com/VCCRI/Falco.

Review of Big Data and Processing Frameworks for Disaster Response Applications

Natural hazards result in devastating losses in human life, environmental assets and personal, and regional and national economies. The availability of different big data such as satellite imageries, Global Positioning System (GPS) traces, mobile Call Detail Records (CDRs), social media posts, etc., in conjunction with advances in data analytic techniques (e.g., data mining and big data processing, machine learning and deep learning) can facilitate the extraction of geospatial information that is critical for rapid and effective disaster response. However, disaster response systems development usually requires the integration of data from different sources (streaming data sources and data sources at rest) with different characteristics and types, which consequently have different processing needs. Deciding which processing framework to use for a specific big data to perform a given task is usually a challenge for researchers from the disaster management field. Therefore, this paper contributes in four aspects. Firstly, potential big data sources are described and characterized. Secondly, the big data processing frameworks are characterized and grouped based on the sources of data they handle. Then, a short description of each big data processing framework is provided and a comparison of processing frameworks in each group is carried out considering the main aspects such as computing cluster architecture, data flow, data processing model, fault-tolerance, scalability, latency, back-pressure mechanism, programming languages, and support for machine learning libraries, which are related to specific processing needs. Finally, a link between big data and processing frameworks is established, based on the processing provisioning for essential tasks in the response phase of disaster management.

Profit Maximization of Big Data Jobs in Cloud Using Stochastic Optimization

Reserved instances offered by cloud providers make it possible to reserve resources and computing capacity for a specific period of time. One should pay for all the hours of that time interval; in exchange, the hourly rate is significantly lower than on-demand instances. Reserved Instances can significantly reduce the monetary cost of resources needed to process big data applications in cloud. However, purchases of these instances are non-refundable, and hence, one should be able to estimate the required resources prior to purchase to avoid over-payment. It becomes important especially when the results obtained by big data job has monetary value, such as business intelligence applications. But, estimating the resource demand of big data processing jobs is hard because of numerous factors that affect them such as data locality, data skew, stragglers, internal settings of big data processing framework, interference among instances, instances availability, etc. To maximize the profit of processing such big data jobs in cloud considering fluctuating nature of their resource demand, as well as reserved instances limitations, we propose Reserved Instances Stochastic Allocation (RISA) approach. Using historical traces of resource demand of big data jobs submitted by user, RISA leverages stochastic optimization to determine the amount of resources needed to be reserved for that user to maximize the profit. Our evaluation using real-world traces shows that RISA can increase the net profit by up to 10x, compared to previous approaches. RISA can also find solutions as close as 2% to the best possible solution.

Real-Time Massive Vector Field Data Processing in Edge Computing.

The spread of the sensors and industrial systems has fostered widespread real-time data processing applications. Massive vector field data (MVFD) are generated by vast distributed sensors and are characterized by high distribution, high velocity, and high volume. As a result, computing such kind of data on centralized cloud faces unprecedented challenges, especially on the processing delay due to the distance between the data source and the cloud. Taking advantages of data source proximity and vast distribution, edge computing is ideal for timely computing on MVFD. Therefore, we are motivated to propose an edge computing based MVFD processing framework. In particular, we notice that the high volume feature of MVFD results in high data transmission delay. To solve this problem, we invent Data Fluidization Schedule (DFS) in our framework to reduce the data block volume and the latency on Input/Output (I/O). We evaluated the efficiency of our framework in a practical application on massive wind field data processing for cyclone recognition. The high efficiency our framework was verified by the fact that it significantly outperformed classical big data processing frameworks Spark and MapReduce.

Energy-Efficient Task Scheduling for CPU-Intensive Streaming Jobs on Hadoop

Hadoop, especially Hadoop 2.0, has been a dominant framework for real-time big data processing. However, Hadoop is not optimized for energy efficiency. Aiming to solve this problem, in this paper, we propose a new framework to improve the energy efficiency of Hadoop 2.0. We focus on the resource manager in Hadoop 2.0, namely YARN, and propose energy-efficient task scheduling mechanisms on YARN. Particularly, we focus on CPU-intensive streaming jobs and classify streaming jobs into two types, namely batch streaming jobs (i.e., a set of jobs are submitted simultaneously) and online streaming jobs (i.e., jobs are continuously submitted one by one). We devise different energy-efficient task scheduling algorithms for each kind of streaming jobs. Specially, we first propose to abstractly model performance and energy consumption by considering the characteristics of tasks as well as the computational resources in YARN. Based on this model, we study the energy efficiency of streaming tasks which consist of the performance model and energy consumption model of task. We propose two key principles for improving energy efficiency: 1) CPU usage aware task allocation, partitions tasks to NMs based on the task characteristic in term of CPU usage; and 2) resource efficient task allocation, reduce idle resource. Then, we propose a D-based binning algorithm for the batch task scheduling and K-based binning algorithm for the online task scheduling that can adapt to continuously arriving tasks. We conduct extensive experiments on a real Hadoop 2.0 cluster and use two kinds of workloads to evaluate the performance and energy efficiency of our proposal. Compared with Storm (the streaming data processing tool in Hadoop 2.0) and other approaches including TAPA and DVFS-MR, our proposal is more energy efficient. The batch task scheduling algorithm reduces up to 10 percent of energy consumption and keeps comparable performance. In addition, the online task scheduling algorithm reduces up to 7 percent over the existing algorithms.

Programmable Big Data Processing Framework to Reduce On-Chip Communications and Computations Towards Reducing Energy of the Processing

Programmable Big Data Processing Framework to Reduce On-Chip Communications and Computations Towards Reducing Energy of the Processing

Fuzzy knowledge based performance analysis on big data

Due to the various emerging technologies, an enormous amount of data, termed as Big Data, gets collected every day and can be of great use in various domains. Clustering algorithms that store the entire data into memory for analysis become unfeasible when the dataset is too large. Many clustering algorithms present in the literature deal with the analysis of huge amount of data. The paper discusses a new clustering approach called an Incremental Random Sampling with Iterative Optimization Fuzzy c-Means (IRSIO-FCM) algorithm. It is implemented on Apache Spark, a framework for Big Data processing. Sparks works really well for iterative algorithms by supporting in-memory computations, scalability, etc. IRSIO-FCM not only facilitates effective clustering of Big Data but also performs storage space optimization during clustering. To establish a fair comparison of IRSIO-FCM, we propose an incremental version of the Literal Fuzzy c-Means (LFCM) called ILFCM implemented in Apache Spark framework. The experimental results are analyzed in terms of time and space complexity, NMI, ARI, speedup, sizeup, and scaleup measures. The reported results show that IRSIO-FCM achieves a significant reduction in run-time in comparison with ILFCM.

Big data processing framework for manufacturing

Data analysis of manufacturing plays a vital part in the intelligent manufacturing service of Product-Service Systems (PSS). In order to solve the problem that, manufacturing companies can’t obtain valuable information from enterprise’s big data through traditional data analysis methods, this paper put forward a data processing architecture framework and introduce the predictive algorithm (Random Forest). Finally, a real-time prediction of quality under this framework which uses the random forest algorithm is given to verify the usefulness of the architecture framework.

Cost-Sensitive Distributed Machine Learning for NetFlow-Based Botnet Activity Detection

The recent advancements of malevolent techniques have caused a situation where the traditional signature-based approach to cyberattack detection is rendered ineffective. Currently, new, improved, potent solutions incorporating Big Data technologies, effective distributed machine learning, and algorithms countering data imbalance problem are needed. Therefore, the major contribution of this paper is the proposal of the cost-sensitive distributed machine learning approach for cybersecurity. In particular, we proposed to use and implemented cost-sensitive distributed machine learning by means of distributed Extreme Learning Machines (ELM), distributed Random Forest, and Distributed Random Boosted-Trees to detect botnets. The system’s concept and architecture are based on the Big Data processing framework with data mining and machine learning techniques. In practical terms in this paper, as a use case, we consider the problem of botnet detection by means of analysing the data in form of NetFlows. The reported results are promising and show that the proposed system can be considered as a useful tool for the improvement of cybersecurity.

The Impact of Big Data Processing Framework for Artificial Intelligence within Corporate Marketing Communication

This research examines what impact the Big Data Processing Framework (BDPF) has on Artificial Intelligence (AI) applications within Corporate Marketing Communication (CMC), and thereby the research question stated is: What is the potential impact of the BDPF on AI applications within the CMC tactical and managerial functions? To fulfill the purpose of this research, a qualitative research strategy was applied, including semi-structured interviews with experts within the different fields of examination: management, AI technology and CMC. The findings were analyzed through performing a thematic analysis, where coding was conducted in two steps. AI has many useful applications within CMC, which currently mainly are of the basic form of AI, so-called rule-based systems. However, the more complicated communication systems are used in some areas. Based on these findings, the impact of the BDPF on AI applications is assessed by examining different characteristics of the processing frameworks. The BDPF initially imposes both an administrative and compliance burden on organizations within this industry, and is particularly severe when machine learning is used. These burdens foremost stem from the general restriction of processing personal data and the data erasure requirement. However, in the long term, these burdens instead contribute to a positive impact on machine learning. The timeframe until enforcement contributes to a somewhat negative impact in the short term, which is also true for the uncertainty around interpretations of the BDPF requirements. Yet, the BDPF provides flexibility in how to become compliant, which is favorable for AI applications. Finally, BDPF compliance can increase company value, and thereby incentivize investments into AI models of higher transparency. The impact of the BDPF is quite insignificant for the basic forms of AI applications, which are currently most common within CMC. However, for the more complicated applications that are used, the BDPF is found to have a more severe negative impact in the short term, while it instead has a positive impact in the long term.

Design and implementation of reconfigurable acceleration for in-memory distributed big data computing

Apache Spark is an efficient distributed computing framework for big data processing. It supports in-memory computation of RDDs (Resilient Distributed Datasets) and provides a provision of reusability, fault tolerance, and real-time stream processing. However, the tasks in Spark framework are only performed on CPU. The low degree of parallelism and power inefficiency of CPU may restrict the performance and scalability of the cluster. In order to improve the performance and power dissipation of the data center, heterogeneous accelerators such as FPGA, GPU, MIC (Many Integrated Core) exhibit more efficient performance than general-purpose processors in big data processing. In this work, we propose a framework to integrate FPGA accelerators into a Spark cluster, which achieves performance improvement and power dissipation reduction for distributed applications. We propose a method for connecting Spark with OpenCL application which is a standard for cross-platform, parallel programming of diverse processors and widely used in heterogeneous computing, and use FPGA to accelerate the Spark tasks developed with Python. We illustrate the performance and the energy efficiency of FPGA based Spark framework with a case study of K-means algorithm acceleration. The results show that FPGA based Spark implementation achieves 3.5x speedup and 4.06x energy efficiency over original Spark framework.

Distributed Collaborative Execution on the Edges and Its Application to AMBER Alerts

In the Internet of Everything era, billions of geographically distributed things will connect to the Internet and generate hundreds of zettabytes of data per year. Pushing that data to the cloud requires tremendous network bandwidth cost and latency. This is too onerous for some latency-sensitive applications, such as vehicle tracking using city-wide cameras. One application currently limited by such obstacles is the America’s Missing Broadcast Emergency Response (AMBER) Alert system—but edge computing could transform this system’s capabilities. Edge computing is a new computing paradigm that greatly diminishes data transmission and response latency by processing data at the proximity of data sources. However, most vision-based analytics are compute-intensive, and an edge device might be overwhelmed given tens of frames each second for real-time analysis. Also, the system needs a customized and flexible interface to implement efficient tracking strategies. To meet these needs, here we extend a big data processing framework, called Firework , to support collaboration between multiple edge devices and customizable task-scheduling strategies. Based on this extended version of Firework , we implement the AMBER alert assistant (A3), which efficiently tracks and locates a vehicle by analyzing city cameras’ data in real time. We also propose two kinds of customized task-scheduling algorithms for vehicle tracking in A3. Comprehensive evaluation results show that A3 achieves real-time video analytics by collaborating among multiple edge devices; and the proposed location-direction-related diffusion strategy effectively controls the searching area for vehicle tracking by smartly selecting candidate cameras.

HTTP and contact‐based features for Botnet detection

Over the past decade, Botnets have been prevailing as a relevant threat on the Internet. A Botnet is compound by many infected computers connected to the global network and subdued to a controller, which uses the computational resources for malicious purposes. Many techniques are used to make difficult the identification of Botnet communications on networks. However, these communications are automated and usually have typical behavioral patterns, which have been used by detection approaches. In this context, we noted that HTTP connections established as command and control (C&C) channels of Botnets commonly have specific implementations, which deviate from the patterns used by legitimate HTTP accesses. Also, C&C channels over other protocols, such as IRC or TCP, typically are compound by similar chunks of data transmission interleaved with times with negligible traffic. We used both particularities to propose new features to distinguish C&C channels from benign traffic. The proposed detection method uses a random forest classifier implemented over Apache Spark, a Big Data processing framework. In the presented experiments, the approach achieved more than 99% of accuracy with virtually zero false positives, which are good results when compared with other available similar approaches. Also, the proposed features can be extracted before the communication end, which enables a premature response.