Big Data Processing Framework Research Articles

A reference architecture for serverless big data processing

Despite significant advances in data management systems in recent decades, the processing of big data at scale remains very challenging. While cloud computing has been well-accepted as a solution to address scalability needs, cloud configuration and operation complexity persist and often present themselves as entry barriers, especially for novice data analysts. Serverless computing and Function-as-a-Service (FaaS) platforms have been suggested to reduce such entry barriers by shifting configuration and operational responsibilities from the application developer to the FaaS platform provider. Naturally, “serverless data processing (SDP)”, that is, using FaaS for (big) data processing, has received increasing interest in recent years.However, FaaS platforms were never intended to support large data processing tasks primarily. SDP, therefore, manifests itself through workarounds and adaptations on the application level, addressing various quirks and limitations of the FaaS platforms in use for data processing needs. This, in turn, creates tensions between the platforms and the applications using them, again encouraging the constant (re-)design of both. Consequently, we present lessons learned from a series of application and platform re-designs that address these tensions, leading to the development of an SDP reference architecture and a platform instantiation and implementation thereof called CREW. Mitigating the tensions through the process of application platform co-design proves to reduce both entry barriers and costs significantly. In some experiments, CREW outperforms traditional, non-SDP big data processing frameworks by factors.

Parametrized Optimization Based on an Investigation of Musical Similarities Using SPARK and Hadoop

The big data processing framework Spark is used to power a parameterizable recommender system that can make recommendations for music based on a user’s individual tastes and take into account a variety of musical tonal qualities. The system as it is presently built is completely scalable, which means that additional songs can be contributed to the data, the cluster size could be increased, and new types of audio information, in addition to more cutting-edge similarity evaluations, may be included. Another issue discussed in this research paper is the parallel collection of required audio characteristics on a computer cluster. Song recommendations for a dataset including more than 114,000 songs may be created on a Spark cluster with 16 nodes in under 12 s by integrating eight distinct audio feature types and similarity assessments. After the features have been retrieved, they are sent to the Spark-based recommender system to be processed. The calculated distance was displayed, examined, and graphically depicted. By computing the distance depending on the melody, rhythmic, and timbral components of the music, the final software controls song suggestion.

A study of performance comparison of databases for HALF data archiving system

The Hefei Advanced Light Facility (HALF) is a 2.2 GeV fourth-generation synchrotron radiation light source. The HALF control system is a distributed control system based on Experimental Physics and Industrial Control System (EPICS). As the essential part of the HALF control system, the Historical Data Archiving System (HDAS) is responsible to store operational data for the entire facility including the accelerator and beamlines, and provides the functions for data query and analysis. According to the estimation based on the HALF scale, approximately 25,000 EPICS PVs will be stored in HDAS, and these accumulated massive data require a dedicated database for persistent storage and management. Under the EPICS PV data scenario of HALF, a fair database test platform is designed and built to test the read-write performance of databases commonly used in the particle accelerator field. The tested objects include EPICS Archiver Appliance and the five databases MongoDB, HBase, InfluxDB, TimescaleDB, and Cassandra. Additionally, research on the five databases software ecosystem is conducted for the development and extension of HDAS. The test results indicate that TimescaleDB has the fastest read performance, and 1.4×10^6 items of data can be read per second. TimescaleDB also has superior write performance. 1.2 × 105 items of data can be written per second, which can fully meet the writing requirements of HDAS. Regarding the software ecosystem, TimescaleDB is compatible with a larger number of data visualization software and big data processing frameworks. Therefore, TimescaleDB is more suitable as the HDAS database. In the future, a TimescaleDB distributed cluster will be designed and deployed, and an HDAS prototype system will be developed based on this cluster.

YARN Schedulers for Hadoop MapReduce Jobs: Design Goals, Issues and Taxonomy

Objective: Big Data processing is a demanding task, and several big data processing frameworks have emerged during recent decades. The performance of these frameworks greatly dependent on resource management models. Methods: YARN is one of such models which acts as a resource management layer and provides computational resources for execution engines (Spark, MapReduce, storm, etc.) through its schedulers. The most important aspect of resource management is job scheduling. Results: In this paper, we first present the design goal of YARN real-life schedulers (FIFO, Capacity, and Fair) for the MapReduce engine. Later, we discuss the scheduling issues of the Hadoop MapReduce cluster. Conclusion: Many efforts have been carried out in the literature to address issues of data locality, heterogeneity, straggling, skew mitigation, stragglers and fairness in Hadoop MapReduce scheduling. Lastly, we present the taxonomy of different scheduling algorithms available in the literature based on some factors like environment, scope, approach, objective and addressed issues.

A New Big Data Processing Framework for the Online Roadshow

The Online Roadshow, a new type of web application, is a digital marketing approach that aims to maximize contactless business engagement. It leverages web computing to conduct interactive game sessions via the internet. As a result, massive amounts of personal data are generated during the engagement process between the audience and the Online Roadshow (e.g., gameplay data and clickstream information). The high volume of data collected is valuable for more effective market segmentation in strategic business planning through data-driven processes such as web personalization and trend evaluation. However, the data storage and processing techniques used in conventional data analytic approaches are typically overloaded in such a computing environment. Hence, this paper proposed a new big data processing framework to improve the processing, handling, and storing of these large amounts of data. The proposed framework aims to provide a better dual-mode solution for processing the generated data for the Online Roadshow engagement process in both historical and real-time scenarios. Multiple functional modules, such as the Application Controller, the Message Broker, the Data Processing Module, and the Data Storage Module, were reformulated to provide a more efficient solution that matches the new needs of the Online Roadshow data analytics procedures. Some tests were conducted to compare the performance of the proposed frameworks against existing similar frameworks and verify the performance of the proposed framework in fulfilling the data processing requirements of the Online Roadshow. The experimental results evidenced multiple advantages of the proposed framework for Online Roadshow compared to similar existing big data processing frameworks.

TurBO: A cost-efficient configuration-based auto-tuning approach for cluster-based big data frameworks

Big data processing frameworks such as Spark usually provide a large number of performance-related configuration parameters, how to auto-tune these parameters for a better performance has been a hot issue in academia as well as industry for years. Through delicately tradeoff between exploration and exploitation, Bayesian Optimization (BO) is currently the most appealing algorithm to achieve configuration auto-tuning. However, considering the tuning cost constraint in practice, there are three critical limitations preventing conventional BO-based approaches from being directly applied into auto-tuning cluster-based big data frameworks. In this paper, we propose a cost-efficient configuration auto-tuning approach named TurBO for big data frameworks based on two enhancements of vanilla BO:1) To reduce the essential iteration times, TurBO integrates a well-designed adaptive pseudo point mechanism with BO; 2) To avoid the time-consuming practical evaluation of sub-optimal configurations as possible, TurBO leverages the proposed CASampling method to intelligently tackle with these sub-optimal configurations based on ensemble learning with historical tuning experiences. To evaluate the performance of TurBO, we conducted a series of experiments on a local Spark cluster with 9 different HiBench benchmark applications. Overall, compared with 3 representative BO-based baseline approaches OpenTuner, Bliss and ResTune, TurBO is able to speedup the tuning procedures respectively by 2.24×, 2.29× and 1.97× on average. Besides, TurBO can always achieve a positive cumulative performance gain under the simulated dynamic workload scenario, which means TurBO is indeed appropriate for workload changes of big data applications.

Distributed fuzzy clustering algorithm for mixed-mode data in Apache SPARK

Fuzzy clustering is an invaluable data mining technique that allows each data point to belong to more than one cluster with some degree of membership. It is widely employed in exploratory data mining to discover overlapping communities in social networks, find structure in spectral data, and capture user interests in recommendation systems. Nowadays, the variety and volume of data are increasing at a tremendous rate. Data is power; the massive data, along with an effective technique, can unravel valuable information. The existing fuzzy clustering algorithms do not perform well on massive heterogeneous datasets. Processing an enormous amount of data is beyond the capacity of a single processor. The need of the hour is to develop fuzzy clustering techniques that can work on a distributed framework for Big Data processing and can handle heterogeneous data. In this research, we evaluate the performance of the recently proposed algorithm for the Fuzzy clustering of mixed-mode data FCMD-MD (D’Urso and Massari in Inf Sci 505:513–534, 2019) with different real-world datasets. We develop a distributed FCMD-MD, a fuzzy clustering algorithm for mixed-mode data in Apache SPARK. The experimental results show that the algorithm is scalable, performs well in a distributed environment, and clusters enormous heterogeneous data with high accuracy. We also compared the performance of distributed FCMD-MD and the distributed k-medoid algorithm.

A brief survey on big data: technologies, terminologies and data-intensive applications

The technical advancements and the availability of massive amounts of data on the Internet draw huge attention from researchers in the areas of decision-making, data sciences, business applications, and government. These massive quantities of data, known as big data, have many benefits and applications for researchers. However, the use of big data consumes a lot of time and imposes enormous computational complexity. This survey describes the significance of big data and its taxonomy and details the basic terminologies used in big data. It also discusses the technologies used in big data applications as well as their various complexities and challenges. The survey focuses on the various techniques presented in the literature to restrain the issues associated with big data. In particular, the review concentrates on big data techniques in accordance with processing, security, and storage. It also discusses the various parameters associated with big data, such as availability and velocity. The study analyses big data terminologies and techniques in accordance with several factors, such as year of publication, performance metrics, achievement of the existing models, and methods utilized. Finally, this review article describes the future direction of research and highlights big data possibilities and solicitations with a detailed sketch of the big data processing frameworks.

SPsync: Lightweight multi-terminal big spatiotemporal data synchronization solution

With the rapid increase in mobile device usage and the popularity of 5G communication technology, rich Big spatiotemporal Data has become an important research resource. How to quickly synchronize the Big spatiotemporal Data collected by various types of devices to multiple distributed data centers has become the key to further utilizing spatio-temporal data. Therefore, in this paper, a new lightweight Big Spatiotemporal Data synchronization scheme called SPsync, based on a traditional file synchronization algorithm, is proposed. The contributions of SPsync are as follows. (1) SPsync leads to the optimization of the file processing strategy, changing the algorithm from a serial algorithm to a parallel one. (2) Combined with Spark, a distributed Big Data processing framework is advanced to achieve optimization of distributed tasks and further improvement of algorithm performance. Further, we demonstrate the excellent performance of SPsync through rich experiments using multiple simulated and real datasets. Compared to the best incremental synchronization algorithms currently available, the synchronization speedup is 30% faster and CPU usage 20% lower. The algorithm itself is 20% faster and CPU usage reduced by more than 25% in a multi-node scenario.

Data-Efficient Performance Modeling for Configurable Big Data Frameworks by Reducing Information Overlap Between Training Examples

To support the various analysis application of big data, big data processing frameworks are designed to be highly configurable. However, for common users, it is difficult to tailor the configurable frameworks to achieve optimal performance for every application. Recently, many automatic tuning methods are proposed to configure these frameworks. In detail, these methods firstly build a performance prediction model through sampling configurations randomly and measuring the corresponding performance. Then, they conduct heuristic search in the configuration space based on the performance prediction model. For most frameworks, it is too expensive to build the performance model since it needs to measure the performance of large amounts of configurations, which cause too much overhead on data collection. In this paper, we propose a novel data-efficient method to build the performance model with little impact on prediction accuracy. Compared to the traditional methods, the proposed method can reduce the overhead of data collection because it can train the performance model with much less training examples. Specifically, the proposed method can actively sample the important examples according to the dynamic requirement of the performance model during the iterative model updating. Hence, it can make full use of the collected informative data and train the performance model with much less training examples. To sample the important training examples, we employ several virtual performance model to estimate the importance of all candidate configurations efficiently. Experimental results show that our method needs less training examples than traditional methods with little impact on prediction accuracy.

A hybrid recommender system for recommending smartphones to prospective customers

Recommender Systems are a subclass of machine learning systems that employ sophisticated information filtering strategies to reduce the search time and suggest the most relevant items to any particular user. Hybrid recommender systems combine multiple recommendation strategies in different ways to benefit from their complementary advantages. Some hybrid recommender systems have combined collaborative filtering and content-based approaches to build systems that are more robust. In this paper, we propose a hybrid recommender system, which combines Alternating Least Squares (ALS) based collaborative filtering with deep learning to enhance recommendation performance as well as overcome the limitations associated with the collaborative filtering approach, especially concerning its “cold-start” problem. In essence, we use the outputs from ALS (collaborative filtering) to influence the recommendations from a Deep Neural Network (DNN), which combines characteristic, contextual, structural and sequential information, in a big data processing framework. We have conducted several experiments in testing the efficacy of the proposed hybrid architecture in recommending smartphones to prospective customers and compared its performance with other open-source recommenders. The results have shown that the proposed system has outperformed several existing hybrid recommender systems.

Query Execution Time Analysis Using Apache Spark Framework for Big Data: A CRM Approach

Customer Relationship Management (CRM) is a systematic way of working with current and prospective customers to manage long-term relationships and interactions between the company and customers. Recently, Big Data has become a buzzword. It consists of huge data repositories, having information collected from online and offline resources, and it is hard to process such datasets with traditional data processing tools and techniques. The presented research work tries to explore the potential of Big Data to create, optimise and transform an insightful customer relationship management system by analysing large amount of datasets for enhancing customer life cycle profitability. In this research work, a dataset, “Book Crossing” is used for Big Data processing and execution time analysis for simple and complex SQL queries. This research tries to analyse the impact of data size on the query execution time for one of the majorly used Big Data frameworks, namely Apache Spark. It is a recently developed in-memory Big Data processing framework with a SPARK SQL module for efficient SQL query execution. It has been found that Apache-Spark gives better results with large size datasets compare to small size datasets and fares better as compared to Hadoop, one of the majorly used Big Data Frameworks (based on qualitative analysis).

A two-level formal model for Big Data processing programs

• A high-level model for iterative and non-iterative large-scale data processing programs. • Formalizing data flow with Petri Nets and data processing operations with Monoid Algebra. • Application of the model to formalize mutation operators for mutation testing in data flow programs. • Proposal of new mutation operators for iterative data flow programs. This paper proposes a model for specifying data flow-based parallel data processing programs agnostic of target Big Data processing frameworks. The paper focuses on the formal abstract specification of non-iterative and iterative programs, generalizing the strategies adopted by data flow Big Data processing frameworks. The proposed model relies on Monoid Algebra and Petri Nets to abstract Big Data processing programs in two levels: a higher level representing the program data flow and a lower level representing data transformation operations (e.g., filtering, aggregation, join). We extend the model for data processing programs proposed in [1] , for modeling iterative data processing programs. The general specification of these programs implemented by data flow-based parallel programming models is essential given the democratization of iterative and greedy Big Data analytics algorithms. Indeed, these algorithms call for revisiting parallel programming models to express iterations. The paper gives a comparative analysis of the iteration strategies proposed by Apache Spark, DryadLINQ, Apache Beam, and Apache Flink. It discusses how the model achieves to generalize these strategies.

Privacy Preserving with Modified Grey Wolf Optimization Over Big Data Using Optimal K Anonymization Approach

An optimal approach to anonymization using small data is proposed in this study. Map Reduce is a big data processing framework used across distributed applications. Prior to the development of a map reduce framework, data are distributed and clustered using a hybrid clustering algorithm. The algorithm used for grouping together similar techniques utilises the k-means clustering algorithm, along with the MFCM clustering algorithm. Clustered data is then fed into the map reduce frame work after it has been clustered. In order to guarantee privacy, the optimal k anonymization method is recommended. When using generalisation and randomization, there are two techniques that can be employed: K-anonymity, which is unique to each, depends on the type of the quasi identifier attribute. Our method replaces the standard k anonymization process by employing an optimization algorithm that dynamically determines the optimal k value. This algorithm uses the Modified Grey Wolf Optimization (MGWO) algorithm for optimization. The memory, execution time, accuracy, and error value are used to assess the recommended method’s practise. This experiment has shown that the suggested method will always finish ahead of the existing method by using the least amount of time while ensuring the greatest level of security. The current technique gets the lowest accuracy and the privacy proposed achieves the maximum accuracy while compared to the current technique. The solution is implemented in Java with Hadoop Map-Reduce, and it is tested and deployed in the cloud on Google Cloud Platform.

A hybrid machine learning approach for detecting unprecedented DDoS attacks.

Service availability plays a vital role on computer networks, against which Distributed Denial of Service (DDoS) attacks are an increasingly growing threat each year. Machine learning (ML) is a promising approach widely used for DDoS detection, which obtains satisfactory results for pre-known attacks. However, they are almost incapable of detecting unknown malicious traffic. This paper proposes a novel method combining both supervised and unsupervised algorithms. First, a clustering algorithm separates the anomalous traffic from the normal data using several flow-based features. Then, using certain statistical measures, a classification algorithm is used to label the clusters. Employing a big data processing framework, we evaluate the proposed method by training on the CICIDS2017 dataset and testing on a different set of attacks provided in the more up-to-date CICDDoS2019. The results demonstrate that the Positive Likelihood Ratio (LR+) of our method is approximately 198% higher than the ML classification algorithms.

Trust-Based Scheduling Framework for Big Data Processing with MapReduce

Security and privacy have become a great concern in cloud computing platforms in which users risk the leakage of their private data. The leakage can happen while the data is at rest (in storage), in processing, or on moving within a cloud or between different cloud infrastructures, e.g., from private to public clouds. This paper focuses on protecting data “in processing”. For big data applications, the MapReduce framework has been proven as an efficient solution and has been widely deployed, e.g., in healthcare and business data analysis. In this article, we propose a trust-based framework for MapReduce in big data processing tasks. Specifically, we first quantify and propose to assign the sensitive values for data and trust values for map and reduce slots. We then compute the trust value of each resource employed in the big data processing tasks. Depending on the data's sensitivity level of a task, the task requires a given level of trust (i.e., higher sensitive data requires servers/slots with higher trust level). The MapReduce scheduling problem is then formulated as the maximum weighted matching problem of a bipartite graph that aims to maximize the total trust value over all possible assignments subject to various trust requirement of different tasks. The problem is known to be NP-hard. To tackle it, we observe that within a computing node (VM), slots share the same trust value granted from the secured transformation phase. This helps reduce the number of slot nodes of a weight bipartite graph. Leveraging this fact, we propose an efficient heuristic algorithm that achieves 94.7 percent of the optimal solution obtained via exhaustive search. Extensive simulations show that the trust-based scheduling scheme provides much higher protection for data sensitivity while ensuring good performance for big data applications.

Straggler Mitigation in Hadoop MapReduce Framework: A Review

Processing huge and complex data to obtain useful information is challenging, even though several big data processing frameworks have been proposed and further enhanced. One of the prominent big data processing frameworks is MapReduce. The main concept of MapReduce framework relies on distributed and parallel processing. However, MapReduce framework is facing serious performance degradations due to the slow execution of certain tasks type called stragglers. Failing to handle stragglers causes delay and affects the overall job execution time. Meanwhile, several straggler reduction techniques have been proposed to improve the MapReduce performance. This study provides a comprehensive and qualitative review of the different existing straggler mitigation solutions. In addition, a taxonomy of the available straggler mitigation solutions is presented. Critical research issues and future research directions are identified and discussed to guide researchers and scholars.

Proactive and intelligent evaluation of big data queries in edge clouds with materialized views

The rise of big data brings extraordinary benefits and opportunities to businesses and governments. Enterprise users can analyze their consumers’ data and infer the business value, such as purchasing goods correlations and customer preferences. To unveil such hidden values from big data, various big data processing frameworks, such as Hadoop and Tensor-flow, are developed. One fundamental approach of accelerating such big data processing frameworks is to efficiently evaluate queries for big data analytics based on materialization of intermediate results. In this paper, we consider problems of dynamic and proactive QoS-aware query evaluation of big data analytics with intermediate result materialization in a mobile edge cloud. We propose a one-shot online algorithm for the dynamic QoS-aware big data query evaluation within a finite time horizon, which can intelligently determine whether some immediate results during query evaluation need to be materialized for later use of other queries, by making use of the Reinforcement Learning (RL) method with predictions. We also devise an online learning algorithm based on the technique of multi-armed bandits, for the proactive QoS-aware big data query evaluation problem with resource reservations, where the query arrival information and their required datasets are uncertain before the actual evaluation of the queries. We finally investigate the performance of the proposed algorithms by experimental simulations, and results show that the performance of the proposed algorithms is promising, by achieving 12% higher system throughput while reducing 50% and 40% average evaluation cost and time per query compared to the comparison benchmarks.

Unified Holistic Memory Management Supporting Multiple Big Data Processing Frameworks over Hybrid Memories

To process real-world datasets, modern data-parallel systems often require extremely large amounts of memory, which are both costly and energy inefficient. Emerging non-volatile memory (NVM) technologies offer high capacity compared to DRAM and low energy compared to SSDs. Hence, NVMs have the potential to fundamentally change the dichotomy between DRAM and durable storage in Big Data processing. However, most Big Data applications are written in managed languages and executed on top of a managed runtime that already performs various dimensions of memory management. Supporting hybrid physical memories adds a new dimension, creating unique challenges in data replacement. This article proposes Panthera, a semantics-aware, fully automated memory management technique for Big Data processing over hybrid memories. Panthera analyzes user programs on a Big Data system to infer their coarse-grained access patterns, which are then passed to the Panthera runtime for efficient data placement and migration. For Big Data applications, the coarse-grained data division information is accurate enough to guide the GC for data layout, which hardly incurs overhead in data monitoring and moving. We implemented Panthera in OpenJDK and Apache Spark. Based on Big Data applications’ memory access pattern, we also implemented a new profiling-guided optimization strategy, which is transparent to applications. With this optimization, our extensive evaluation demonstrates that Panthera reduces energy by 32–53% at less than 1% time overhead on average. To show Panthera’s applicability, we extend it to QuickCached, a pure Java implementation of Memcached. Our evaluation results show that Panthera reduces energy by 28.7% at 5.2% time overhead on average.

Gradient Boosting Machine and Deep Learning Approach in Big Data Analysis

Designing a system for analytics of high-frequency data (Big data) is a very challenging and crucial task in data science. Big data analytics involves the development of an efficient machine learning algorithm and big data processing techniques or frameworks. Today, the development of the data processing system is in high demand for processing high-frequency data in a very efficient manner. This paper proposes the processing and analytics of stochastic high-frequency stock market data using a modified version of suitable Gradient Boosting Machine (GBM). The experimental results obtained are compared with deep learning and Auto-Regressive Integrated Moving Average (ARIMA) methods. The results obtained using modified GBM achieves the highest accuracy (R2 = 0.98) and minimum error (RMSE = 0.85) as compared to the other two approaches.