Spark Application Research Articles

A mediation system for continuous spatial queries on a unified schema using Apache Spark

ABSTRACT Recent advances in big and streaming data systems have enabled real-time analysis of data generated by Internet of Things (IoT) systems and sensors in various domains. In this context, many applications require integrating data from several heterogeneous sources, either stream or static sources. Frameworks such as Apache Spark are able to integrate and process large datasets from different sources. However, these frameworks are hard to use when the data sources are heterogeneous and numerous. To address this issue, we propose a system based on mediation techniques for integrating stream and static data sources. The integration process of our system consists of three main steps: configuration, query expression and query execution. In the configuration step, an administrator designs a mediated schema and defines mapping between the mediated schema and local data sources. In the query expression step, users express queries using customized SQL grammar on the mediated schema. Finally, our system rewrites the query into an optimized Spark application and submits the application to a Spark cluster. The results are continuously returned to users. Our experiments show that our optimizations can improve query execution time by up to one order of magnitude, making complex streaming and spatial data analysis more accessible.

Optimizing computational costs of Spark for SARS‐CoV‐2 sequences comparisons on a commercial cloud

SummaryCloud computing is currently one of the prime choices in the computing infrastructure landscape. In addition to advantages such as the pay‐per‐use bill model and resource elasticity, there are technical benefits regarding heterogeneity and large‐scale configuration. Alongside the classical need for performance, for example, time, space, and energy, there is an interest in the financial cost that might come from budget constraints. Based on scalability considerations and the pricing model of traditional public clouds, a reasonable optimization strategy output could be the most suitable configuration of virtual machines to run a specific workload. From the perspective of runtime and monetary cost optimizations, we provide the adaptation of a Hadoop applications execution cost model extracted from the literature aiming at Spark applications modeled with the MapReduce paradigm. We evaluate our optimizer model executing an improved version of the Diff Sequences Spark application to perform SARS‐CoV‐2 coronavirus pairwise sequence comparisons using the AWS EC2's virtual machine instances. The experimental results with our model outperformed 80% of the random resource selection scenarios. By only employing spot worker nodes exposed to revocation scenarios rather than on‐demand workers, we obtained an average monetary cost reduction of 35.66% with a slight runtime increase of 3.36%.

Spark discharge energy effect on in-cylinder characteristics performance of rapid compression and expansion machine with spark ignition direct injection strategy

A spark ignition strategy in compression ignition has been considered to achieve high performance of gasoline direct injection engines. The spark ignition energy depends on the discharge current and the spark discharge duration. However, the respective roles of both on the rapid compression and expansion machine (RCEM) running with spark ignition and gasoline direct injection in-cylinder pressure have not been fully investigated. Both experiments and simulations were carried out in this study to investigate the spark discharge energy’s effect on in-cylinder characteristics performance. Three spark ignition strategy ranging from 50 mA up to 200 mA is applied to enhance the ignition discharge energy according to six selected cases of ignition timing variation (0.7 ms, 1.0 ms, 2.0 ms, 3.0 ms, 4.0 ms, and 5.0 ms). The results show the total released energy at the simultaneous ignition strategy was the largest (around 190 mJ/s), followed in order by the pair ignition strategy and individual ignition strategy. The highest turbulent kinetic energy of gasoline RCEM with spark ignition strategy is 1.43 m2/s2 (an increase 27.96 % from RCEM fueled with diesel). When the start of injection (SOI) of direct injection gasoline was set the same as the SOI of direct injection diesel at 10 oCA BTDC resulted in the thermal efficiency of gasoline RCEM with spark application slightly increased when spark time duration is prolonged at the average of 15.96 %. Finally, the establishment of a compression ignition engine combined with spark energy discharge might contribute to faster plasma formation, as well as expose the kernel development to a wider range of spark duration.

A Novel Reinforcement Learning Approach for Spark Configuration Parameter Optimization.

Apache Spark is a popular open-source distributed data processing framework that can efficiently process massive amounts of data. It provides more than 180 configuration parameters for users to manually select the appropriate parameter values according to their own experience. However, due to the large number of parameters and the inherent correlation between them, manual tuning is very tedious. To solve the problem of tuning through personal experience, we designed and implemented a reinforcement-learning-based Spark configuration parameter optimizer. First, we trained a Spark application performance prediction model with deep neural networks, and verified the accuracy and effectiveness of the model from multiple perspectives. Second, in order to improve the search efficiency of better configuration parameters, we improved the Q-learning algorithm, and automatically set start and end states in each iteration of training, which effectively improves the agent’s poor performance in exploring better configuration parameters. Lastly, comparing our proposed configuration with the default configuration as the baseline, experimental results show that the optimized configuration gained an average performance improvement of 47%, 43%, 31%, and 45% for four different types of Spark applications, which indicates that our Spark configuration parameter optimizer could efficiently find the better configuration parameters and improve the performance of various Spark applications.

Analisa dan Perancangan Machine Learning Untuk Mendeteksi Kegagalan Job di Apache Spark

A collection of data stored in a database, so the longer the data, the bigger the data, because the data processed is very large, processing time in Apache Spark can take up to a dozen or tens of hours. Sometimes, the Apache Spark application even fails. Therefore, to minimize the waiting time that could have been avoided or reduced, artificial intelligence through Machine Learning will be used to detect whether an Apache Spark application will fail or run smoothly. Factors to determine this failure are called features and are generated through the feature engineering process. The purpose of this research is to design Machine Learning so that it is able to find out what features will determine the success or failure of the Apache Spark application. The research method used is the Prototyping process model.

Spark Performance Optimization Analysis With Multi-Layer Parameter Using Shuffling and Scheduling With Data Serialization in Different Data Caching Options

As social networking services and e-commerce are growing rapidly, the number of online users also dynamically growing that facilitate contribution of huge contents to digital world. In such dynamic environment, meeting the demand of computing is very challenging special with existing computing model. Although Spark is recently introduced to alleviate the problems with concept of in-memory computing for big data analytic with many parameters configuration that allow to configure and improve its performance, still it has performance bottleneck which require to investigate performance improvement mechanism by focus on the combinations of Scheduling and Shuffle Manager with data serialization with intermediate data caching options. Standalone cluster computing model was selected as experimental methodology with submit command line for data submission. Three Spark application such as WorkCount, TeraSort and PageRank were selected and developed for experiment. As a result, 2.45% and 8.01% performance improvement are achieved in OFFHEAP and Memory Only Ser data caching option, respectively.

Investigation of the electrospark coating, alloying and strengthening technology

Introduction. The work objectives were the analysis and application of the technology of electrospark deposition of wear-resistant metal coatings on cutting tools or machine parts for their hardening or dimensional restoration.Materials and Methods. The technology, device and principle of operation of the modernized installation intended for electric spark application of wear-resistant metal coatings with composites T15K6, VK8 and VK6 are considered.Results. To determine the parameters of the upgraded electrospark alloying plant, experiments were carried out on hardening of polished samples made of steel 45 with hard alloy T15K6 with dimensions of 25×25×25 mm. As a result of using the experiment planning method, the possibility of selecting and adjusting the installation parameters was confirmed. The following parameters were selected for hardening samples made of steel 45 with hard alloy T15K6: current I = 1-2–A, voltage U = 40-75 V, capacitor bank capacity = 60–100 µF.Discussion and Conclusions. The use of carbon dioxide as a protective medium enables to increase the number of passes and, accordingly, the number of coating layers to twenty, to obtain a total thickness of up to 0.3 mm with a dense structure without oxides. Coatings of this thickness make it possible not only to strengthen, but also to restore the dimensions of worn machine parts. The parameters of the technological modes of electrospark alloying significantly affect the intensity of coating application and the quality of the resulting surface. A rise in the electrical parameters causes an increase in the intensity of each individual discharge and, within certain limits, contributes to an increase in the amount of the transferred coating material, as well as to deeper transformations of the coated surface in the discharge zone. Thus, an electrospark alloying plant equipped with monitoring and diagnostic tools, as well as with a protective gas supply system, can be used for hardening and restoring machine parts and cutting tools.

A comprehensive social media data processing and analytics architecture by using big data platforms: a case study of twitter flood-risk messages.

The main objective of the article is to propose an advanced architecture and workflow based on Apache Hadoop and Apache Spark big data platforms. The primary purpose of the presented architecture is collecting, storing, processing, and analysing intensive data from social media streams. This paper presents how the proposed architecture and data workflow can be applied to analyse Tweets with a specific flood topic. The secondary objective, trying to describe the flood alert situation by using only Tweet messages and exploring the informative potential of such data is demonstrated as well. The predictive machine learning approach based on Bayes Theorem was utilized to classify flood and no flood messages. For this study, approximately 100,000 Twitter messages were processed and analysed. Messages were related to the flooding domain and collected over a period of 5 days (14 May – 18 May 2018). Spark application was developed to run data processing commands automatically and to generate the appropriate output data. Results confirmed the advantages of many well-known features of Spark and Hadoop in social media data processing. It was noted that such technologies are prepared to deal with social media data streams, but there are still challenges that one has to take into account. Based on the flood tweet analysis, it was observed that Twitter messages with some considerations are informative enough to be used to estimate general flood alert situations in particular regions. Text analysis techniques proved that Twitter messages contain valuable flood-spatial information.

Diaspore: Diagnosing Performance Interference in Apache Spark

Apache Spark is being increasingly used to execute big data applications on cluster computing platforms. To increase system utilization, cluster operators often configure their clusters such that multiple co-located applications can simultaneously share the resources of a cluster node. With resource sharing, applications can compete with each other for shared node resources thereby interfering with each other's performance. Many Spark applications take a long time to execute. Performance interference from other applications can thus cause a Spark application to fail or take even longer time to execute thereby wasting cluster resources and frustrating users. This motivates the need for an automated technique that can detect interference quickly and also diagnose the root cause of the interference to facilitate mitigation of the problem. Most existing approaches are not designed to offer quick interference detection and diagnosis. For example, they typically require extensive training data for every application of interest under various possible input data sizes and resource allocations. In this paper, we systematically investigate the design of a Machine Learning (ML) based technique that addresses this open problem. We implement a tool called Diaspore that integrates our findings. We evaluate the tool with a diverse set of 13 Spark applications executing on a real cluster. Experimental results show that Diaspore requires only small scale training data, i.e., executions under small input sizes and resource allocations. Furthermore, our results show that the tool can offer accurate predictions for applications not present in the training data. Consequently, Diaspore reduces the training time needed to offer predictions. Finally, the feature engineering underlying Diaspore ensures that the tool can detect and diagnose interference quickly in an online manner by sampling only a small fraction of a long running application's execution. This can allow cluster operators to mitigate interference in an agile manner.

Efficient Performance Prediction for Apache Spark

Spark is a more efficient distributed big data processing framework following Hadoop. It provides users with more than 180 adjustable configuration parameters, and how to choose the optimal configuration automatically to make the Spark application run effectively is challenging. The key to address the above challenge is having the ability to predict the performance of Spark applications in different configurations. This paper proposes a new approach based on Adaboost, which can efficiently and accurately predict the performance of a given application with a given Spark configuration. In our approach, Adaboost is used to build a set of performance models at the stage-level for Spark. To minimize the overhead of the modeling, we use the classic projective sampling, a data mining technique that allows us to collect as few training samples as possible while meeting the accuracy requirements. We evaluate the proposed approach on six typical Spark benchmarks with five input datasets. The experimental results show that our approach is less than the previously proposed approach in prediction error and cost.

Optimal Resource Allocation of Cloud-Based Spark Applications

Nowadays, the big data paradigm is consolidating its central position in the industry, as well as in society at large. Lots of applications, across disparate domains, operate on huge amounts of data and offer great advantages both for business and research. According to analysts, cloud computing adoption is steadily increasing to support big data analyses and Spark is expected to take a prominent market position for the next decade. As big data applications gain more and more importance over time and given the dynamic nature of cloud resources, it is fundamental to develop an intelligent resource management system to provide Quality of Service guarantees to end-users. This article presents a set of run-time optimization-based resource management policies for advanced big data analytics. Users submit Spark applications characterized by a priority and by a hard or soft deadline. Optimization policies address two scenarios: i) identification of the minimum capacity to run a Spark application within the deadline; ii) re-balance of the cloud resources in case of heavy load, minimising the weighted soft deadline application tardiness. The solution relies on an initial non-linear programming model formulation and a search space exploration based on simulation-optimization procedures. Spark application execution times are estimated by relying on a gamut of techniques, including machine learning, approximated analyses, and simulation. The benefits of the approach are evaluated on Microsoft Azure HDInsight and on a private cloud cluster based on POWER8 by considering the TPC-DS industry benchmark and SparkBench. The results obtained in the first scenario demonstrate that the percentage error of the prediction of the optimal resource usage with respect to system measurement and exhaustive search is in the range 4–29 percent while literature-based techniques present an average error in the range 6–63 percent. Moreover, in the second scenario, the proposed algorithms can address complex problems like computing the optimal redistribution of resources among tens of applications in less than a minute with an error of 8 percent on average. On the same considered tests, literature-based approaches obtain an average error of about 57 percent.

Detection of Erratic Behavior in Load Balanced Clusters of Servers Using a Machine Learning Based Method

With the explosion of the number of distributed applications, a new dynamic server environment emerged grouping servers into clusters, whose utilization depends on the current demand for the application. To provide reliable and smooth services it is crucial to detect and fix possible erratic behavior of individual servers in these clusters. Use of standard techniques for this purpose delivers suboptimal results. We have developed a method based on machine learning techniques which allows detecting outliers indicating a possible problematic situation. The method inspects the performance of the rest of the cluster and provides system operators with additional information which allows them to identify quickly the failing nodes. We applied this method to develop a Spark application using the CERN MONIT architecture and with this application, we analyzed monitoring data from multiple clusters of dedicated servers in the CERN data center. In this contribution, we present our results achieved with this new method and with the Spark application for analytics of CERN monitoring data.

A Machine Learning Approach for Predicting Execution Time of Spark Jobs

Spark has gained growing attention in the past couple of years as an in-memory cloud computing platform. It supports execution of various types of workloads such as SQL queries and machine learning applications. Currently, many enterprises use Spark to exploit its fast in-memory processing of large scale data. Additionally, speeding up the execution in Spark is an important problem for many real-time applications. This can be achieved by improving the scheduling approaches employed by Spark, optimizing the execution plans generated by Spark for various applications, and selecting the best cluster configuration to run an input workload. A first step for all these optimization approaches is to predict the execution time of an input Spark application. In this paper, we present a new platform that predicts with high accuracy the execution time of SQL queries and machine learning applications executed by Spark. We evaluate our proposed platform by measuring the accuracy of predicting execution time of various types of Spark jobs including TPC-H queries and machine learning classification/clustering applications. The evaluation experiments show that we are able to predict the execution time of Spark jobs using our proposed platform with accuracy greater than 90% for SQL queries and greater than 75% for machine learning jobs.

A gray-box performance model for Apache Spark

Apache Spark is a powerful open source data processing platform. It is getting more and more popular with the growing need of processing massive amounts of data. A performance prediction model not only helps administrators to have a better understanding of system behavior, but also is useful in performance tuning. However, considering the complex application processing mechanism of Spark, it is not an easy job to model the relationship between system performance and configuration settings.In this paper, we present a gray-box performance model for Spark applications based on machine learning algorithms. Given a specific Spark application, the size of its input data and some key system parameters, this performance model is able to forecast its execution time according to history information. To achieve better accuracy, our model takes basic hardware information and the resource allocation strategy of Spark into consideration.In our experiments, result shows our gray-box model is better than typical black-box approaches in most of the cases. We consider this model is helpful for further researches on Apache Spark.

Query-able Kafka

Due to their promise of delivering real-time network insights, today's streaming analytics platforms are increasingly being used in the communications networks where the impact of the insights go beyond sentiment and trend analysis to include real-time detection of security attacks and prediction of network state (i.e., is the network transitioning towards an outage). Current streaming analytics platforms operate under the assumption that arriving traffic is to the order of kilobytes produced at very high frequencies. However, communications networks, especially the telecommunication networks, challenge this assumption because some of the arriving traffic in these networks is to the order of gigabytes, but produced at medium to low velocities. Furthermore, these large datasets may need to be ingested in their entirety to render network insights in real-time. Our interest is to subject today's streaming analytics platforms --- constructed from state-of-the art software components (Kafka, Spark, HDFS, ElasticSearch) --- to traffic densities observed in such communications networks. We find that filtering on such large datasets is best done in a common upstream point instead of being pushed to, and repeated, in downstream components. To demonstrate the advantages of such an approach, we modify Apache Kafka to perform limited native data transformation and filtering, relieving the downstream Spark application from doing this. Our approach outperforms four prevalent analytics pipeline architectures with negligible overhead compared to standard Kafka. (Our modifications to Apache Kafka are publicly available at https://github.com/Esquive/queryable-kafka.git)

Dynamic Configuration of Partitioning in Spark Applications

Spark has become one of the main options for large-scale analytics running on top of shared-nothing clusters. This work aims to make a deep dive into the parallelism configuration and shed light on the behavior of parallel spark jobs. It is motivated by the fact that running a Spark application on all the available processors does not necessarily imply lower running time, while may entail waste of resources. We first propose analytical models for expressing the running time as a function of the number of machines employed. We then take another step, namely to present novel algorithms for configuring dynamic partitioning with a view to minimizing resource consumption without sacrificing running time beyond a user-defined limit. The problem we target is NP-hard. To tackle it, we propose a greedy approach after introducing the notions of dependency graphs and of the benefit from modifying the degree of partitioning at a stage; complementarily, we investigate a randomized approach. Our polynomial solutions are capable of judiciously use the resources that are potentially at user's disposal and strike interesting trade-offs between running time and resource consumption. Their efficiency is thoroughly investigated through experiments based on real execution data.

Integrating Apache Spark and External Data Sources Using Hadoop Interfaces

Requirements of data-processing are undergoing a profound transition with the dramatic increase of various application data. Along with this demands, sorts of storage systems with highly scalable are developed for large-scale data sources, and Apache Spark for immense amounts of data calculation has also captured attentions and excitements of the industry since release for its excellent performance. Technologies of combining Spark and external data sources have lots of potential to produce valuable data analysis platforms for solving a wide range of data-handling needs, accordingly it's very easy to shift from Hadoop to Spark application development since the industry pour a lot in these datastore system for Hadoop in the past and Spark can work well with Hadoop-supported systems. Paper delves into integration mechanism of spark and external data sources, such as NoSQL and relational databases (RDBs) and proposes a reference for tight and efficient integration of the two using Hadoop interfaces.