Spark Tool Research Articles

Containers for Specification in SPARK

The SPARK tool analyzes Ada programs statically. It can be used to verify both that a program is free from runtime exceptions and that it conforms to a specification expressed through contracts. To facilitate dynamic analysis, Ada contracts are regular expressions which can be evaluated at execution. As a result, the annotation language of the SPARK tool is restricted to executable constructs. In this context, high-level concepts necessary for specification by contracts need to be supplied as libraries. For example, the latest version of the Ada language introduces unbounded integers and rational numbers to the standard library. In this article, we present the functional containers library which provides collections suitable for use in specification. We explain how they can be used to specify and verify complex programs through concrete examples that have been developed over many years.

Self-Tuning Parameters for Decision Tree Algorithm Based on Big Data Analytics

Big data is usually unstructured, and many applications require the analysis in real-time. Decision tree (DT) algorithm is widely used to analyze big data. Selecting the optimal depth of DT is time-consuming process as it requires many iterations. In this paper, we have designed a modified version of a (DT). The tree aims to achieve optimal depth by self-tuning running parameters and improving the accuracy. The efficiency of the modified (DT) was verified using two datasets (airport and fire datasets). The airport dataset has 500000 instances and the fire dataset has 600000 instances. A comparison has been made between the modified (DT) and standard (DT) with results showing that the modified performs better. This comparison was conducted on multi-node on Apache Spark tool using Amazon web services. Resulting in accuracy with an increase of 6.85% for the first dataset and 8.85% for the airport dataset. In conclusion, the modified DT showed better accuracy in handling different-sized datasets compared to standard DT algorithm.

Digital agriculture based on big data analytics: a focus on predictive irrigation for smart farming in Morocco

Due to the spead of objects connected to the internet and objects connected to each other, agriculture nowadays knows a huge volume of data exchanged called big data. Therefore, this paper discusses connected agriculture or agriculture 4.0 instead of a traditional one. As irrigation is one of the foremost challenges in agriculture, it is also moved from manual watering towards smart watering based on big data analytics where the farmer can water crops regularly and without wastage even remotely. The method used in this paper combines big data, remote sensing and data mining algorithms (neural network and support vector machine). In this paper, we are interfacing the databricks platform based on the apache Spark tool for using machine learning to predict the soil drought based on detecting the soil moisture and temperature.

Building a Personalized Fitness Recommendation Application based on Sequential Information

Nowadays sports plays a very important role in the life of the human being and it allows to keep him healthy and make him always active. Sport is essential for people to have a healthy mind. However, the practice of a sport can have negative effects on the body and human health if it is practiced incorrectly or if it is not adapted to the body or the human health. This is why, in this paper, we have proposed a recommendations system that allows the selection of the right person to practice the right sport according to several factors such as heart rate, speed and size. The implementation was applied to the FitRec dataset with the help of SPARK tool, and the results show that the proposed method is capable of generating the appropriate training for different groups according to their information, where each group gets the appropriate training. The grouping of this data was done by the k-means method.

Real‐Time Twitter Trend Analysis Using Big Data Analytics and Machine Learning Techniques

Twitter is a popular microblogging social media, using which its users can share useful information. Keeping a track of user postings and common hashtags allows us to understand what is happening around the world and what are people’s opinions on it. As such, a Twitter trend analysis analyzes Twitter data and hashtags to determine what topics are being talked about the most on Twitter. Feature extraction and trend detection can be performed using machine learning algorithms. Big data tools and techniques are needed to extract relevant information from continuous steam of data originating from Twitter. The objectives of this research work are to analyze the relative popularity of different hashtags and which field has the maximum share of voice. Along with this, the common interests of the community can also be determined. Twitter trends plan an important role in the business field, marketing, politics, sports, and entertainment activities. The proposed work implemented the Twitter trend analysis using latent Dirichlet allocation, cosine similarity, K means clustering, and Jaccard similarity techniques and compared the results with Big Data Apache SPARK tool implementation. The LDA technique for trend analysis resulted in an accuracy of 74% and Jaccard with an accuracy of 83% for static data. The results proved that the real‐time tweets are analyzed comparatively faster in the Big Data Apache SPARK tool than in the normal execution environment.

Modified Data Storage and Replication Mechanism with Frequent Use-Case Based Indexing

The real time applications demands high speed and reliable data access from the remote database. An effective logical data management strategy that handles simultaneous connections with better performance negotiation is inevitable. This work considers an e-health care application that proposes MongoDB based modified indexing and performance tuning methods. To cope with certain high frequency use case and its performance mandates, a flexible and efficient logical data management may be preferred. By analysing the data dependency, data decomposition concerns and the performance requirements of the specific use case of the medical application, a logical schema may be customized on an ala-carte basis. This work focused on the flexible logical data modeling schemes and its performance factors of the NoSql DB. The efficiency of unstructured data base management in storing and retrieving the e-health care data was analysed with a web based tool. To enable faster data retrieval and query processing over the distributed nodes, a Spark based storage engine was built on top of the MongoDB based data storage management. With Spark tool, the database has been made distributed as master–slave structures with suitable data replication mechanisms. In such distributed database the fail-over also implemented with the suitable replication mechanism. This work considered MongoDB based flexible schema modeling and Spark based distributed computation with multiple chunks of data. The flexible data modeling scheme with MongoDB with the on-demand Spark based computation framework was proposed. To facilitate the eventual consistency, scalability aspects of the e-health care applications, use case based indexing was proposed. With the effective data management, faster query processing the horizontal scalability has been increased. The overall efficiency and scalability of the proposed logical data management approach was analysed. Through the simulation studies, the proposed approach has been claimed to boost the performance of the bigdata based application to a considerable extent.

An advanced comparison on big data world computing frameworks

Today’s internet world becomes really introducing a landmark between Peta to Exa byte which is significantly generating an enormous size of data while computing the digital things including the format of each datasets which signifies highly unstructured because which could be generating from different social sites, IOT, Google engine, Twitter, Yahoo, monitoring and controlling through sensors essentially called big data. Because of this fast era, we apply just contemporary techniques with common tools regarding having focused performing, smooth process and to execute computations on huge data. Though such tremendous universal data has some shortcomings for getting effective processing, analyzing the universe immense datasets and scalability techniques. Apache open free source Hadoop does the latest big data weapon which can process Zetta byte dimensions of databases by its most developed and popular components as hdfs and Map Reduce, to make up vast storage facility plus great administration in the sense to process zettabyte of datasets as powerfully, flexible. MR likes more famous software popular structure for handling big-data existing issues with full parallel, highly distributed and most scalable manner. However, public and unrestricted source tools on Hadoop, map reduces become major limitations like poor allocate process on needy resources working regarding stream-oriented processing, Shortage significant viewpoints like latency, dynamic manner execution, optimization, computing as online and diverse logical solutions. We consider significant various complex data computing orientated techniques. This study paper address Apache fastest spark tool, online-oriented tool public and unrestricted source and Flink are in Apache project are efficient frameworks to conquer that limitation.

Detección de anomalías en grandes volúmenes de datos

El desarrollo de la era digital ha traído como consecuencia un incremento considerable de los volúmenes de datos. A estos grandes volúmenes de datos se les ha denominado big data ya que exceden la capacidad de procesamiento de sistemas de bases de datos convencionales. Diversos sectores consideran varias oportunidades y aplicaciones en la detección de anomalías en problemas de big data. Para realizar este tipo de análisis puede resultar muy útil el empleo de técnicas de minería de datos porque permiten extraer patrones y relaciones desde grandes cantidades de datos. El procesamiento y análisis de estos volúmenes de datos, necesitan de herramientas capaces de procesarlos como Apache Spark y Hadoop. Estas herramientas no cuentan con algoritmos específicos para la detección de anomalías. El objetivo del trabajo es presentar un nuevo algoritmo para la detección de anomalías basado en vecindad para de problemas big data. A partir de un estudio comparativo se seleccionó el algoritmo KNNW por sus resultados, con el fin de diseñar una variante big data. La implementación del algoritmo big data se realizó en la herramienta Apache Spark, utilizando el paradigma de programación paralela MapReduce. Posteriormente se realizaron diferentes experimentos para analizar el comportamiento del algoritmo con distintas configuraciones. Dentro de los experimentos se compararon los tiempos de ejecución y calidad de los resultados entre la variante secuencial y la variante big data. La variante big data obtuvo mejores resultados con diferencia significativa. Logrando que la variante big data, KNNW-BigData, pueda procesar grandes volúmenes de datos.

HANN: A Hybrid Model for Liver Syndrome Classification by Feature Assortment Optimization.

Early detection of any sort of disease is mandatory for effective medical treatment. Medical diagnosis relies heavily on Data Mining for automated disease classification and detection. It relies on data mining algorithms to examine medical data. Liver diseases have become more common these days with many new patients being diagnosed with Heptasis B and C. Early diagnosis of Liver Disorder is essential for treatment. It can be achieved by setting up intelligent systems for early diagnose and prognosis of Liver diseases. The existing automated classification systems lack accuracy in results when compared to surgical biopsy. We propose a new hybrid model for liver syndrome classification for analysis of the patient's medical data via hybrid artificial neural network. The medical records are classified based on the possibility of existence of disease. The proposed method uses M-PSO for feature selection of input variables and M-ANN algorithm for disease classification. The presented hybrid approach significantly improves the accuracy compared to existing classification algorithms. The results of the algorithm were examined and evaluated using Spark tool in this work.

A Binary Cuckoo Search Big Data Algorithm Applied to Large-Scale Crew Scheduling Problems

The progress of metaheuristic techniques, big data, and the Internet of things generates opportunities to performance improvements in complex industrial systems. This article explores the application of Big Data techniques in the implementation of metaheuristic algorithms with the purpose of applying it to decision-making in industrial processes. This exploration intends to evaluate the quality of the results and convergence times of the algorithm under different conditions in the number of solutions and the processing capacity. Under what conditions can we obtain acceptable results in an adequate number of iterations? In this article, we propose a cuckoo search binary algorithm using the MapReduce programming paradigm implemented in the Apache Spark tool. The algorithm is applied to different instances of the crew scheduling problem. The experiments show that the conditions for obtaining suitable results and iterations are specific to each problem and are not always satisfactory.

The SPARK Tool to prioritise questions for systematic reviews in health policy and systems research: development and initial validation

BackgroundGroups or institutions funding or conducting systematic reviews in health policy and systems research (HPSR) should prioritise topics according to the needs of policymakers and stakeholders. The aim of this study was to develop and validate a tool to prioritise questions for systematic reviews in HPSR.MethodsWe developed the tool following a four-step approach consisting of (1) the definition of the purpose and scope of tool, (2) item generation and reduction, (3) testing for content and face validity, (4) and pilot testing of the tool. The research team involved international experts in HPSR, systematic review methodology and tool development, led by the Center for Systematic Reviews on Health Policy and Systems Research (SPARK). We followed an inclusive approach in determining the final selection of items to allow customisation to the user’s needs.ResultsThe purpose of the SPARK tool was to prioritise questions in HPSR in order to address them in systematic reviews. In the item generation and reduction phase, an extensive literature search yielded 40 relevant articles, which were reviewed by the research team to create a preliminary list of 19 candidate items for inclusion in the tool. As part of testing for content and face validity, input from international experts led to the refining, changing, merging and addition of new items, and to organisation of the tool into two modules. Following pilot testing, we finalised the tool, with 22 items organised in two modules – the first module including 13 items to be rated by policymakers and stakeholders, and the second including 9 items to be rated by systematic review teams. Users can customise the tool to their needs, by omitting items that may not be applicable to their settings. We also developed a user manual that provides guidance on how to use the SPARK tool, along with signaling questions.ConclusionWe have developed and conducted initial validation of the SPARK tool to prioritise questions for systematic reviews in HPSR, along with a user manual. By aligning systematic review production to policy priorities, the tool will help support evidence-informed policymaking and reduce research waste. We invite others to contribute with additional real-life implementation of the tool.