Processing Large Volumes Of Data Research Articles

Performance Evaluation of Big Data Processing of Cloak-Reduce

Big Data has introduced the challenge of storing and processing large volumes of data (text, images, and videos). The success of centralised exploitation of massive data on a node is outdated, leading to the emergence of distributed storage, parallel processing and hybrid distributed storage and parallel processing frameworks. The main objective of this paper is to evaluate the load balancing and task allocation strategy of our hybrid distributed storage and parallel processing framework CLOAK-Reduce. To achieve this goal, we first performed a theoretical approach of the architecture and operation of some DHT-MapReduce. Then, we compared the data collected from their load balancing and task allocation strategy by simulation. Finally, the simulation results show that CLOAK-Reduce C5R5 replication provides better load balancing efficiency, MapReduce job submission with 10% churn or no churn.

Optimal Monitoring of Server Rooms with Home Assistant Platform

High-performance IT systems with high computing power resources, used especially for processing large volumes of data or performing laborious calculations at significant speeds, are installed in special server rooms. Such a room specially built for High-Performance Computing systems, also called HPC, involves soundproofing and fireproofing of the walls, air conditioning equipment, as well as permanent monitoring of the room with a series of sensors, among which can be mentioned: temperature and humidity sensors, smoke sensors, air quality sensors, water and flood sensors, motion sensors, video cameras for realtime remote viewing. Given the wide variety of smart devices on the market equipped with such sensors, it is advisable to choose a suitable software environment, which offers the possibility to integrate, monitor, and control all these smart devices. One of the most popular free, opensource automation applications is Home Assistant. This software platform brings an improvement in the server rooms through the capabilities of monitoring, automation, and notification of events through text messages (email, SMS) or acoustic and video alerts. The paper presents an IoT implementation of a server room monitoring system, applied at INCD INSEMEX headquarters.

Using HBase to Implement Speed Layer in Time Series Data Storage Systems

In recent years, modern systems have become increasingly integrated, and the challenges are focused on delivering real-time analytics based on big data. Thus, using standard software tools to extract information from such datasets is not always possible. The Lambda Architecture proposed by Marz is an architectural solution that can manage the processing of large data volumes by combining real-time and data batch processing techniques. Choosing a suitable database management system for storing large volumes of time series data is not a trivial issue as various aspects such as low latency, high performance and the possibility of horizontal scalability must be taken into account. The new NoSQL approaches use for this purpose non-relational databases with significant advantages in terms of flexibility and performance in comparison with the traditional relational databases. With reference to this, the purpose of this paper is to analyse the general characteristics of time series data and the main activities performed by the Speed layer in a system based on the Lambda Architecture. Based on this, the use of a column-oriented NoSQL DBMS as a system for storing time series data is justified. The paper also addresses the challenges of using HBase as a system for storing and analysing time series data. These questions are related to the design of an appropriate database schema, the need to achieve balance between ease of access to the data and performance as well as considering the factors that affect the overload of individual nodes in the system.

Towards functional safety compliance of matrix–matrix multiplication for machine learning-based autonomous systems

Autonomous systems execute complex tasks to perceive the environment and take self-aware decisions with limited human interaction. This autonomy is commonly achieved with the support of machine learning algorithms. The nature of these algorithms, that need to process large data volumes, poses high-performance demands on the underlying hardware. As a result, the embedded critical real-time domain is adopting increasingly powerful processors that combine multi-core processors with accelerators such as GPUs. The resulting hardware and software complexity makes it difficult to demonstrate that the system will run safely and reliably. This is the main objective of functional safety standards, such as IEC 61508 or ISO 26262, that deal with the avoidance, detection and control of hardware or software errors. In this paper, we adopt those measures for the safe inference of machine learning libraries on multi-core devices, two topics that are not explicitly covered in the current version of standards. To this end, we adapt the matrix-matrix multiplication function, a central element of existing machine learning libraries, according to the recommendations of functional safety standards. The paper makes the following contributions: (i) adoption of recommended programming practices for the avoidance of programming errors in the matrix-matrix multiplication, (ii) inclusion of diagnostic mechanisms based on widely used checksums to control runtime errors, and (iii) evaluation of the impact of previous measures in terms of performance and a quantification of the achieved diagnostic coverage. For this purpose, we implement the diagnostic mechanisms on one of the ARM R5 cores of a Zynq UltraScale+ multi-processor system-on-chip and we then adapt them to an Intel i7 processor with native code employing vectorization for the sake of performance.

The Cultural Outcomes of Social Movements: A Computational Linguistics Approach

Scholars have long established the importance of the cultural outcomes of social movements in the context of political power and representation. However, they have also acknowledged the methodo-logical difficulties associated with studying cultural outcomes, especially when culture is manifested through linguistic practices. This paper addresses the potential for dealing with movements and culture as mani-fested in symbols, public discourse, narratives, and rhetoric and makes two contributions: It links the social movement literature studying culture through language with Natural Language Processing (NLP) techniques for systematic and comprehensive cultural analysis; and introduces a state-of the-art method which pro-vides a better understanding of language change and linguistic influence given the capacity of computa-tional analyses to process large volumes of data for multiple actors and varied data sources during long pe-riods of time. The paper describes the cultural influence of women organizations in Spain between 2000-2020 on issues such as gender inequalities, abortion, gender violence, prostitution, and surrogacy. Tweets and manifestos by women's organizations', as well as national press coverage of women issues and inter-ventions by MPs in the parliamentary arena, are used to describe the advantages and limitations of the method for the study of cultural outcomes. Computational linguistics provides new possibilities for scholarly research on cultural outcomes of social movements but also shows that these methods should be accompa-nied by precise definitions of cultural outcomes, detailed and replicable operationalisation processes, and theoretical models that identify the mechanisms that explain the linguistic phenomena that underly cultural change.

Improving Barnes-Hut t-SNE Algorithm in Modern GPU Architectures with Random Forest KNN and Simulated Wide-Warp

The t-Distributed Stochastic Neighbor Embedding (t-SNE) is a widely used technique for dimensionality reduction but is limited by its scalability when applied to large datasets. Recently, BH-tSNE was proposed; this is a successful approximation that transforms a step of the original algorithm into an N-Body simulation problem that can be solved by a modified Barnes-Hut algorithm. However, this improvement still has limitations to process large data volumes (millions of records). Late studies, such as t-SNE-CUDA, have used GPUs to implement highly parallel BH-tSNE. In this research we have developed a new GPU BH-tSNE implementation that produces the embedding of multidimensional data points into three-dimensional space. We examine scalability issues in two of the most expensive steps of GPU BH-tSNE by using efficient memory access strategies , recent acceleration techniques , and a new approach to compute the KNN graph structure used in BH-tSNE with GPU. Our design allows up to 460% faster execution when compared to the t-SNE-CUDA implementation. Although our SIMD acceleration techniques were used in a modern GPU setup, we have also verified a potential for applications in the context of multi-core processors.

Efficient multi-angle polarimetric inversion of aerosols and ocean color powered by a deep neural network forward model

Abstract. NASA's Plankton, Aerosol, Cloud, ocean Ecosystem (PACE) mission, scheduled for launch in the timeframe of 2023, will carry a hyperspectral scanning radiometer named the Ocean Color Instrument (OCI) and two multi-angle polarimeters (MAPs): the UMBC Hyper-Angular Rainbow Polarimeter (HARP2) and the SRON Spectro-Polarimeter for Planetary EXploration one (SPEXone). The MAP measurements contain rich information on the microphysical properties of aerosols and hydrosols and therefore can be used to retrieve accurate aerosol properties for complex atmosphere and ocean systems. Most polarimetric aerosol retrieval algorithms utilize vector radiative transfer models iteratively in an optimization approach, which leads to high computational costs that limit their usage in the operational processing of large data volumes acquired by the MAP imagers. In this work, we propose a deep neural network (NN) forward model to represent the radiative transfer simulation of coupled atmosphere and ocean systems for applications to the HARP2 instrument and its predecessors. Through the evaluation of synthetic datasets for AirHARP (airborne version of HARP2), the NN model achieves a numerical accuracy smaller than the instrument uncertainties, with a running time of 0.01 s in a single CPU core or 1 ms in a GPU. Using the NN as a forward model, we built an efficient joint aerosol and ocean color retrieval algorithm called FastMAPOL, evolved from the well-validated Multi-Angular Polarimetric Ocean coLor (MAPOL) algorithm. Retrievals of aerosol properties and water-leaving signals were conducted on both the synthetic data and the AirHARP field measurements from the Aerosol Characterization from Polarimeter and Lidar (ACEPOL) campaign in 2017. From the validation with the synthetic data and the collocated High Spectral Resolution Lidar (HSRL) aerosol products, we demonstrated that the aerosol microphysical properties and water-leaving signals can be retrieved efficiently and within acceptable error. Comparing to the retrieval speed using a conventional radiative transfer forward model, the computational acceleration is 103 times faster with CPU or 104 times with GPU processors. The FastMAPOL algorithm can be used to operationally process the large volume of polarimetric data acquired by PACE and other future Earth-observing satellite missions with similar capabilities.

Анализ больших объемов данных: возможности GDELT Project при использовании языка программирования Python. Опыт гуманитария, решившего постичь Big Data

Научный мир развивается по междисциплинарному пути, одно из самых популярных направлений — соединение возможностей программирования, позволяющего обрабатывать большие объемы данных, и гуманитарного знания. В статье я описываю свой опыт освоения Big Data, анализирую возможности ресурса GDELT и показываю, как с помощью команд на языке программирования Python обрабатывать большие данные. Благодаря этому данная статья может иметь и вполне практическое применение — в ней перечислены и проанализированы шаги, которые позволят ученым, не знакомым с обработкой больших объемов данных, не только разобраться в сути метода, но и самостоятельно сделать первые шаги в обработке данных на Python. Кроме того, работа проиллюстрирована кейсом французских «желтых жилетов», что позволяет лучше разобраться в структуре кода и принципах работы GDELT.

Unmanned Aerial Vehicle (UAV) applications in coastal zone management-a review.

Climate change and intense anthropogenic activities have heightened the vulnerability of coastal areas globally. The intensification in the dynamism and uncertainty of coastal processes and change in the past few decades have led researchers and coastal managers to explore new tools with the capability of undertaking a rapid assessment of coastal resources at a relatively lower cost compared with the conventional in situ data collection. The latest advances in unmanned aerial vehicle (UAV) platforms and sensor technologies have made them useful environmental remote sensing tools due to the high temporal and spatial resolution and relatively inexpensive operating costs. This study reviews literature that explored UAV applications in five different areas of the coastal zone comprising the intertidal, coastal organisms and habitats, marine litter, coastal zone disaster management, and coastal zone land use and land cover mapping. The review provides evidence of the potentials and effectiveness of UAVs for coastal zone management (CZM). However, factors such as difficulty in imaging water, setting out ground control points (GCPs) for geolocation of images, and processing large volumes of data can pose a challenge to coastal managers. Extensive review shows the capabilities of current UAV technologies for monitoring and tracking changes in the coastal environment at high spatial and temporal resolution.

Formation of sets of independent components of a multidimensional random variable based on a nonparametric pattern recognition algorithm

The possibility of circumventing the problem of decomposition of the range of values of random variables when testing various hypotheses is considered. A brief review of the literature on this problem is given. A method for forming sets of independent components of a multidimensional random variable is proposed, based on hypotheses testing about the independence of paired combinations of components of a multidimensional random variable. The method uses a two-dimensional non-parametric algorithm for pattern recognition of the kernel type, corresponding to the criterion of maximum likelihood. In contrast to the traditional method based on the application of the Pearson criterion, the proposed approach avoids the problem of decomposing the range of values of random variables into multidimensional intervals. The results of computational experiments performed according to the method of forming sets of independent random variables are presented. Using the information obtained, an information graph is constructed, the vertices of which correspond to the components of a multidimensional random variable, and the edges determine their independence. Then the vertices of the complete subgraphs correspond to groups of independent components of a random variable. The obtained results form the basis for the synthesis of a multi-level nonparametric large volume data processing system, each level of which corresponds to a specific set of independent random variables.

Parallel Coordinates Visualization in the ELK Stack

Modern large-scale distributed computing systems, processing large volumes of data, require mature monitoring systems able to control and track in re-sources, networks, computing tasks, queues and other components. In recent years, the ELK stack has become very popular for the monitoring of computing environment, largely due to the efficiency and flexibility of the Elastic Search storage and wide variety of Kibana visualization tools. The analysis of computing infrastructure metadata often requires the visual exploration of multiple parameters simultaneously on one graphical image. Stacked bar charts, heat maps, radar charts are widely used for the multivariate visual data analysis, but these methods have limitations on the number of parameters. In this research the authors propose to enhance the capacity of Kibana, adding Parallel Coordinates diagram - one of the most powerful method for visual interactive analysis of high-dimensional data. It allows to compare many variables together and observe correlations between them. This work describes the development process of Parallel Coordinates as a Kibana plugin, and demonstrates an example of visual data analysis based on the Nginx logs metadata.

System for Automated Infrared Image Processing Based on Neural Network Technologies

The article describes the main problems that non-destructive control experts encounter in processing and analyzing infrared (IR) images obtained using thermographic cameras: noisiness, information redundancy, excessive data volume, etc. A gradual process of automated digital processing and analysis of images on the basis of a software system is considered, which allows not only overcoming the drawbacks of the raw data described above, but also drawing a conclusion about the state of the object based on the obtained image independently. The ability to make expert decision is provided by a neural network component in a system on the basis of Kohonen network used for cluster image analysis. There are the examples of system operation applied to IR images of the lining of solid domestic waste (SDW) incinerator. The use of such systems allows reducing workload of an expert in processing large volumes of data and reducing the probability of errors due to such effect as vision "blurring".

Weighted Complete Graphs for Condensing Data

In many real-world problems (such as industrial applications, chemistry models, social network analysis, among others), their solution can be obtained by transforming the problem in terms of vertices and edges, that is to say, using graph theory. Data Science applications are characterized by processing large volumes of data, in some cases, the data size can be higher than the resources for their processing, situation that makes prohibitive to use the traditional methods. In this way, to develop solutions based on graphs for condensing data can be a good strategy for handling big datasets. In this paper we include two methods for condensing data based on graphs, the two proposals consider a weighted complete graph by acquiring an induced subgraph or a minimum spanning tree from the whole datasets. We conducted some experiments in order to validate our proposals, using 24 benchmark real-datasets for training the 1NN, C4.5, and SVM classifiers. The results prove that our methods condensed the datasets without reducing the performance of the classifier, in terms of geometric means and the Wilcoxon's test.

A CUDA-Based Parallel Geographically Weighted Regression for Large-Scale Geographic Data

Geographically weighted regression (GWR) introduces the distance weighted kernel function to examine the non-stationarity of geographical phenomena and improve the performance of global regression. However, GWR calibration becomes critical when using a serial computing mode to process large volumes of data. To address this problem, an improved approach based on the compute unified device architecture (CUDA) parallel architecture fast-parallel-GWR (FPGWR) is proposed in this paper to efficiently handle the computational demands of performing GWR over millions of data points. FPGWR is capable of decomposing the serial process into parallel atomic modules and optimizing the memory usage. To verify the computing capability of FPGWR, we designed simulation datasets and performed corresponding testing experiments. We also compared the performance of FPGWR and other GWR software packages using open datasets. The results show that the runtime of FPGWR is negatively correlated with the CUDA core number, and the calculation efficiency of FPGWR achieves a rate of thousands or even tens of thousands times faster than the traditional GWR algorithms. FPGWR provides an effective tool for exploring spatial heterogeneity for large-scale geographic data (geodata).

Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme

The analysis of massive databases is a key issue for most applications today and the use of parallel computing techniques is one of the suitable approaches for that. Apache Spark is a widely employed tool within this context, aiming at processing large amounts of data in a distributed way. For the Statistics community, R is one of the preferred tools. Despite its growth in the last years, it still has limitations for processing large volumes of data in single local machines. In general, the data analysis community has difficulty to handle a massive amount of data on local machines, often requiring high-performance computing servers. One way to perform statistical analyzes over massive databases is combining both tools (Spark and R) via the sparklyr package, which allows for an R application to use Spark. This paper presents an analysis of Brazilian public data from the Bolsa Família Programme (BFP—conditional cash transfer), comprising a large data set with 1.26 billion observations. Our goal was to understand how this social program acts in different cities, as well as to identify potentially important variables reflecting its utilization rate. Statistical modeling was performed using random forest to predict the utilization rated of BFP. Variable selection was performed through a recent method based on the importance and interpretation of variables in the random forest model. Among the 89 variables initially considered, the final model presented a high predictive performance capacity with 17 selected variables, as well as indicated high importance of some variables for the observed utilization rate in income, education, job informality, and inactive youth, namely: family income, education, occupation and density of people in the homes. In this work, using a local machine, we highlighted the potential of aggregating Spark and R for analysis of a large database of 111.6 GB. This can serve as proof of concept or reference for other similar works within the Statistics community, as well as our case study can provide important evidence for further analysis of this important social support programme.

Predicting clinically significant motor function improvement after contemporary task-oriented interventions using machine learning approaches

BackgroundAccurate prediction of motor recovery after stroke is critical for treatment decisions and planning. Machine learning has been proposed to be a promising technique for outcome prediction because of its high accuracy and ability to process large volumes of data. It has been used to predict acute stroke recovery; however, whether machine learning would be effective for predicting rehabilitation outcomes in chronic stroke patients for common contemporary task-oriented interventions remains largely unexplored. This study aimed to determine the accuracy and performance of machine learning to predict clinically significant motor function improvements after contemporary task-oriented intervention in chronic stroke patients and identify important predictors for building machine learning prediction models.MethodsThis study was a secondary analysis of data using two common machine learning approaches, which were the k-nearest neighbor (KNN) and artificial neural network (ANN). Chronic stroke patients (N = 239) that received 30 h of task-oriented training including the constraint-induced movement therapy, bilateral arm training, robot-assisted therapy and mirror therapy were included. The Fugl-Meyer assessment scale (FMA) was the main outcome. Potential predictors include age, gender, side of lesion, time since stroke, baseline functional status, motor function and quality of life. We divided the data set into a training set and a test set and used the cross-validation procedure to construct machine learning models based on the training set. After the models were built, we used the test data set to evaluate the accuracy and prediction performance of the models.ResultsThree important predictors were identified, which were time since stroke, baseline functional independence measure (FIM) and baseline FMA scores. Models for predicting motor function improvements were accurate. The prediction accuracy of the KNN model was 85.42% and area under the receiver operating characteristic curve (AUC-ROC) was 0.89. The prediction accuracy of the ANN model was 81.25% and the AUC-ROC was 0.77.ConclusionsIncorporating machine learning into clinical outcome prediction using three key predictors including time since stroke, baseline functional and motor ability may help clinicians/therapists to identify patients that are most likely to benefit from contemporary task-oriented interventions. The KNN and ANN models may be potentially useful for predicting clinically significant motor recovery in chronic stroke.

AI in the Intensive Care Unit: Up-to-Date Review.

AI is the latest technologic trend that likely will have a huge impact in medicine. AI's potential lies in its ability to process large volumes of data and perform complex pattern analyses. The ICU is an area of medicine that is particularly conducive to AI applications. Much AI ICU research currently is focused on improving high volumes of data on high-risk patients and making clinical workflow more efficient. Emerging topics of AI medicine in the ICU include AI sensors, sepsis prediction, AI in the NICU or SICU, and the legal role of AI in medicine. This review will cover the current applications of AI medicine in the ICU, potential pitfalls, and other AI medicine-related topics relevant for the ICU.

Traveltime-table compression using artificial neural networks for Kirchhoff-migration processing of microseismic data

Massive computation of seismic traveltimes is widely used in seismic processing, for example, for the Kirchhoff migration of seismic and microseismic data. Implementation of the Kirchhoff migration operators uses large precomputed traveltime tables (for all sources, receivers, and densely sampled imaging points). We have tested the idea of using artificial neural networks for approximating these traveltime tables. The neural network has to be trained for each velocity model, but then the whole traveltime table can be compressed by several orders of magnitude (up to six orders) to the size of less than 1 MB. This makes it convenient to store, share, and use such approximations for processing large data volumes. We evaluate some aspects of choosing neural-network architecture, training procedure, and optimal hyperparameters. On synthetic tests, we find a reasonably accurate approximation of traveltimes by neural networks for various velocity models. A final synthetic test shows that using the neural-network traveltime approximation results in good accuracy of microseismic event localization (within the grid step) in the 3D case.

Analysis and mathematical modeling of big data processing

Big data processing is an urgent and unresolved challenge that originates from the intensive development of information technology. The recent techniques lose their effectiveness rapidly as the volumes of data increase. In this article, we will put down our vision of the basic approaches and models related to problem solving, based on processing large data volumes. This article introduces a two-stage decomposition of a problem, related to assessing management options. The first stage of our original approach implies a semantic analysis of textual information; the second stage is built around finding association rules in a database, processing them via mathematical statistics methods, and converting data and objectives to a vector. We suggest processing the collected news events by a semantic model, which describes their key features and interconnections between them in a specified subject area. The classification-based association rules allow assessing the likelihood of a particular event using a set chain of events. This approach can be applied through the analysis of online news in a specified market segment.

Stock Market Prediction

The Stock Market is a challenging forum for investment and requires immense brainstorming before one shall put their hard earned money to work. This project aims at processing large volumes of data and running comprehensive regression algorithms on the dataset; that will predict the future value of a stock using the regression model with the highest accuracy. The purpose of this paper is to analyze the shortcomings of the current system and building a time-series model that would mitigate most of them by implementing more efficient algorithms. Using this model, anyone can monitor the preferred stock that they want to invest in; and maximize profit by purchasing volume at the lowest price and liquidating the stock when it’s at its highest.