Real-time Big Data Processing Research Articles

The Analysis of Resilientnet-Realtime Disaster Response System

Responding to India's urgent need for effective disaster management, proposed framework ResilientNet, an innovative system leveraging real-time big data processing and advanced AI technologies. ResilientNet gathers diverse multimedia content from a wide range of social media services, including Twitter, Instagram, Facebook, etc., and utilises the GEMINI API, enabling comprehensive analysis and verification. Data is stored in the NEO4J database and visually represented on a user-friendly website dashboard for easy accessibility and insights. This research explores the efficacy of crowdsourced fact- checking, contributing to a novel disaster-focused tweet verification system. ResilientNet's amalgamation of crowdsourcing and AI creates a comprehensive graph of critical metrics and trends, enabling authorities to counter misinformation and direct disaster response efforts efficiently.

Neural network models of apple fruit identification in tree crowns: comparative analysis

The article presents the results of an analysis conducted from 2022 to 2023 to assess the quality of modern neural network models of apple fruit identification in tree crowns shown in images. In order to conduct the studies on identifying the best detector, the following neural networks were used: SSD (Single Shot MultiBox Detector), YOLOv4 (You Only Look Once, Version 4), YOLOv5, YOLOv7, and YOLOv8. The performance of the considered models of apple fruit identification was assessed using such binary classification metrics as precision, recall, accuracy, F-score, and AUC-ROCTotal (area under the curve). To assess the accuracy in predicting apple fruit identification, the mean absolute percentage error (MAPE) of the analyzed neural network models was calculated. The neural network performance analysis used 300 photographs taken at an apple garden. The conducted studies revealed that the SSD model provides lower speed and accuracy, as well as having high requirements for computing resources, which may limit its use in lower performance devices. The YOLOv4 model surpasses the YOLOv5 model in terms of accuracy by 10.2 %, yet the processing speed of the YOLOv5 model is over twice that of the YOLOv4 model. This fact makes the YOLOv5 model preferable for tasks related to real-time big data processing. The YOLOv8 model is superior to the YOLOv7 model in terms of speed (by 37.3 %); however, the accuracy of the YOLOv7 model is 9.4 % higher. The highest area under the Precision-Recall curve amounts to 0.94 when using the YOLOv7 model. This fact suggests a high probability that the classifier can accurately distinguish between the positive and negative values of the apple fruit class. MAPE calculation for the analyzed neural network models showed that the lowest error in apple fruit identification amounted to 5.64 % for the YOLOv7 model as compared to the true value determined using the visual method. The performance analysis of modern neural network models shows that the YOLO family of neural networks provides high speed and accuracy of object detection, which allows them to operate in real time. The use of transfer learning (tuning of only the last layers to solve highly specialized problems) to adjust the performance of models for different apple fruit varieties can further improve the accuracy of apple fruit identification.

Supernova Light Curves Approximation based on Neural Network Models

Photometric data-driven classification of supernovae becomes a challenge due to the appearance of real-time processing of big data in astronomy. Recent studies have demonstrated the superior quality of solutions based on various machine learning models. These models learn to classify supernova types using their light curves as inputs. Preprocessing these curves is a crucial step that significantly affects the final quality. In this talk, we study the application of multilayer perceptron (MLP), bayesian neural network (BNN), and normalizing flows (NF) to approximate observations for a single light curve. We use these approximations as inputs for supernovae classification models and demonstrate that the proposed methods outperform the state-of-the-art based on Gaussian processes applying to the Zwicky Transient Facility Bright Transient Survey light curves. MLP demonstrates similar quality as Gaussian processes and speed increase. Normalizing Flows exceeds Gaussian processes in terms of approximation quality as well.

A Real-Time Deep Learning-based Smart Surveillance Using Fog Computing: A Complete Architecture

Fog computing offers low-latency and real-time big-data processing capabilities closer to the network edge. This particular benefit addresses the main bottleneck in a centralized cloud framework, which is, it cannot process latency-sensitive large video frames generated from the Internet of Things-based video surveillance cameras in real-time. Besides, the recent advancements in the computer vision field offer many state-of-the-art image processing capabilities that can be utilized for real-time surveillance data processing. Deploying those processing powers at several fog computing layers can bring novel solutions for computer vision-based real-time security solutions. This paper proposes a deep learning-based framework for smart video surveillance that can process the real-time frames on two consecutive fog layers, one for action recognition and the other for criminal threat-based response generation. The proposed architecture consists of three major modules. The first module is responsible for capturing surveillance videos by deploying RaspberryPi cameras in a distributed network. The second module is responsible for action recognition using a deep learning-based model installed inside NVIDIA Jetson Nano-devices placed on two fog layers. Finally, the security response is generated and broadcast to the law-enforcement agency. To evaluate the proposed model, experiments on semantic segmentation-based scene object recognition were run. The experimental results came up with a suitable recognition model that can be deployed in the fog layers of our proposed framework.

Real-Time Big Data Processing and Analytics: Concepts, Technologies, and Domains

In the digital era, data is one of the most important assets since it conceals valuable information. Developers of data-intensive systems have new challenges at each level of streaming, storing, and processing large quantities of data in a variety of forms and speeds. Obtaining useful information at the proper time and place is also crucial. Since the value of information is inversely proportional to time, real-time data processing and analytics are receiving more attention. Due to the importance of real-time data processing and analytics, this study focuses on real-time data processing concepts and terminology, popular technologies used in real-time data processing and analytics, popular NoSQL storage technologies used in real-time data processing, and real-time data processing application areas. The purpose of this paper is to provide researchers of real-time analysis and developers of data-intensive systems with a comparative perspective on real-time data processing by highlighting the key characteristics of real-time data processing technologies, NoSQL storage technologies, their application domains, and selected examples from previous studies.

GraphA: An efficient ReRAM-based architecture to accelerate large scale graph processing

Graph analytics is the basis for many modern applications, e.g., machine learning and streaming data problems. With an unprecedented increase in data size of many emerging domains such as the social networks, which generate a lot of images and documents, real-time big data processing is crucial. Graph processing on traditional computer architectures faces irregular memory accesses that lead to significant data movements and waste a large amount of energy and time. ReRAM-based Processing-in-memory (PIM) is a novel technology that addresses the memory wall problem. Additionally, it provides a high parallelism level and a significant reduction in energy consumption with negligible leakage power. In this paper, we propose a ReRAM-based PIM architecture, named GraphA, which includes a novel reordering algorithm and mapping data to ReRAM Graph Engines (RGE) that cause RGEs to be used with high utilization. Furthermore, we present a compressed format, a memory layout, and proper graph partitioning for graph traversal to eliminate extra communication and useless computation. Moreover, we investigate the computation patterns of graph processing to find a suitable preprocessing model for the proposed GraphA architecture based on reorganizing classified supernode graphs and offering a runtime execution that fits it. Evaluations of GraphA on various real-world graphs show an average performance enhancement and energy saving of 5.3x and 6.0x, respectively, compared with the state-of-the-art GraphR architecture.

Real-Time Based Big Data and E-Learning: A Survey and Open Research Issues

Big data is considered a remarkable aspect of development communication and information systems. In terms of delivery, retrieval, and storage, a vast volume of complex data exceeds the capability of conventional software and device capabilities. As a result, advanced alternative solutions that allow their control flow is becoming more prevalent. One of the most difficult fields of information management and technology is the real-time management, analysis, and processing of big data. These issues can be seen in a large amount of everyday produced data in a variety of places, including online social networks and the method of logging cell phone data. This survey looks at several studies that use a range of methods to handle, interpret, and process big data in real-time. Hence, the objective of this survey is to provide a comprehensive overview of the integration between real-time and big data fields of study with the field of E-learning. Finally, this survey also presents the colorful aspects of big data and their relationship to E-learning domains such that e-learning platforms, big data frameworks, and datasets used.

Smart Mobility and Aspects of Vehicle-to-Infrastructure: A Data Viewpoint

The aim of this article is to describe estimates of data difficulty and aspects of the data viewpoint within Vehicle-to-Infrastructure (V2I) communication in the Smart Mobility concept. The historical development of the database system’s architecture, that stores and processes a larger amount of data, is currently sufficient and effective for the needs of today’s society. The goal of vehicle manufacturers is the continual increase in driving comfort and the use of multiple sensors to sense the vehicle’s surroundings, as well as to help the driver in critical situations avoid danger. The increasing number of sensors is directly related to the amount of data generated by the vehicle. In the automotive industry, it is crucial that autonomous vehicles can process data in real time or can locate itself in precise accuracy, for the decision-making process. To meet these requirements, we will describe HD maps as a key segment of autonomous control. It alerts the reader to the need to address the issue of real-time Big Data processing, which represents an important role in the concept of Smart Mobility.

SPOT: Testing Stream Processing Programs with Symbolic Execution and Stream Synthesizing

Adoption of distributed stream processing (DSP) systems such as Apache Flink in real-time big data processing is increasing. However, DSP programs are prone to be buggy, especially when one programmer neglects some DSP features (e.g., source data reordering), which motivates development of approaches for testing and verification. In this paper, we focus on the test data generation problem for DSP programs. Currently, there is a lack of an approach that generates test data for DSP programs with both high path coverage and covering different stream reordering situations. We present a novel solution, SPOT (i.e., Stream Processing Program Test), to achieve these two goals simultaneously. At first, SPOT generates a set of individual test data representing each path of one DSP program through symbolic execution. Then, SPOT composes these independent data into various time series data (a.k.a, stream) in diverse reordering. Finally, we can perform a test by feeding the DSP program with these streams continuously. To automatically support symbolic analysis, we also developed JPF-Flink, a JPF (i.e., Java Pathfinder) extension to coordinate the execution of Flink programs. We present four case studies to illustrate that: (1) SPOT can support symbolic analysis for the commonly used DSP operators; (2) test data generated by SPOT can more efficiently achieve high JDU (i.e., Joint Dataflow and UDF) path coverage than two recent DSP testing approaches; (3) test data generated by SPOT can more easily trigger software failure when comparing with those two DSP testing approaches; and (4) the data randomly generated by those two test techniques are highly skewed in terms of stream reordering, which is measured by the entropy metric. In comparison, it is even for test data from SPOT.

Application Research of Big Data Real-time Processing Technology in Smart Grid

Big data is valuable, and big data processing technology can efficiently process massive data and get the value in big data, so the research and application of big data technology become popular nowadays. Smart grid is one of the most potential field for application of big data technology, applying big data technology to smart grid can improve the reliability and economy of smart grid. The operating conditions of power system changes rapidly, therefore, real-time processing technology became the focus of application field. This paper reviewed the data sources, data characteristics and data application scenarios of big data in smart grid, analyzed data processing modes of big data processing technology: batch processing and stream processing, and compared the realization methods of real-time big data processing, summarized the application of big data real-time processing technology in smart grid, and the challenges in the application, and finally, prospected the application of real-time big data processing technology in smart grid.

Designing real-time research data portal of JTI POLINEMA

Research culture in Department of Information Technology Polinema is always developed continuously every time. In addition, many research titles that are critical of environmental problems are beginning to be seen. The Information Technology Department has an outstanding focus on issues, namely smart systems which include smart education, smart living, smart healthy, smart city and smart tourism. Research resources in the Information Technology Department Polinema are competing to realize that research focus. For this reason, efforts are needed to increase the availability of facilities that are not only material. The fast response can be solved by providing Real-Time Research Data Portal. The contribution of this paper is to design of middle ware on web portals that have high scalability capabilities for providing large data facilities and fast access. Real-time big data processing is the path that will be described in the architecture that will be created and adapted to the needs of the Department of Information Technology Polinema. The results of the latency evaluation test indicated that RabbitMQ allowed us to enforce concurrency, thereby reducing overall RTC by up to 50% compared to traditional HTTP.

Adaptive Trust Management and Data Process Time Optimization for Real-Time Spark Big Data Systems

Applications supporting businesses, smart systems, social networks, and advanced video applications such as eXtended Reality (XR) require large amounts of data processing to be provided in real-time. Therefore, the processing speed of big data systems is more important than ever. On the other hand, protecting a big data system is not easy, as various types of nodes and clusters are supported by various wired and wireless networks. Commonly security procedures slow down the response time of big data networks, and therefore, enhanced security and performance speed techniques need to be co-designed into the system. In this paper, a trusted streaming adaptive function (TSAF) is proposed that uses a trust management scheme to identify malicious nodes in Spark big data systems, exclude them from job/task processing, and calculate the number of nodes that can satisfy the process’s object completion time. The TSAF scheme shows an improved processing performance when there are attacks on the big data system compared to other existing real-time big data processing schemes. For the case of no security attack, the results show that the processing time of TSAF is faster by about 1 ~ 2% compared to the existing big data processing schemes when the process completion object time is set to 0.5 s. Even when the ratio of malicious nodes performing security attacks on worker nodes reaches 0.5, the results show that TSAF can satisfy over 75% of the tasks within the object time, which is significantly higher compared to the existing big data processing schemes.

The real-time big data processing method based on LSTM or GRU for the smart job shop production process

With the wide application of intelligent sensors and internet of things (IoT) in the smart job shop, a large number of real-time production data is collected. Accurate analysis of the collected data can help producers to make effective decisions. Compared with the traditional data processing methods, artificial intelligence, as the main big data analysis method, is more and more applied to the manufacturing industry. However, the ability of different AI models to process real-time data of smart job shop production is also different. Based on this, a real-time big data processing method for the job shop production process based on Long Short-Term Memory (LSTM) and Gate Recurrent Unit (GRU) is proposed. This method uses the historical production data extracted by the IoT job shop as the original data set, and after data preprocessing, uses the LSTM and GRU model to train and predict the real-time data of the job shop. Through the description and implementation of the model, it is compared with KNN, DT and traditional neural network model. The results show that in the real-time big data processing of production process, the performance of the LSTM and GRU models is superior to the traditional neural network, K nearest neighbor (KNN), decision tree (DT). When the performance is similar to LSTM, the training time of GRU is much lower than LSTM model.

Conceptual Model for Connected Vehicles Safety and Security using Big Data Analytics

The capability of Connected Vehicles (CVs) connecting to a nearby vehicle, surrounding infrastructure and cyberspace presents a high risk in the aspect of safety and security of the CV and others. Data volume generated from the sensors and infrastructure in CVs environment are enormous. Thus, CVs implementations require a real-time big data processing and analytics to detect any anomaly in the CVs’s environment which are physical layer, network layer and application layer. CVs are exposed to various vulnerabilities associated with exploitations or malfunctions of the components in each layer that could result in various safety and security event such as congestion and collision. The safety and security risks added an extra layer of required protection for the CVs implementation that need to be studied and refined. To address this gap, this research aims to determine the basic components of safety and security for CVs implementation and propose a conceptual model for safety and security in CVs by applying the machine learning and deep learning techniques. The proposed model is highly correlated to safety and security and could be applied in congestion and collision prediction.

Real-time big data processing for instantaneous marketing decisions: A problematization approach

The collection of big data from different sources such as the internet of things, social media and search engines has created significant opportunities for business-to-business (B2B) industrial marketing organizations to take an analytical view in developing programmatic marketing approaches for online display advertising. Cleansing, processing and analyzing of such large datasets create challenges for marketing organizations — particularly for real-time decision making and comparative implications. Importantly, there is limited research for such interplays. By utilizing a problematization approach, this paper contributes through the exploration of links between big data, programmatic marketing and real-time processing and relevant decision making for B2B industrial marketing organizations that depend on big data-driven marketing or big data-savvy managers. This exploration subsequently encompasses appropriate big data sources and effective batch and real-time processing linked with structured and unstructured datasets that influence relative processing techniques. Consequently, along with directions for future research, the paper develops interdisciplinary dialogues that overlay computer-engineering frameworks such as Apache Storm and Hadoop within B2B marketing viewpoints and their implications for contemporary marketing practices.

IoT Analysis and Processing System using Sensed Data for Laboratory Safety Management

This study proposes and implements a real-time message analysis processing system using sensed data for safety management of a laboratory composed of Internet of Things (IoT) sensor devices. Existing IoT systems have difficulty storing data in a relational database system due to the different data formats between sensor devices, and they also have preprocessing overhead due to nonstandard communications protocols. Furthermore, they are centralized systems with unstable service continuity, and it is difficult to add new functions or extend the system because of the complex connections among sensor devices and services. Studies on real-time analysis and processing of large volumes of sensor data are also insufficient. To address these problems, the proposed system uses IoT sensor devices, based on standard communications protocols, and message queue brokers based on standard messaging protocols. The message queue brokers can facilitate adding sensor devices or services by delivering the sensor data from the gateway and central server to the services. Services can continue, even in the event of network trouble, because the services that had been provided from the central server are now provided through smart gateways. The central server can provide risk analysis and prevention services for laboratory safety management through Apache Spark, a real-time big data analysis and processing platform. Through the experiments in this study, the proposed real-time message analysis processing system for laboratory safety management was implemented. The message queue brokers between the standard Message Queuing Telemetry Transport (MQTT) protocol and the smart gateway receiving the data were built into sensor devices equipped with network and sensor modules. Exchange routing keys are also specified by topic and service for each sensor device in line with the laboratory safety management environment. Data transmission processing is handled by a federated connection of the gateway brokers and the central server brokers.

실시간 철도안전관제 플랫폼에서 빅데이터의 실시간 처리를 위한 고속 필터 및 인터페이스 방안 연구

실시간 철도안전관제 플랫폼에서 빅데이터의 실시간 처리를 위한 고속 필터 및 인터페이스 방안 연구

Energy-Efficient Task Scheduling for CPU-Intensive Streaming Jobs on Hadoop

Hadoop, especially Hadoop 2.0, has been a dominant framework for real-time big data processing. However, Hadoop is not optimized for energy efficiency. Aiming to solve this problem, in this paper, we propose a new framework to improve the energy efficiency of Hadoop 2.0. We focus on the resource manager in Hadoop 2.0, namely YARN, and propose energy-efficient task scheduling mechanisms on YARN. Particularly, we focus on CPU-intensive streaming jobs and classify streaming jobs into two types, namely batch streaming jobs (i.e., a set of jobs are submitted simultaneously) and online streaming jobs (i.e., jobs are continuously submitted one by one). We devise different energy-efficient task scheduling algorithms for each kind of streaming jobs. Specially, we first propose to abstractly model performance and energy consumption by considering the characteristics of tasks as well as the computational resources in YARN. Based on this model, we study the energy efficiency of streaming tasks which consist of the performance model and energy consumption model of task. We propose two key principles for improving energy efficiency: 1) CPU usage aware task allocation, partitions tasks to NMs based on the task characteristic in term of CPU usage; and 2) resource efficient task allocation, reduce idle resource. Then, we propose a D-based binning algorithm for the batch task scheduling and K-based binning algorithm for the online task scheduling that can adapt to continuously arriving tasks. We conduct extensive experiments on a real Hadoop 2.0 cluster and use two kinds of workloads to evaluate the performance and energy efficiency of our proposal. Compared with Storm (the streaming data processing tool in Hadoop 2.0) and other approaches including TAPA and DVFS-MR, our proposal is more energy efficient. The batch task scheduling algorithm reduces up to 10 percent of energy consumption and keeps comparable performance. In addition, the online task scheduling algorithm reduces up to 7 percent over the existing algorithms.

Real-Time Big Data Processing and Wearable Internet of Medical Things Sensor Devices for Health Monitoring

Real-Time Big Data Processing and Wearable Internet of Medical Things Sensor Devices for Health Monitoring

A Distributed Data Processing Platform over Meteorological Big Data Using MapReduce

In the era of big data, the data of the meteorological departments grows explosively, which puts higher requirements on the real-time processing of meteorological big data. Besides, the efficient storage for the massive meteorological data has also attracted much attention from the meteorological departments. Therefore, in response to the urgent requirements of meteorological big data in processing and storage, a distributed data processing platform over meteorological big data using MapReduce is designed. Technically, the platform develops corresponding real-time strategies according to various data properties, and obtains meteorological big data in real-time from multiple channels. Based on the MapReduce real-time processing, we realise the distributed storage to store the meteorological big data in the platform. Overall, our platform improves the real-time, reliability, availability and the access efficiency of meteorological big data which is easy to expand and also has a good reference value for big data processing in other similar industries.