Real-time Big Data Research Articles

The Analysis of Resilientnet-Realtime Disaster Response System

Responding to India's urgent need for effective disaster management, proposed framework ResilientNet, an innovative system leveraging real-time big data processing and advanced AI technologies. ResilientNet gathers diverse multimedia content from a wide range of social media services, including Twitter, Instagram, Facebook, etc., and utilises the GEMINI API, enabling comprehensive analysis and verification. Data is stored in the NEO4J database and visually represented on a user-friendly website dashboard for easy accessibility and insights. This research explores the efficacy of crowdsourced fact- checking, contributing to a novel disaster-focused tweet verification system. ResilientNet's amalgamation of crowdsourcing and AI creates a comprehensive graph of critical metrics and trends, enabling authorities to counter misinformation and direct disaster response efforts efficiently.

Study of Access Control Techniques on the Blockchain-enabled Secure Data Sharing Scheme in Edge Computing

Background: The pervasive adoption of edge computing is reshaping real-time big data analysis, smart city management, intelligent transportation, and various other domains. Its appeal lies in its distributed nature, decentralization, low latency, mobile support, and spatial awareness. However, this ubiquity exposes data to security threats, jeopardizing privacy and integrity. Consequently, access control assumes paramount importance in securing data sharing within edge computing and blockchain technologies. Methods: This research addresses this critical issue by conducting a comprehensive study on access control techniques within the context of edge computing and blockchain for secure data sharing. Our methodology commences with an exhaustive review of relevant articles, aiming to identify and expound upon gaps in existing research. Subsequently, we perform a meticulous analysis of access control methods, mechanisms, and performance metrics, seeking to establish a holistic understanding of the landscape. Results: The culmination of this research effort is a multifaceted contribution. We distill insights from a diverse range of access control schemes, shedding light on their nuances and effectiveness. Our analysis extends to evaluating the performance metrics vital for ensuring robust access control. Through this research, we also pinpoint critical research gaps within traditional access control methods, creating a foundation for innovative approaches to address the evolving challenges within edge computing and blockchain environments. Conclusion: In conclusion, this research venture paves the way for secure data sharing in edge computing and blockchain by offering a thorough examination of access control. The findings from this study are anticipated to guide future developments in access control techniques and facilitate the evolution of secure, privacy-conscious, and efficient data sharing practices in the dynamic landscape of digital technology.

RNEP: Random Node Entropy Pairing for Efficient Decentralized Training with Non-IID Local Data

The proliferation of edge devices and advancements in Internet of Things technology have created a vast array of distributed data sources, necessitating machine learning models that can operate closer to the point of data generation. Traditional centralized machine learning approaches often struggle with real-time big data applications, such as climate prediction and traffic simulation, owing to high communication costs. In this study, we introduce random node entropy pairing, a novel distributed learning method for artificial neural networks tailored to distributed computing environments. This method reduces communication overhead and addresses data imbalance by enabling nodes to exchange only the weights of their local models with randomly selected peers during each communication round, rather than sharing the entire dataset. Our findings indicate that this approach significantly reduces communication costs while maintaining accuracy, even when learning from non-IID local data. Furthermore, we explored additional learning strategies that enhance system accuracy by leveraging the characteristics of this method. The results demonstrate that random node entropy pairing is an effective and efficient solution for distributed learning in environments where communication costs and data distribution present significant challenges.

Real-time Analysis of Big Data Streams and Its Application in Social Networks

In the digital era, big data has penetrated into all aspects of our lives, from business operations, social networks to medical health, urban planning, its impact is everywhere. Especially in the field of social networks, big data is not only huge in scale, but also has high realtime requirements. How to effectively conduct real-time big data analysis and extract valuable information has become a hot topic of current research. Based on this, this study is based on the real-time analysis technology of big data in social networks, and discusses it, so as to provide good theoretical support and practical experience for the future development of big data analysis and social networks.

Merger of internet of things and machine learning: The internet of everything sector projects, benefits, and future roles

The convergence of the Internet of Things and Machine Learning shapes key developments in information technologies, such as the Internet of Everything environment. The Internet of Things' Machine-to-Machine connectivity advances, real-time Big Data storage, and data-driven Machine Learning synergize the scope of Internet of Everything innovations. As the Internet of Things and Machine Learning merger continues, the Internet of Everything reshapes the world sector map by providing unprecedented possibilities and efficiencies. The propelled Internet of Everything is ushering in a new era of transformative projects in various sectors such as Healthcare and Life Science, Manufacturing, Agriculture, Energy, Transportation, and Construction. The high-tech transformative projects offer many applications in process optimization, cost reduction, increased productivity, and efficiency. Our work examines the prominent joint Internet of Things and Machine Learning initiatives, projects and works to encourage new initiatives and multidisciplinary work worldwide. We also discuss the benefits of transformative projects in different sectors in detail and put forward a forecast for their future roles. Our study is supported by solid academic literature and project examples.

Comparative Analysis of Hadoop and Spark Performance for Real-time Big Data Smart Platforms Utilizing IoT Technology in Electrical Facilities

Comparative Analysis of Hadoop and Spark Performance for Real-time Big Data Smart Platforms Utilizing IoT Technology in Electrical Facilities

Improving Shopping Mall Revenue by Real-Time Customized Digital Coupon Issuance

With the development of big data and deep learning technology, big data and deep learning technology have also been applied to the marketing field, which was a part of business administration. Customer churn management is one of the most important areas of marketing. In this paper, we proposed a method to prevent customer churn and increase purchase conversion rate by issuing customized discount coupons to customers with high churn rate based on big data in real time. After segmenting customer segments with two-dimensional segment analysis, a real-time churn rate estimation model based on clickstream data was generated for each segment. After that, we issued customized coupons to our customers. Finally, we tested the conversion rate and sales growth. A two-dimensional cluster analysis-based churn rate estimation combined with a recommendation system was found to be significantly more useful than the respective simple models. Using this proposed model, it is possible to increase sales by automatically estimating the customer’s churn probability and shopping propensity without the burden of marketing costs in the online shopping mall. Keyword: Customer churn prediction, deep learning, digital marketing, ecommerce, recommendation system.

REVIEW OF TRENDS IN APPLICATION OF VIRTUAL REALITY IN ENGINEERING

It is known that the growth rate of the mechanical engineering industry in the world is about 3% every year, and the growth leaders in the industry are motor vehicles, electronic components, communication equipment, computers and peripheral equipment, and aerial vehicles. The Industry 4.0 program is the next stage of the manufacturing enterprises digital transformation, the purpose of which is to increase the competitiveness of the Ukrainian economy machine-building sector. Industry 4.0 aims at the accelerated implementation of digital technologies, including the industrial Internet of Things, real-time big data analysis, artificial intelligence, the new generation of robots, virtual and augmented reality, etc. It is expected that the synergy of IT and production technologies will lead to the change in business models and a significant acceleration of innovative development at all stages from the development to the introduction into production of new products. On the basis of the National Technical University "Dniprovska Polytechnic", the direction of training students under the educational program "Technologies of virtual and augmented reality in mechanical engineering" specialty 133 Industrial mechanical engineering has been launched. The training of specialists with methodological and practical skills in the field of modern technologies of virtual and augmented reality is extremely important for the implementation of the Industry 4.0 strategy.

Integrating Azure Services for Real Time Data Analytics and Big Data Processing

Integrating Azure services for real-time data analytics and big data processing is a transformative approach that leverages the power of cloud computing to handle vast amounts of data at scale. Azure offers a wide array of services such as Azure Synapse Analytics, Azure Data Lake, Azure Stream Analytics, and Azure Databricks, which together enable businesses to extract valuable insights from both structured and unstructured data in real time. These services provide seamless integration, allowing for real-time data ingestion, transformation, and analysis, facilitating quicker decision-making processes. Azure’s capabilities in big data processing allow for scalable and efficient storage of data, ensuring optimal performance even as data volumes grow. By utilizing tools like Azure Data Factory and Azure HDInsight, organizations can automate complex workflows and apply machine learning algorithms for predictive analytics, driving innovation and competitive advantage. The integration of Azure's real-time analytics services also supports the use of advanced technologies like artificial intelligence (AI) and Internet of Things (IoT), enabling companies to monitor live streams of data from various devices and platforms. This infrastructure not only enhances operational efficiency but also ensures a more proactive approach to problem-solving by identifying trends and anomalies as they occur. In conclusion, Azure’s comprehensive suite of services offers a robust platform for managing and analysing big data in real time, empowering organizations to leverage data for better decision-making and improved business outcomes.

Analysis of carbon emission reduction contribution during usage phase of new energy vehicles based on big data: A case study of Tianjin

Countries around the world are paying increasing attention to greenhouse gas emissions reduction. Under the guidance of “dual carbon” strategy in China, the development of the new energy vehicle industry as an important means of carbon emission reduction in the transportation sector has become a trend. At the same time, the Tianjin New Energy Vehicle Monitoring Platform has recorded real-time big data on the operation of new energy vehicles since 2017, covering vehicle ownership and mileage data, which can support the calculation of carbon emission reduction contributions during usage stage of new energy vehicles. Based on the big data of the platform, this article calculates the carbon emission reduction contribution of new energy vehicles in Tianjin during usage stage, and analyzes the carbon emission reduction contributions under different technology types, vehicle types, vehicle classes, and usage scenarios. Moreover, an analysis was conducted on the top ten enterprises in Tianjin that contribute to carbon emission reduction. The reasons for the carbon emission reduction contributions of different enterprises are slightly different, mainly due to high ownership or actual driving mileage.

An Analysis of Post COVID-19 Scenario using Data Science in Digital Marketing

The Purpose of the study is to analyze the uses of data science in digital marketing in post COVID-19 scenario. General design of the study includes survey of relevant research literature, primary data collection, structural modeling and critical analysis. The study uses Interpretive Structural Modeling (ISM) for structural modeling and Matriced' Impacts Croise's Multiplication Appliquée a UN Classement (MICMAC) for analysis. Introducing new products, personalizing customers’ online experience and improve user experience occupy top (Level I) and tracking social media commentary/interactions occupies bottom (Level IX) of ISM model. Analyzing user generated content, tracking social media commentary/interactions, analysis of online sales data, analyzing social media trends, analyzing product recommendations and reviews and analyze real-time big data are categorized as independent uses. Optimize customers’ preferences, optimize stock levels in e-commerce businesses, introducing new products, improve user experience and identify fake news & false content are categorized as dependent uses but others are categorized as linkage uses and no one is categorized in autonomous. It is an original study because it uses real time market survey data the findings of which are useful for folks of its stakeholders. It is particularly useful for marketers. It has serious implications for businesses since nowadays there is influx of data generation that has become a type of a noise for businesses. Use of data science not only converts this data noise into useful information but also an opportunity. This study provides lot of information about uses of data science particularly for marketing.

Neural network models of apple fruit identification in tree crowns: comparative analysis

The article presents the results of an analysis conducted from 2022 to 2023 to assess the quality of modern neural network models of apple fruit identification in tree crowns shown in images. In order to conduct the studies on identifying the best detector, the following neural networks were used: SSD (Single Shot MultiBox Detector), YOLOv4 (You Only Look Once, Version 4), YOLOv5, YOLOv7, and YOLOv8. The performance of the considered models of apple fruit identification was assessed using such binary classification metrics as precision, recall, accuracy, F-score, and AUC-ROCTotal (area under the curve). To assess the accuracy in predicting apple fruit identification, the mean absolute percentage error (MAPE) of the analyzed neural network models was calculated. The neural network performance analysis used 300 photographs taken at an apple garden. The conducted studies revealed that the SSD model provides lower speed and accuracy, as well as having high requirements for computing resources, which may limit its use in lower performance devices. The YOLOv4 model surpasses the YOLOv5 model in terms of accuracy by 10.2 %, yet the processing speed of the YOLOv5 model is over twice that of the YOLOv4 model. This fact makes the YOLOv5 model preferable for tasks related to real-time big data processing. The YOLOv8 model is superior to the YOLOv7 model in terms of speed (by 37.3 %); however, the accuracy of the YOLOv7 model is 9.4 % higher. The highest area under the Precision-Recall curve amounts to 0.94 when using the YOLOv7 model. This fact suggests a high probability that the classifier can accurately distinguish between the positive and negative values of the apple fruit class. MAPE calculation for the analyzed neural network models showed that the lowest error in apple fruit identification amounted to 5.64 % for the YOLOv7 model as compared to the true value determined using the visual method. The performance analysis of modern neural network models shows that the YOLO family of neural networks provides high speed and accuracy of object detection, which allows them to operate in real time. The use of transfer learning (tuning of only the last layers to solve highly specialized problems) to adjust the performance of models for different apple fruit varieties can further improve the accuracy of apple fruit identification.

Quality of service management in telecommunication network using machine learning technique

<span>Designing and implementing a fail-safe, real-time telecommunications network is complex. In modern networks, traditional quality of service (QoS) methods for monitoring and analyzing data have some problems, such as accuracy and efficient processing of big data in real time. To solve this problem, should use an appropriate intelligent crash classification system to detect and diagnose run-time errors. The article proposes to use a comprehensive fault detection system that includes QoS and machine learning technologies using information about the state of a point-to-point protocol over ethernet (PPPoE) session on PPPoE active discovery termination (PADT) virtual local area network (VLAN) routes. </span><span>This intelligent system is built using the machine learning method and is independent of the main real-time system. Demonstrated the operation of seven machine learning algorithms and presented the results of training and fault detection. Based on the received information about the state of the PPPoE session, the PADT type allows you to control the behavior of the real-time system.</span>

Pervasive Augmented Reality to support real-time data monitoring in industrial scenarios: Shop floor visualization evaluation and user study

Augmented Reality (AR) is a key technology in the transition to Industry 4.0 and smart manufacturing, gaining reputation in a wide range of industrial fields One promising application is in another field of Industry 4.0, the data monitoring, where AR can be used to visualize and interact with complex and big data in real time, potentially improving the efficiency and accuracy of decision-making In this work, we propose a Pervasive AR tool for data monitoring, developed in collaboration with industry partners. An web application of the same data monitoring function is also created for comparison purposes A Human-Centered Design (HCD) methodology was used to identify the needs and requirements of industrial analysts, which led to the development of these systems Preliminary user studies, with 17 participants having distinct levels of expertise in industry, data analysis and computer science, were conducted to collect opinions and suggestions for improvements, as well as, quantitative data regarding the technologies considered A succeeding user study was then prepared, in which 12 participants used two conditions (C1 — Hands-free Pervasive AR tool for HMDs and C2 — Web tool for tablet devices) to fulfill a set of data monitoring tasks in an industry assembly line The result of these studies confirm the potential of Pervasive AR for data monitoring, as it engages users, promote environmental awareness, contextualizes data and allows fast self-localization.

An Improved Big Data Analytics Architecture Using Federated Learning for IoT-Enabled Urban Intelligent Transportation Systems

The exponential growth of the Internet of Things has precipitated a revolution in Intelligent Transportation Systems, notably in urban environments. An ITS leverages advancements in communication technologies and data analytics to enhance the efficiency and intelligence of transport networks. At the same time, these IoT-enabled ITSs generate a vast array of complex data classified as Big Data. Traditional data analytics frameworks need help to efficiently process these Big Data due to its sheer volume, velocity, variety, and significant data privacy concerns. Federated Learning, known for its privacy-preserving attributes, is a promising technology for implementation within ITSs for IoT-generated Big Data. Nevertheless, the system faces challenges due to the variable nature of devices, the heterogeneity of data, and the dynamic conditions in which ITS operates. Recent efforts to mitigate these challenges focus on the practical selection of an averaging mechanism during the server’s aggregation phase and practical dynamic client training. Despite these efforts, existing research still relies on personalized FL with personalized averaging and client training. This paper presents a personalized architecture, including an optimized Federated Averaging strategy that leverages FL for efficient and real-time Big Data analytics in IoT-enabled ITSs. Various personalization methods are applied to enhance the traditional averaging algorithm. Local fine-tuning and weighted averaging tailor the global model to individual client data. Custom learning rates are utilized to boost the performance further. Regular evaluations are advised to maintain model efficacy. The proposed architecture addresses critical challenges like real-life federated environment settings, data integration, and significant data privacy, offering a comprehensive solution for modern urban transportation systems using Big Data. Using the Udacity Self-Driving Car Dataset foe vehicle detection, we apply the proposed approaches to demonstrate the efficacy of our model. Our empirical findings validate the superiority of our architecture in terms of scalability, real-time decision-making capabilities, and data privacy preservation. We attained accuracy levels of 93.27%, 92.89%, and 92.96% for our proposed model in a Federated Learning architecture with 10 nodes, 20 nodes, and 30 nodes, respectively.

Geostatistics on Real-Time Geodata Streams—High-Frequent Dynamic Autocorrelation with an Extended Spatiotemporal Moran’s I Index

The availability of spatial and spatiotemporal big data is increasing rapidly. Spatially and temporally high resolved data are especially gathered via the Internet of Things. This data can often be accessed as data streams that push new data tuples continuously and make the data available in real time. Such real-time spatiotemporal data have great potential for new analysis approaches based on modern data processing technologies. The ability to retrieve spatial big data in real time, as well as process it in real time, demands new analysis methodologies that catch up with the instantaneous and continuous character of today’s spatiotemporal data. In this work, we present an evaluation of a high-frequent dynamic spatiotemporal autocorrelation approach. This approach allows for geostatistical analysis of streaming spatiotemporal data in real time and can provide insights into spatiotemporal processes while they are still ongoing. To evaluate this new approach, it was applied to mobility data from New York City. The results show that a high-frequent dynamic spatiotemporal autocorrelation approach provides comparable and meaningful results. In this way, high-frequent geostatistical analyses in real time can become an addition to retrospective analyses based on historical data.

An Optimized Ensemble Support Vector Machine-Based Extreme Learning Model for Real-Time Big Data Analytics and Disaster Prediction

An Optimized Ensemble Support Vector Machine-Based Extreme Learning Model for Real-Time Big Data Analytics and Disaster Prediction

Real-Time Big Data Analytics and Proactive Traffic Safety Management Visualization System

Big data and data-driven analysis could be utilized for traffic management to improve road safety and the performance of transportation systems. This paper introduces a web-based proactive traffic safety management (PATM) and real-time big data visualization tool, which is based on an award-winning system that won the US Department of Transportation (USDOT) Solving for Safety Visualization Challenge and was selected as one of the USDOT Safety Data Initiative (SDI) Beta Tools. State-of-the-art research, especially for real-time crash prediction and PATM, are deployed in this study. A significant amount of real-time data is accessed by the system in order to conduct data-driven analysis, such as traffic data, weather data, and video data from closed-circuit television (CCTV) live streams. Based on the data, multiple modules have been developed, including real-time crash/secondary crash prediction, CCTV-based expedited detection, PATM recommendation, data sharing, and report generation. Both real-time data and the system outputs are visualized at the front end using interactive maps and various types of figures to represent the data distribution and efficiently reveal hidden patterns. Evaluation of the real-time crash prediction outputs is conducted based on one-month real-world crash data and the prediction results from the system. The comparison results indicate excellent prediction performance. When considering spatial-temporal tolerance, the sensitivity and false alarm rate of the prediction results [i.e., high crash potential event (HCPE)] are 0.802 and 0.252, respectively. Current and potential implementation are also discussed in this paper.

Cleaning Big Data Streams: A Systematic Literature Review

In today’s big data era, cleaning big data streams has become a challenging task because of the different formats of big data and the massive amount of big data which is being generated. Many studies have proposed different techniques to overcome these challenges, such as cleaning big data in real time. This systematic literature review presents recently developed techniques that have been used for the cleaning process and for each data cleaning issue. Following the PRISMA framework, four databases are searched, namely IEEE Xplore, ACM Library, Scopus, and Science Direct, to select relevant studies. After selecting the relevant studies, we identify the techniques that have been utilized to clean big data streams and the evaluation methods that have been used to examine their efficiency. Also, we define the cleaning issues that may appear during the cleaning process, namely missing values, duplicated data, outliers, and irrelevant data. Based on our study, the future directions of cleaning big data streams are identified.

A Comprehensive Approach to Assessing Yield Map Quality in Smart Agriculture: Void Detection and Spatial Error Mapping

Smart agriculture relies on accurate yield maps as a crucial tool for decision-making. Many yield maps, however, suffer from spatial errors that can compromise the quality of their data, while several approaches have been proposed to address some of these errors, detecting voids or holes in the maps remains challenging. Additionally, the quality of yield datasets is typically evaluated based on root mean squared errors after interpolation. This evaluation method relies on weighbridge data, which can occasionally be inaccurate, impacting the quality of decisions made using the datasets. This paper introduces a novel algorithm designed to identify voids in yield maps. Furthermore, it maps three types of spatial errors (GPS errors, yield surges, and voids) to two standard data quality dimensions (accuracy and completeness). Doing so provides a quality score that can be utilized to assess the quality of yield datasets, eliminating the need for weighbridge data. The paper carries out three types of evaluations: (1) evaluating the algorithm’s efficacy by applying it to a dataset containing fields with and without voids; (2) assessing the benefits of integrating void detection and other spatial error identification techniques into the yield data processing chain; and (3) examining the correlation between root mean squared error and the proposed quality score before and after filtering out spatial errors. The results of the evaluations demonstrate that the proposed algorithm achieves a 100% sensitivity, 91% specificity, and 82% accuracy in identifying yield maps with voids. Additionally, there is a decrease in the root mean squared error when various spatial errors, including voids after applying the proposed data pre-processing chain. The inverse correlation observed between the root mean squared error and the proposed quality score (−0.577 and −0.793, before and after filtering spatial errors, respectively) indicates that the quality score can effectively assess the quality of yield datasets. This assessment enables seamless integration into real-time big data quality assessment solutions based on various data quality dimensions.