Massive Data Streams Research Articles

Real-time dynamic polygenic prediction for streaming data.

Polygenic risk scores (PRSs) are promising tools for advancing precision medicine. However, existing PRS construction methods rely on static summary statistics derived from genome-wide association studies (GWASs), which are often updated at lengthy intervals. As genetic data and health outcomes are continuously being generated at an ever-increasing pace, the current PRS training and deployment paradigm is suboptimal in maximizing the prediction accuracy of PRSs for incoming patients in healthcare settings. Here, we introduce real-time PRS-CS (rtPRS-CS), which enables online, dynamic refinement and calibration of PRS as each new sample is collected, without the need to perform intermediate GWASs. Through extensive simulation studies, we evaluate the performance of rtPRS-CS across various genetic architectures and training sample sizes. Leveraging quantitative traits from the Mass General Brigham Biobank and UK Biobank, we show that rtPRS-CS can integrate massive streaming data to enhance PRS prediction over time. We further apply rtPRS-CS to 22 schizophrenia cohorts in 7 Asian regions, demonstrating the clinical utility of rtPRS-CS in dynamically predicting and stratifying disease risk across diverse genetic ancestries.

A Novel Data Analysis Architectural Framework for Data Reduction on Edges for Smart City Healthcare

The most difficult task is managing and digesting the massive amount of heterogeneous data created by IoT. It questioned the efficiency and computational storage capacity of today's infrastructure resources. Furthermore, because there is so much data, it becomes difficult to give the user enough information to make a decision. A technology known as Complex Event Processing (CEP) has been created to glean important insights from massive real-time data streams. The goal of this research is to create a system that can extract complicated events from large amounts of sensor data and only transmit those events to a cloud server. Global healthcare services have been expanding at an exponential rate because of the vast amount of data that clinical and medical organizations generate daily. Healthcare systems can use e-health services to meet the medical and assistive requirements of people. Edge technology is proven in this direction. Mixing the old classic concepts with the innovative we have proposed the Novel data analysis architecture for smart heart disease prediction. We have created the fuzzy-based inference system at the edges for the data analysis and used the resulting Complex Events at the end predictions.

Integrating Artificial Intelligence with Big Data for Real-Time Insights and Decision-Making in Complex Systems

Artificial intelligence and big data are paramount for generating real-time insights enabling decision-making in complex systems. Integrating massive data streams and AI algorithms presents huge opportunities for extracting actionable insights at unprecedented speed and precision. This paper discusses how integrating artificial intelligence and big data facilitates handling complex, dynamic tasks in the health, finance, and supply chain management industries. The study describes how advanced machine learning models, neural networks, and decision algorithms enable these systems to process big data in real-time, even as that helps improve predictive and adaptive decision-making. The paper discusses our performance evaluation of AI-driven decision systems, focusing on architecture that supports efficient data processing. In addition, we present a framework that elucidates how AI models interpret Big Data within multi-layered, real-time environments. The study will also include results in terms of impedance and multi-line graphs to demonstrate system performances. We will also provide some of the tables in the key metrics. This study highlights the benefits and drawbacks of AI data integration and its potential implementation.

Privacy Challenges and Solutions in Big Data Analytics: A Comprehensive Review

Abstract: Large data collections with a more intricate and diversified structure are referred to as "big data." These traits are typically associated with more challenges when it comes to storing, analyzing, and applying additional procedures or getting results. The technique of analyzing vast quantities of intricate data to uncover patterns or connections that are concealed is known as "big data analytics." Nonetheless, there is a clear conflict between big data's extensive use and its security and privacy. This paper examines privacy needs in big data, distinguishes between privacy and security, and focuses on privacy and security issues in large data. This study addresses the applications of privacy in business by utilizing established techniques like HybrEx, k-anonymity, T-closeness, and L-diversity. Many privacy-preserving techniques have been created for protection of privacy at various phases of the big data life cycle (such as data generation, storage, and processing). This work aims to explain the issues facing current privacy preservation strategies in big data and to give a comprehensive evaluation of them. This study also introduces new approaches to privacy preservation in big data, such as quick anonymization of massive data streams, identity-based anonymization, differential privacy, privacy preserving big data publishing, and hiding a needle in a haystack. This study discusses big data privacy and security issues in healthcare. A comparative analysis of some contemporary large data privacy approaches is also conducted. A comparative analysis of the latest big data privacy approaches is also conducted.

Improved sand cat swarm optimization with deep learning based enhanced malicious activity recognition for cybersecurity

The main concept of a smart city is to join manual items with electronics, software, sensors, and network connectivity for data contact via Internet of Things (IoT) gadgets. IoT improves production and efficiency by cleverly utilizing remote management, but security as well as privacy risk, rises. Cyber threats are progressing every day, affecting inadequate actions of safety and privacy. In a smart city, the exposed action of an individual or business can keep the whole city in danger. Owing to the trust of numerous modules of smart cities on data and communication techniques, cybersecurity tasks like information leakage and mischievous cyber-attacks. Malicious Activity Detection in the area of cybersecurity for smart cities is dominant in protecting the consistent organization of urban atmospheres. Leveraging innovative machine learning (ML) and deep learning (DL) approaches, mainly anomaly detection methods permits the identification of abnormal patterns and probable threats within the massive streams of data produced by smart city methods. So, this paper designs a Fusion of Optimization Algorithms and DL for Enhanced Malicious Activity Recognition (FOADL-EMAR) technique for cybersecurity in smart cities. The purpose of the FOADL-EMAR technique is to recognize the presence of malicious activities using optimal DL techniques in the environment of a smart city. To accomplish this, the FOADL-EMAR technique uses a Z-score data normalization model to measure the input dataset into a useful format. In addition, the FOADL-EMAR technique uses an improved sand cat swarm optimization (ISCSO) algorithm that can be applied to select an optimal subset of features. An ensemble of three DL techniques is used namely long short-term memory (LSTM), deep neural networks (DNNs), and extreme learning machine (ELM) for malicious activity detection. The Harris Hawks Optimization (HHO) approach has been applied to improve the detection results of the ensemble models. The simulation analysis of the FOADL-EMAR methodology occurs utilizing a benchmark database. The experimentation values demonstrate the improved detection outcomes of the FOADL-EMAR algorithm over other existing methods in terms of different measures.

A novel data Analysis architectural framework for data reduction on edges for Smart City Healthcare

The most difficult task is managing and digesting the massive amount of heterogeneous data created by IoT. It questioned the efficiency and computational storage capacity of today's infrastructure resources. Furthermore, because there is so much data, it becomes difficult to give the user enough information to make a decision. A technology known as Complex Event Processing (CEP) has been created to glean important insights from massive real-time data streams. The goal of this research is to create a system that can extract complicated events from large amounts of sensor data and only transmit those events to a cloud server. Global healthcare services have been expanding at an exponential rate because of the vast amount of data that clinical and medical organizations generate daily. Healthcare systems can use e-health services to meet the medical and assistive requirements of people. Edge technology is proven in this direction. Mixing the old classic concepts with the innovative we have proposed the Novel data analysis architecture for smart heart disease prediction. We have created the fuzzy-based inference system at the edges for the data analysis and used the resulting Complex Events at the end predictions.

A Novel Data Analysis Architectural Framework for Data Reduction on Edges for Smart City Healthcare

The most difficult task is managing and digesting the massive amount of heterogeneous data created by IoT. It questioned the efficiency and computational storage capacity of today's infrastructure resources. Furthermore, because there is so much data, it becomes difficult to give the user enough information to make a decision. A technology known as Complex Event Processing (CEP) has been created to glean important insights from massive real-time data streams. The goal of this research is to create a system that can extract complicated events from large amounts of sensor data and only transmit those events to a cloud server. Global healthcare services have been expanding at an exponential rate because of the vast amount of data that clinical and medical organizations generate daily. Healthcare systems can use e-health services to meet the medical and assistive requirements of people. Edge technology is proven in this direction. Mixing the old classic concepts with the innovative we have proposed the Novel data analysis architecture for smart heart disease prediction. We have created the fuzzy-based inference system at the edges for the data analysis and used the resulting Complex Events at the end predictions.

The Importance of Educational Data Mining and Learning Analytics for Improving Teaching and Learning: An Issue Brief

“The words educational data mining and learning analytics are frequently used interchangeably, despite their being an increase in their investigation and implementation. This may be as a result of the fact that both areas have similar conceptual components. One way to ensure precision, homogeneity, and consistency It aims to pinpoint themes that are similar to and different from one other in the two domains as they develop. This a topic modelling study of papers on educational data mining and learning analytics was carried out in the elucidate the two areas' respective themes. In particular, we used structural topic modelling to find the two domains' subjects from the abstracts. For instructional purposes, we use structural topic modelling on N 1 4192 articles. For both educational data and survey data, we infer five-topic models analytics for mining and learning. While there may be disciplinary variations in research, our findings show that beyond their various lineages, there is no evidence to indicate a clear separation between the two disciplines. the area of educational research on the uses of advanced statistical methods is trending toward convergence for improving teaching and learning, discover how to mine massive data streams for insights that may be put to use. Over the past five years, both areas have converged on a growing emphasis on student behaviour. This study topic has advanced greatly, and a variety of related words, including Academic Analytics, Institutional Analytics, Teaching Analytics, Data-Driven Education, Data-Driven Decision-Making in Education, Big Data in Education, and Educational Data Science, are now used in the paper. The main publications, significant turning points, cycle of knowledge discovery, primary educational settings, specialised tools, freely accessible datasets, widely used methodologies, primary goals, and anticipated trends in this field of study are reviewed to provide the state of the art at this time.

Towards flexible data stream collaboration: Federated Learning in Kafka-ML

Federated learning is applied in scenarios where organisations lack sufficient data volume for modelling their business logic and cannot share their data with external parties. Moreover, Industry 4.0 and IoT scenarios generate massive data streams, which normally are fed to ML/AI solutions for model training and prediction. However, in most cases, ML/AI frameworks are not prepared to work with these streaming pipelines. In this paper, we present an asynchronous federated learning solution based on the Kafka-ML data stream framework, which is able to combine federated learning and data stream capabilities within ML/AI applications. While most federated learning approaches are tailored to a specific ML model or a use case, the solution provided adapts itself to the availability of both data and ML models, achieving a flexible and dynamic federated learning solution. To validate its performance, an evaluation of the federated learning solution is carried out on different scenarios in a multi-node state-of-the-art infrastructure. Results show that this framework can work with multiple federated clients, being the resulting accuracy dependent on the amount of data and the behaviour of clients during training.

A semi-supervised load identification method with class incremental learning

With the proposal of the carbon peaking and neutrality targets, non-intrusive load monitoring (NILM) is crucial for energy saving and demand response. How to achieve accurate load identification (LI) is the key to the application of NILM. However, it faces a major challenge: how to identify loads accurately from massive incremental unlabeled data streams. To tackle this challenge, we propose a novel method that combines class incremental learning (CIL) and semi-supervised learning (SSL). Our method prevents catastrophic forgetting by preserving samples, distilling knowledge and aligning weights in incremental tasks. Moreover, our method leverages a semi-supervised learning structure called the Temporal Ensembling to exploit unlabeled data and overcome the semi-supervised problem in incremental learning. We test our method on PLAID and WHITED public datasets and demonstrate its effectiveness.

Rapid and Automatic Image Acquisition System for Structural Surface Defects of High-Speed Rail Tunnels

Continuous high-speed (>80 km/h) tunnel-image detection introduces new challenges to defect image acquisition, including machine recognition of entry and exit from a tunnel, storage of massive image data streams, image distortion due to surface photography, and encoder errors. In this study, an automatic image acquisition system is designed, and several critical technologies are proposed for high-speed rail tunnels. The system realizes quick tunnel identification and automatic camera control and uses the proposed line-scan software-matching method to obtain accurate images. Before storage, the real-time modification of the distorted images is implemented using an error-correction algorithm. An alternative mapping storage algorithm is proposed to improve the efficiency and stability of long-term storage. The test results show that the proposed method effectively reduces photographic errors. The lateral pixel-error rate of the corrected image is 1.60%, which is 10 times lower than that of the pre-correction, and the error rate of the longitudinal image is controlled within 10% when the system is moving at a variable speed and within 1% when it is moving at a constant speed. Furthermore, experiments have proven that the alternate mapping storage algorithm improves the storage efficiency of RAID by 50% and ensures data integrity; storage is accomplished by 8 HDDs at a camera throughput of 1.52 GB/s. This study will contribute to improvements in the speed and accuracy of tunnel defect-image detection.

Publish/Subscribe Method for Real-Time Data Processing in Massive IoT Leveraging Blockchain for Secured Storage.

In the Internet of Things (IoT) era, the surge in Machine-Type Devices (MTDs) has introduced Massive IoT (MIoT), opening new horizons in the world of connected devices. However, such proliferation presents challenges, especially in storing and analyzing massive, heterogeneous data streams in real time. In order to manage Massive IoT data streams, we utilize analytical database software such as Apache Druid version 28.0.0 that excels in real-time data processing. Our approach relies on a publish/subscribe mechanism, where device-generated data are relayed to a dedicated broker, effectively functioning as a separate server. This broker enables any application to subscribe to the dataset, promoting a dynamic and responsive data ecosystem. At the core of our data transmission infrastructure lies Apache Kafka version 3.6.1, renowned for its exceptional data flow management performance. Kafka efficiently bridges the gap between MIoT sensors and brokers, enabling parallel clusters of brokers that lead to more scalability. In our pursuit of uninterrupted connectivity, we incorporate a fail-safe mechanism with two Software-Defined Radios (SDR) called Nutaq PicoLTE Release 1.5 within our model. This strategic redundancy enhances data transmission availability, safeguarding against connectivity disruptions. Furthermore, to enhance the data repository security, we utilize blockchain technology, specifically Hyperledger Fabric, known for its high-performance attributes, ensuring data integrity, immutability, and security. Our latency results demonstrate that our platform effectively reduces latency for 100,000 devices, qualifying as an MIoT, to less than 25 milliseconds. Furthermore, our findings on blockchain performance underscore our model as a secure platform, achieving over 800 Transactions Per Second in a dataset comprising 14,000 transactions, thereby demonstrating its high efficiency.

VEAD: Variance profile Exploitation for Anomaly Detection in real-time IoT data streaming

The explosion of online streaming services in the Internet-of-Things (IoT) ecosystem poses new difficulties in detecting anomalies in real-time and continuous data. The IoT data anomalies are classified into long-term and short-term, with unique characteristics that make it difficult to develop a common detection mechanism for both. The existing techniques require excessive training data and suffer from high variability. This paper overcomes these challenges by proposing a Variance profile Exploitation for Anomaly Detection (VEAD) scheme using discrete wavelet transform and k-means clustering. VEAD is initialized by a fast training phase with a single data segment, from which a sensor variance profile is created. This variance profile reflects the degree of deviation in the collected data from different sensors at a specific time period and is continuously updated by integrating new data segments for effective anomaly detection. Overlapping data collection in the detection phase shows correlations among consecutive data segments, leading to improved detection accuracy (ACC). The Intel Berkeley Research Lab dataset with injected synthetic anomalies is used to perform numerical experiments. A comparative performance evaluation with state-of-the-art methods confirms the effectiveness of VEAD in achieving a higher ACC and a lower false-positive rate (FPR). Notably, 95% and 97% ACCs are achieved in detecting long-term and short-term anomalies, respectively. The high specificity of VEAD is also revealed by the low FPR of at most 2% in all cases. The low computational complexity and fast anomaly detection make VEAD suitable for deployment in live systems with massive data streams.

Network Traffic Prediction Model in a Data-Driven Digital Twin Network Architecture

The proliferation of immersive services, including virtual reality/augmented reality, holographic content, and the metaverse, has led to an increase in the complexity of communication networks, and consequently, the complexity of network management. Recently, digital twin network technology, which applies digital twin technology to the field of communication networks, has been predicted to be an effective means of managing complex modern networks. In this paper, a digital twin network data pipeline architecture is proposed that demonstrates an integrated structure for flow within the digital twin network and network modeling from a data perspective. In addition, a network traffic modeling technique using data feature extraction techniques is proposed to realize the digital twin network, which requires the use of massive streaming data. The proposed method utilizes the data generated in the OMNeT++ environment and verifies that the learning time is reduced by approximately 25% depending on the feature extraction interval, while the accuracy remains similar.

A big data exploration approach to exploit in-vehicle data for smart road maintenance

In modern Smart Cities, pervasive collection of sensor-based and IoT data streams is a challenging opportunity for improving mobility resilience. Among the potential applications, sensor-based data streams provide valuable information about the quality of the area-wide road surface. Modern vehicle black boxes are also able to estimate the type of anomaly (e.g., bump, hole, rough ground, depression), based on real-time analysis of acceleration data streams. Road maintainers may use all this information to improve monitoring and maintenance activities. However, the volume of data streams, the variety of road network and different degrees of seriousness of detected anomalies call for methods to support maintainers in the exploration of available data. To this aim, in this paper, we propose a methodological approach, based on big data exploration techniques. The approach is grounded on: (i) a multi-dimensional model, apt to organise data streams according to different dimensions and enable data exploration; (ii) data summarisation techniques, based on an incremental clustering algorithm, to simplify the overall view over massive data streams and to cope with their dynamic nature; (iii) a measure of relevance, to focus the attention on road portions that present critical conditions. The innovative contributions regard the formalisation of the exploration methodology, the definition of exploration scenarios, based on road maintainers’ goals and the measure of relevance, and an extensive experimentation on a real world case study, addressed in a research project on smart and resilient mobility. Experimental results show how relevance evaluation is able to efficiently attract the road maintainers’ attention on road portions that present the most critical conditions and the proposed incremental clustering algorithm outperforms existing algorithms in the literature.

Application of k-sigma Algorithm Based on Monte Carlo in Outlier Detection

Under the background of intelligent times based on information technology, the analysis and replacement of outliers become particularly critical for databases in the face of massive sequential data streams. In order to improve the effectiveness and practicability of the detection method, this paper determines the abnormal scores of data points in a specific data set by using the K-Sigma algorithm of the Monte Carlo method, adaptively adjusts the k value according to the abnormal scores, and marks the abnormal points.The advantages of the traditional k-sigma algorithm are fast operation speed and theoretical basis, while the disadvantages are that each index dimension is determined independently. However, in industrial production, due to the multi-index dimension of time series data, multiple size indicators of a data set are related to each other. Therefore, it is necessary to comprehensively consider whether an anomaly is abnormal by combining it with the data of other indicators. In addition, when there are no outliers in the data set, the traditional k-sigma algorithm is used to take the mean and variance of the data set as parameters. Due to the limitations of the data set itself, some data points will be mislabeled as outliers. Through the k-sigma algorithm based on the Monte Carlo method, we can effectively solve the above problems.The k-sigma algorithm based on the Monte Carlo method can generate a normal distribution according to the distribution of original data points, and extract a large quantity of data from the distribution to generate Monte Carlo data set. The Mahalanobis distance li from each sample point to the mean in the Monte Carlo data set is calculated and compared with the Mahalanobis distance lj from the samples to be detected to the mean in the original data set. According to the number of sample points satisfying li < lj in the Monte Carlo data set, the value of the parameter kj is determined adaptively, and thus the outliers are determined. We implemented the k-sigma algorithm based on the Monte Carlo method through python, evaluated the effectiveness of the algorithm by accuracy, recall rate, and F1 score, and compared it with some machine learning algorithms. The results verified the feasibility and effectiveness of the algorithm, which can be used for real-time anomaly detection in the energy management database.

Efficient Framework for Operating on Data Sketches

We study the problem of analyzing massive data streams based on concise data sketches. Recently, a number of papers have investigated how to estimate the results of set-theory operations based on sketches. In this paper we present a framework that allows to estimate the result of any sequence of set-theory operations. The starting point for our solution is the solution from 2021. Compared to this solution, the newly presented sketching algorithm is much more computationally efficient as it requires on average O (log n ) rather than O ( n ) comparisons for n stream elements. We also show that the estimator dedicated to sketches proposed in that reference solution is, in fact, a maximum likelihood estimator.

Efficient and Secure Quantile Aggregation of Private Data Streams

Computing the quantile of a massive data stream has been a crucial task in networking and data management. However, existing solutions assume a centralized model where one data owner has access to all data. In this paper, we put forward a study of secure quantile aggregation between <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">private</i> data streams, where data streams owned by different parties would like to obtain a quantile of the union of their data without revealing anything else about their inputs. To this end, we designed efficient cryptographic protocols that are secure in the semi-honest setting as well as the malicious setting. By incorporating differential privacy, we further improve the efficiency by 1.1× to 73.1×. We implemented our protocol, which shows practical efficiency to aggregate real-world data streams efficiently.

Linking scientific instruments and computation: Patterns, technologies, and experiences

SummaryPowerful detectors at modern experimental facilities routinely collect data at multiple GB/s. Online analysis methods are needed to enable the collection of only interesting subsets of such massive data streams, such as by explicitly discarding some data elements or by directing instruments to relevant areas of experimental space. Thus, methods are required for configuring and running distributed computing pipelines—what we call flows—that link instruments, computers (e.g., for analysis, simulation, artificial intelligence [AI] model training), edge computing (e.g., for analysis), data stores, metadata catalogs, and high-speed networks. We review common patterns associated with such flows and describe methods for instantiating these patterns. We present experiences with the application of these methods to the processing of data from five different scientific instruments, each of which engages powerful computers for data inversion,model training, or other purposes. We also discuss implications of such methods for operators and users of scientific facilities.

IFogLearn++: A new platform for fog layer's IoT attack detection in critical infrastructure using machine learning and big data processing

Nowadays, with an increasing expansion of the internet of things (IoT) that has created massive data streams, online attack detection via stream processing has become a matter of extensive attention. Current encryption and authentication methods cannot satisfy the security requirements of IoT critical infrastructures because of many heterogeneous connected devices, a wide network geographical scope, rapid software development, the possibility of security bugs, and the emergence of new attacks. Therefore, an extended method based on machine learning, which can process data stream to detect the fog layer's IoT attacks and prevent the spread of intrusion to other network segments, was proposed in this paper. The IFogLearn++ uses a fog layer to facilitate fast data stream processing in the fog layer. Using a fog layer helps the network tolerate more attacks due to the annexation of an extra security layer before the cloud layer. Based on the results, the IFogLearn++ has similar accuracy to competitors and 18 and 8 times faster performance compared to SVM and Learn++. This characteristic helps secure massive data streams in IoT.