Data Streams Research Articles

A unified framework for high-dimensional data stream analysis in fault diagnosis

Modern industrial systems routinely generate data in high volume at high velocity. These high-dimensional data streams (HDDS) provide valuable information at granular levels to quality personnel during root cause investigation in cases of a system fault. The goal of fault analysis using HDDS is twofold: (1) identify abnormal data streams and (2) locate the change point when the processes become out of control. Existing research has largely focused on addressing the two issues separately. In this article, we propose a unified framework by formulating the problem as optimal control of hierarchical missed discovery rates in multiple classifications. Theoretically, we establish that our approach minimizes the number of false discoveries while controlling the missed discovery rates at desired levels. Numerically, we develop a computationally efficient algorithm for solving the optimization and demonstrate its superior performance over the existing methods. A data-driven version of the proposed approach is suggested as well. An application to a real data set in semiconductor manufacturing shows that our approach works well in practice.

Digital Microbe: a genome-informed data integration framework for team science on emerging model organisms

The remarkable pace of genomic data generation is rapidly transforming our understanding of life at the micron scale. Yet this data stream also creates challenges for team science. A single microbe can have multiple versions of genome architecture, functional gene annotations, and gene identifiers; additionally, the lack of mechanisms for collating and preserving advances in this knowledge raises barriers to community coalescence around shared datasets. “Digital Microbes” are frameworks for interoperable and reproducible collaborative science through open source, community-curated data packages built on a (pan)genomic foundation. Housed within an integrative software environment, Digital Microbes ensure real-time alignment of research efforts for collaborative teams and facilitate novel scientific insights as new layers of data are added. Here we describe two Digital Microbes: 1) the heterotrophic marine bacterium Ruegeria pomeroyi DSS-3 with > 100 transcriptomic datasets from lab and field studies, and 2) the pangenome of the cosmopolitan marine heterotroph Alteromonas containing 339 genomes. Examples demonstrate how an integrated framework collating public (pan)genome-informed data can generate novel and reproducible findings.

Fishing event detection and species classification using computer vision and artificial intelligence for electronic monitoring

Fisheries regulations require detailed catch reporting on commercial fishing vessels. Vital components for the sustainable management of fish stocks include a robust estimate of the number of fish caught and the species composition. Catch recording is often done manually by human observers on fishing vessels. Human observers are costly, and consistent data streams can be subject to observer availability and the weather. On-vessel cameras (electronic monitoring, EM) are a growing alternative to human observers. However, on-land human auditors are required to review hundreds of hours of videos recorded during fishing trips that can last for weeks. In this paper, a framework is presented to automatically detect fish in EM videos, count the total fishing events, and classify the fish species. For this purpose, a deep learning and computer vision-based model is developed to efficiently detect fish and fishers onboard a vessel. Secondly, a vision-based tracking pipeline tracks the detected fish and counts the total fishing events in the videos. Thirdly, the extracted fishing events are classified through a deep learning-based fish species classifier, to provide the distribution of different fish species caught for a fishing trip. For our experiments, the datasets were prepared using the electronic monitoring data of multiple fishing trips of a fishing vessel. The videos were recorded on Australian longline vessels targeting tunas and billfish. For the fish detection task, video frames were extracted and labelled manually to provide a digital ground-truth. For the fish species classification task, hundreds of fish images of multiple species were cropped to provide a training dataset for the fish classifier. For the fish counting task, manual counts for the fishing events of individual fish species were generated for the test fishing trips. The developed fish and fisher detector achieves a mean Average Precision of 87.0 % for fish and 94.0 % for fishers on test video frames. The fishing event detection pipeline achieves an Average Precision of 81.0 % and an Average Recall of 74.5 % on test videos. The fish species classifier achieves an Accuracy (Top-1) of 91.11 % for the classification of cropped fish images and 89.05 % for the classification of extracted fishing events from the videos. Experimental results show that our proposed computer vision and artificial intelligence-based solution for video analysis has great potential to automate the auditing process from electronic monitoring footage and contribute to the sustainable management of fish stocks.

Online imbalance learning with unpredictable feature evolution and label scarcity

Recently, online learning with imbalanced data streams has aroused wide concern, which reflects an uneven distribution of different classes in data streams. Existing approaches have conventionally been conducted on stationary feature space and they assume that we can obtain the entire labels of data streams in the case of supervised learning. However, in many real scenarios, e.g., the environment monitoring task, new features flood in, and old features are partially lost during the changing environment as the different lifespans of different sensors. Besides, each instance needs to be labeled by experts, resulting in expensive costs and scarcity of labels. To address the above problems, this paper proposes a novel Online Imbalance learning with unpredictable Feature evolution and Label scarcity (OIFL) algorithm. First, we utilize margin-based online active learning to selectively label valuable instances. After obtaining the labels, we handle imbalanced class distribution by optimizing F-measure and transforming F-measure optimization into a weighted surrogate loss minimization. When data streams arrive with augmented features, we combine the online passive-aggressive algorithm and structural risk minimization to update the classifier in the divided feature space. When data streams arrive with incomplete features, we leverage variance to identify the most informative features following the empirical risk minimization principle and continue to update the existing classifier as before. Finally, we obtain a sparse but reliable learner by the strategy of projecting truncation. We derive theoretical analyses of OIFL. Also, experiments on the synthetic datasets and real-world data streams to validate the effectiveness of our method.

Machine learning based damage identification in SiC/SiC composites from acoustic emissions using autoencoders

Developing the ability to leverage machine learning (ML) to identify damage mechanisms in heterogeneous materials from their acoustic emissions (AE) has wide-reaching ramifications for multi-modal experimentation. It would allow researchers to augment damage triangulation, lifetime prediction, and high-resolution optical studies with complementary mechanism-informed data streams. However, developing this capability hinges on the collection of ground truth acoustic libraries from damage in realistic geometries. Due to time and monetary considerations, there is a dearth of ground truth libraries which can be used to robustly characterize ML mechanism identification frameworks. Addressing this gap, we present a multi-modal acoustic emission and x-ray computed tomography study where AE is gathered, and subsequently labeled, from SiC/SiC unidirectional composites under monotonic tension. This library is used to demonstrate that acoustic signals from early fiber breaks are obscured by matrix cracking. A first-order micromechanical model is used to explain the origin of this obscuring effect, and identify fundamental limitations of unsupervised frameworks. An autoencoder-based anomaly detector approach is used for the first time to overcome these limitations, additionally demonstrating that the frequency distribution of fiber break acoustic signals is narrow. Implications of these findings for enhanced multi-modal testing and online health monitoring are discussed, and strategies for implementation of supervised damage mechanism identification frameworks are proposed.

Online updating Huber robust regression for big data streams

ABSTRACT Big data streams have garnered significant attention in multiple industries. However, the immense volume and the presence of outliers in high-velocity streaming data pose great challenges to its analysis. To address these concerns, this paper introduces a novel Online Updating Huber Robust Regression algorithm. By efficiently capturing the salient features of new data subsets, a computationally efficient online updating estimator is proposed without the need for storing historical data. Furthermore, by incorporating Huber regression into its framework, the estimator exhibits robustness to heavy-tailed, heterogeneous as well as outlier-contaminated data. Theoretically, the proposed online updating estimator is asymptotically equivalent to an Oracle estimator derived from the entire dataset. Extensive numerical simulations and a real-world data analysis have been conducted to demonstrate the effectiveness and practicality of the proposed method.

Learning from high-dimensional cyber-physical data streams: a case of large-scale smart grid

Learning from high-dimensional cyber-physical data streams: a case of large-scale smart grid

An Investigation on Handling IoT Big Data with RBSEE Architecture

Internet of Things (IoT) devices have resulted in an unprecedented surge in data generation, presenting both opportunities and challenges in data management and analytics. This paper investigates the feasibility and effectiveness of utilizing the RBSEE (Resource-Based Storage, Edge Computing, and Elasticity) architecture for handling IoT Big Data. Firstly, the paper elucidates the fundamental concepts of IoT and Big Data, highlighting the unique characteristics and challenges associated with managing large volumes of heterogeneous data streams generated by IoT devices. Subsequently, it introduces the RBSEE architecture, which integrates resource-based storage, edge computing, and elasticity to address the scalability, latency, and resource constraints inherent in IoT environments. The architecture's ability to distribute data storage and processing tasks across edge devices and cloud infrastructure while dynamically adapting to fluctuating workloads makes it particularly suitable for handling IoT Big Data. In any typical IoT ecosystem, real-time and instantaneous handling of request and dataset is pivotal and a fundamental property. This aims to propose energy efficient distributed system architecture for persuasive, privacy-preserved and real time handling of data generated from IoT devices.

A dynamic multicast routing protocol for visible light communication networks

Due to the current challenges of wireless communications, optical wireless communications, like visible light communication (VLC), have become very popular in the last decade for local area networks. However, current VLC systems focus mainly on one-hop direct transmissions, ignoring that many networking applications require multicast communications. In this sense, we propose MCAST-VLC, a fully functional cross-layer dynamic multicast scheme over a dynamic routing protocol for VLC. MCAST-VLC considers the particular characteristics of the VLC as the need for group management, traffic isolation, a challenging multi-hop route discovery protocol, and a complex cross-layer VLC mechanism to maintain the proper system state. We evaluate our protocol following close real indoor VLC scenarios, previously discussed in the literature, where a light source in a room streams data to a few other VLC devices, and we extend our results numerically. Our results show that MCAST-VLC can handle considerably more network flows than a unicast VLC system under a negligible overhead.

Enhanced query performance for stored streaming data through structured streaming within spark SQL

Traditional database systems like relational databases can store data which are structured with predefined schema, but in the case of bigdata, the data comes in different formats or are collected from diverse sources. The distributed databases like not only spark querying language (NoSQL) repositories are often used in relation to bigdata analytics, but a continual updating is required in business because of the streaming data that comes from stock trading, online activities of website visitors, and from the mobile applications in real time. It will not have to delay, for some report to show up, to assess and analyse the current situation, to move forward with the next business choice. Apache Spark’s structured streaming offer capabilities for handling streaming data in a batch processing mode with faster responses compared to MongoDB which is a document-based NoSQL database. This study completes similar queries to evaluate Spark SQL and NoSQL database performance, focusing on the upsides of Spark SQL over NoSQL databases in streaming data exploration. The queries are completed with streaming data stored in a batch mode.

Data streaming infra-red video bolometer (IRVB) of Korea Superconducting Tokamak Advanced Research (KSTAR).

Due to the increasing demands for active plasma control operations, in situ diagnostics are highly sought after. Tungsten plasma-facing components have been utilized in the Korea Superconducting Tokamak Advanced Research (KSTAR) lower divertor since the 2023 campaign. Plasma radiation is a key parameter for plasma control, especially in radiation front control experiments. Therefore, the KSTAR infra-red video bolometer (IRVB) needs to be reconfigured into an in situ data streaming diagnostic. This requires comprehensive changes in both infra-red camera control and data analysis compared to the previous system. To ensure the stability of the reconfigured system, functional parts are grouped and separated into individual processes to protect camera acquisitions from errors in other processes. In addition, to enhance the speed of data streaming analysis, the analysis code has been optimized and converted into graphics processing unit (GPU) code. Besides the data streaming analysis, the system is also designed to support post-shot analysis with the entire frame data for the same shot to address frame drop issues encountered during data streaming. Radiation front control experiments with N2 gas seeding are successful results of the data streaming IRVB for its commissioning. This paper focuses on the development of the data streaming IRVB system.

Overview of Wind and Photovoltaic Data Stream Classification and Data Drift Issues

The development in the fields of clean energy, particularly wind and photovoltaic power, generates a large amount of data streams, and how to mine valuable information from these data to improve the efficiency of power generation has become a hot spot of current research. Traditional classification algorithms cannot cope with dynamically changing data streams, so data stream classification techniques are particularly important. The current data stream classification techniques mainly include decision trees, neural networks, Bayesian networks, and other methods, which have been applied to wind power and photovoltaic power data processing in existing research. However, the data drift problem is gradually highlighted due to the dynamic change in data, which significantly impacts the performance of classification algorithms. This paper reviews the latest research on data stream classification technology in wind power and photovoltaic applications. It provides a detailed introduction to the data drift problem in machine learning, which significantly affects algorithm performance. The discussion covers covariate drift, prior probability drift, and concept drift, analyzing their potential impact on the practical deployment of data stream classification methods in wind and photovoltaic power sectors. Finally, by analyzing examples for addressing data drift in energy-system data stream classification, the article highlights the future prospects of data drift research in this field and suggests areas for improvement. Combined with the systematic knowledge of data stream classification techniques and data drift handling presented, it offers valuable insights for future research.

SSKG: Subject stream knowledge graph, a new approach for event detection from text

Events play a crucial role in shaping societal meaning and discourse. They are recorded in text data streams from social networks or online sources and analyzing them is vital for predicting and managing future occurrences. Methods for event detection from text data streams have been developed to enhance accuracy and efficiency in analyzing large datasets. This study focuses on events published as text data streams in Telegram. Two main perspectives are considered for better results: firstly, major events can overshadow the detection of smaller ones with fewer narrative instances. Secondly, due to diverse narratives surrounding each event, establishing meaningful connections between narrative sets is essential. In this research, two new concepts, “subject stream” and “stream graph,” are introduced. The subject stream aims to process topics to mitigate the impact of bursty events on detecting others. The stream graph models data stream and identifies various narratives associated with each event for better identification and categorization. Combining these approaches accurately represents events and their characteristics in the real world. Implementing the system in three versions demonstrates the effectiveness of using the “stream graph” alongside the “subject stream,” resulting in improved execution speed and accuracy. Evaluation results show a 6% enhancement in topic recall.

The influence of supervised and unsupervised methods in the analysis of low density stream sediments in the regional prospecting of pegmatites in the state of Espírito Santo, Araçuaí Orogen - Eastern Pegmatite Province of Brazil

The state of Espírito Santo is located in Brazil southeastern and it is geologically inserted in the Eastern Brazilian Pegmatite Province in the Araçuaí Orogen. This province is one of the most important pegmatite provinces in the world, known mainly for the occurrence of gem-quality minerals and high grade lithium deposits. Although Espírito Santo has a geological framework with many granite intrusions similar to those hosting mineralized pegmatites in Minas Gerais, it hasn't been acknowledged as a potential pegmatitic area. There are reports of mineralization of beryl (aquamarine), topaz, amethyst, and chrysoberyl in the state's mineral history. I this research we use regional stream sediment geochemical data of the Espirito Santo State, to create prospective geochemical maps to map gemological pegmatites. To do so, primary descriptive statistical analyses as well as Pearson and Spearman correlations and Principal Component Analysis (PCA) were used to select elements associated with pegmatites. The geochemical prospectivity mapping was carried out by supervised and unsupervised approaches. The supervised method was based on integration of the elements by C-A fractal models, used to define the geochemical thresholds, and fuzzy logic methodology to integrate weighted geochemical maps. The unsupervised approach, however, was based on interpolation of the representative principal component scores. All models were validated using the Prediction Area plot (P-A plot). The results of our research showed that unsupervised methodology overperformed the other weighted supervised models, with a prediction rate of 78.14 % and prospectivity of 0.87. The final results also show a pegmatite prospectivity map for the Espirito Santo state, that indicates a great exploration potential for gemological minerals.)

Dynamic modeling and online monitoring of tensor data streams with application to passenger flow surveillance

Dynamic modeling and online monitoring of tensor data streams with application to passenger flow surveillance

Online Learning for Data Streams With Incomplete Features and Labels

Online Learning for Data Streams With Incomplete Features and Labels

An Effective Methodology for Cloud-Based Priority Generation System using an ANFIS-CNN Classification Architecture

The cloud architecture improves the memory capacity and optimizes bandwidth across the entire network architecture, facilitating the transmission and the reception of large amounts of multimedia data. The cloud system requires workflow computation and a scheduling algorithm to enhance the network performance. Therefore, this algorithm generates the priority of each multimedia data stream for effective transmission over the wireless network. In this research, a priority (PR) generation system is proposed for the efficient transmission of large multimedia videos over wireless networks through cloud architecture. This proposed system involves preprocessing video frames, computing features, and classifying them using the ANFIS-CNN classifier. This classifier is the integration of an adaptive neuro-fuzzy inference system (ANFIS) and the Convolutional Neural Network (CNN) classifier. The ANFIS-CNN based priority generation algorithm has been tested on two independent cloud platforms: Microsoft Azure and Amazon EC2. Experiment results from these cloud platforms have been compared with other existing algorithms for the priority generation process.

Discovering outlying attributes of outliers in data streams

Data streams, continuous sequences of timestamped data points, necessitate real-time monitoring due to their time-sensitive nature. In various data stream applications, such as network security and credit card transaction monitoring, real-time detection of outliers is crucial, as these outliers often signify potential threats. Equally important is the real-time explanation of outliers, enabling users to glean insights and thereby shorten their investigation time. The investigation time for outliers is closely tied to their number of attributes, making it essential to provide explanations that detail which attributes are responsible for the abnormality of a data point, referred to as outlying attributes. However, the unbounded volume of data and concept drift of data streams pose challenges for discovering the outlying attributes of outliers in real time. In response, in this paper we propose EXOS, an algorithm designed for discovering the outlying attributes of multi-dimensional outliers in data streams. EXOS leverages cross-correlations among data streams, accommodates varying data stream schemas and arrival rates, and effectively addresses challenges related to the unbounded volume of data and concept drift. The algorithm is model-agnostic for point outlier detection and provides real-time explanations based on the local context of the outlier, derived from time-based tumbling windows. The paper provides a complexity analysis of EXOS and an experimental analysis comparing EXOS with existing algorithms. The evaluation includes an assessment of performance on both real-world and synthetic datasets in terms of average precision, recall, F1-score, and explanation time. The evaluation results show that, on average, EXOS achieves a 45.6% better F1 Score and is 7.3 times lower in explanation time compared to existing outlying attribute algorithms.

Online learning from incomplete data streams with partial labels for multi-classification

Online learning from the data streams is a research hotspot due to adaptive responses to real-time data arrival and fleeting. Existing approaches can only handle real-world scenarios partially due to constraints such as binary classification, complete labels, and fixed feature spaces. Learning from incomplete data streams with partial labels for multi-classification is crucial but rarely investigated due to its complexity and variability. To address this issue, we propose a novel Online Learning approach from Incomplete Data Streams with Partial Labels for Multi-classification, named OLIDSPLM. OLIDSPLM includes three main ideas: a) exploiting feature similarity to re-weight the most informative features in incomplete feature space (IFS) to avoid bias caused by filling in missing features, b) using self-train to label unlabeled instances and filter outliers, and c) utilizing the difference in the distribution between instances and model generated to detect concept drifts adaptively. We experimentally evaluated OLIDSPLM and its rivals in handling the IFS, partial labels, and concept drifts to validate its effectiveness. The code is released at https://github.com/youdianlong/OLIDSPLM.

Design and Evaluation of a Novel Venturi-Based Spirometer for Home Respiratory Monitoring.

The high cost and limited availability of home spirometers pose a significant barrier to effective respiratory disease management and monitoring. To address this challenge, this paper introduces a novel Venturi-based spirometer designed for home use, leveraging the Bernoulli principle. The device features a 3D-printed Venturi tube that narrows to create a pressure differential, which is measured by a differential pressure sensor and converted into airflow rate. The airflow is then integrated over time to calculate parameters such as the Forced Vital Capacity (FVC) and Forced Expiratory Volume in one second (FEV1). The system also includes a bacterial filter for hygienic use and a circuit board for data acquisition and streaming. Evaluation with eight healthy individuals demonstrated excellent test-retest reliability, with intraclass correlation coefficients (ICCs) of 0.955 for FVC and 0.853 for FEV1. Furthermore, when compared to standard Pulmonary Function Test (PFT) equipment, the spirometer exhibited strong correlation, with Pearson correlation coefficients of 0.992 for FVC and 0.968 for FEV1, and high reliability, with ICCs of 0.987 for FVC and 0.907 for FEV1. These findings suggest that the Venturi-based spirometer could significantly enhance access to spirometry at home. However, further large-scale validation and reliability studies are necessary to confirm its efficacy and reliability for widespread use.